What should I use for schema management and event versioning in a stream?

For schema management and event versioning in a streaming system, the usual best practice is:

1) Use a schema registry

A schema registry is the standard tool for managing event schemas over time.

Common choices:

• Confluent Schema Registry
• Apicurio Registry
• AWS Glue Schema Registry
• Redpanda Schema Registry / Kafka-compatible alternatives

What it gives you:

• Central place to store schema definitions
• Compatibility checks before deploying changes
• Ability to serialize/deserialize safely between producers and consumers
• Support for evolution without breaking older consumers

2) Use explicit schema evolution rules

For event versioning, don’t rely only on topic names or file naming. Instead, evolve schemas using compatibility rules such as:

• Backward compatible: new consumers can read old data
• Forward compatible: old consumers can read new data
• Full compatibility: both directions

Typical safe changes:

• Add optional fields with defaults
• Remove fields only after consumers no longer need them
• Avoid renaming/removing required fields without a migration plan

3) Prefer schema-based formats

Use data formats that work well with schema registries:

• Avro
• Protobuf
• JSON Schema

Best fit in many streaming systems:

• Avro for compactness and strong schema evolution
• Protobuf for efficiency and language support
• JSON Schema if human-readability matters more

4) Version events by schema, not by topic when possible

A common anti-pattern is creating a new topic for every event version. Better approach:

• Keep the same event type/topic
• Evolve the schema compatibly
• Only create a new event type or topic when the change is truly breaking or semantically different

5) Include an event type and version metadata

Even with a registry, it helps to include:

• event_type
• schema_version or event_version
• occurred_at
• event_id
• producer

This helps with debugging, replay, and interoperability.

Practical recommendation

If you’re using Kafka:

• Avro + Confluent Schema Registry is the most common default
• If you want stronger type guarantees across services, Protobuf + Schema Registry is also excellent

Rule of thumb

• Use a schema registry for structure
• Use compatibility rules for evolution
• Use event metadata for traceability
• Avoid versioning by topic unless necessary

If you want, I can also suggest a recommended setup for Kafka, Pulsar, or Kinesis, or give you a decision matrix for Avro vs Protobuf vs JSON Schema.