I need a pipeline that supports schema evolution without breaking loads

Yes — the key is to design the pipeline so schema changes are additive and backward-compatible by default, and to make the ingestion layer tolerant of field drift.

Recommended pattern

1) Use a schema registry or contract

Define an explicit schema contract for each dataset/event stream:

• field name
• data type
• nullability
• default behavior
• version

If you’re using Kafka, Avro/Protobuf/JSON Schema + Schema Registry is a common approach.

2) Enforce compatibility rules

Prefer:

• additive changes: new nullable columns/fields
• type widening where safe: int -> long, float -> double
• optional fields instead of required
• deprecated fields kept for a grace period

Avoid:

• renaming fields without aliases/mapping
• dropping fields immediately
• changing types incompatibly
• making previously optional fields required

3) Land raw data first

Store incoming data in a raw/staging layer exactly as received:

• keep original payload
• include ingestion timestamp and source metadata
• don’t fail the whole load because one field appeared/disappeared

Then transform into curated tables/views.

4) Make transformations schema-tolerant

In your ETL/ELT logic:

• select known fields explicitly
• use coalesce / defaults for missing values
• ignore unknown fields unless needed
• validate only critical fields
• quarantine bad records instead of blocking the batch

5) Use a wide, nullable target schema

For warehouse tables:

• add new columns as nullable
• backfill later if needed
• don’t require full-table rewrites for every evolution

6) Version your transforms

Keep transformation logic versioned:

• v1 consumes old schema
• v2 handles new fields
• use feature flags or branch-by-schema-version if needed

7) Add automated compatibility checks

Before deployment:

• compare new schema vs old schema
• fail CI if a breaking change is introduced
• test sample payloads from multiple versions

---

Practical example

If your source originally has:

{
  "order_id": "123",
  "amount": 50.0
}

Later it evolves to:

{
  "order_id": "123",
  "amount": 50.0,
  "currency": "USD"
}

Your pipeline should:

• accept both versions
• load currency as nullable
• set a default if needed in downstream transforms

---

Simple rules that prevent broken loads

1. Never require new fields immediately
2. Treat unknown fields as ignorable
3. Keep old fields alive until consumers migrate
4. Land raw data before transformation
5. Validate compatibility before release

---

If you want, I can also give you:

• a concrete design for Kafka + Spark + Snowflake/BigQuery
• a SQL table evolution strategy
• a Python/ETL example that handles schema drift gracefully