If you need data pipelines that handle schema changes well, the best choice depends on whether you’re doing batch ETL, streaming, or warehouse ingestion. In practice, these are the most common good options:

Best options by use case

1. Apache Kafka + Schema Registry

Best for: streaming/event-driven pipelines

• Kafka is strong for moving data between systems.
• Add Confluent Schema Registry or an equivalent to manage schema evolution.
• Supports formats like Avro, Protobuf, JSON Schema with versioning rules.
• Good when producers and consumers evolve independently.

Why use it:
It gives you explicit schema control and compatibility checks, which is the cleanest way to manage change over time.

---

2. Apache Spark / Databricks

Best for: batch and large-scale transformations

• Spark can read/write with schema evolution support in some storage formats.
• Works well with Delta Lake, Iceberg, or Hudi.
• These table formats are designed for schema changes, upserts, time travel, and incremental processing.

Why use it:
If your pipelines touch lakehouse-style storage, Spark + a table format is a very common and practical choice.

---

3. dbt

Best for: warehouse transformations

• Excellent for SQL-based transformation pipelines.
• Works with warehouses like Snowflake, BigQuery, Redshift, Databricks SQL.
• Schema change handling is good when paired with warehouse features or incremental models.

Why use it:
If your data already lands in a warehouse, dbt is one of the best tools for maintainable transformations.

---

4. Apache Airflow / Dagster / Prefect

Best for: orchestration, not schema handling by themselves

• These tools schedule and manage pipeline steps.
• They don’t solve schema evolution directly.
• Often used together with Spark, dbt, Kafka, or ingestion tools.

Why use it:
Great for pipeline control, but you still need a storage/processing layer that understands schema evolution.

---

5. Delta Lake / Apache Iceberg / Apache Hudi

Best for: schema evolution in lakehouse storage

These are probably the most important technologies if your question is specifically about schema change handling:

• Delta Lake: strong support in Databricks ecosystem, simple operationally.
• Apache Iceberg: very strong open format support, increasingly popular across engines.
• Apache Hudi: good for incremental updates and CDC-style workloads.

Why use it:
They support adding columns, evolving schemas, and managing data over time much better than plain Parquet files.

---

Simple recommendation

If you want a practical default:

• Streaming pipelines: Kafka + Schema Registry + Avro/Protobuf
• Batch/lakehouse pipelines: Spark + Delta Lake or Iceberg
• Warehouse analytics: dbt on top of Snowflake/BigQuery/Databricks
• Pipeline scheduling: Airflow or Dagster

If you want the safest “schema change friendly” stack

A very common modern stack is:

Kafka → Spark/Databricks → Delta Lake/Iceberg → dbt → BI tools

or for batch:

Source → Airflow/Dagster → Spark → Iceberg/Delta → dbt

Rule of thumb

• If schema changes happen often, avoid raw file pipelines with just CSV/JSON and no schema management.
• Prefer typed serialization + schema registry for streaming.
• Prefer open table formats for analytical storage.

If you tell me:

1. batch or streaming,
2. where the data lands, and
3. your cloud/warehouse setup,

I can recommend a specific stack.