I need a pipeline that can handle schema drift and retries, recommend tools

For a pipeline that needs to handle schema drift and retries, I’d recommend thinking in terms of ingestion + orchestration + storage/transform + observability. The best tools depend on whether you’re building batch, streaming, or both.

Strong tool recommendations

1) Ingestion / CDC / schema evolution

• Kafka + Schema Registry (Confluent or Apicurio)
  • Good for event-driven pipelines
  • Handles schema evolution well with Avro/Protobuf/JSON Schema
  • Useful when you want explicit compatibility controls
• Debezium
  • Great for CDC from databases
  • Works well with Kafka
  • Captures schema changes from source DBs
• Airbyte
  • Good for lots of source connectors
  • Easier to use for ELT-style ingestion
  • Can help when source schemas change, though it’s not a full schema-governance solution
• Fivetran
  • Managed option for ingestion with schema drift support
  • Less operational burden, but more expensive and less flexible

2) Orchestration and retries

• Apache Airflow
  • Mature orchestration
  • Great retry controls, backoff, alerting, dependencies
  • Best for batch and scheduled workflows
• Dagster
  • Stronger data-aware orchestration
  • Nice type/asset modeling, good for schema-aware pipelines
  • Good developer experience
• Prefect
  • Lightweight and flexible
  • Easy retries and failure handling
  • Good if you want simpler orchestration
• Temporal
  • Best if you need durable workflows and advanced retry/state handling
  • Excellent for long-running pipelines and exactly-once-ish business workflows

3) Transformation / schema handling

• dbt
  • Excellent for SQL transformations
  • Works well with schema drift if you use staging models and tests
  • Great for warehouse/lakehouse-centric pipelines
• Spark / Databricks
  • Better for large-scale data processing and semi-structured data
  • Can handle evolving schemas in JSON/Parquet workflows
• Delta Lake / Apache Iceberg / Apache Hudi
  • Strong choices for lakehouse storage
  • Better schema evolution support than raw files
  • Helpful for append-heavy or incremental data

4) Validation and drift detection

• Great Expectations
  • Data quality checks and schema validation
• Soda
  • Similar purpose, often simpler to adopt
• Pandera
  • Good for Python-based data validation
• Deequ
  • Strong for Spark-based validation

Best practical stacks

Option A: Modern batch/ELT stack

• Airbyte for ingestion
• Airflow / Dagster / Prefect for orchestration and retries
• dbt for transforms
• Snowflake / BigQuery / Databricks as warehouse/lakehouse
• Great Expectations for validation

Best if your data is mostly batch and you want manageable operations.

Option B: Streaming / event-driven stack

• Kafka + Schema Registry
• Debezium for CDC
• Flink / Spark Structured Streaming for processing
• Temporal / Airflow for workflow retries and recovery
• Iceberg / Delta Lake for storage

Best if schema drift happens often and near-real-time processing matters.

Option C: Lakehouse with strong schema evolution

• Databricks + Delta Lake
• Auto Loader for ingestion
• Delta schema evolution
• Jobs/Workflows or Airflow for orchestration
• Great Expectations / Deequ for quality checks

Best if you want fewer moving parts and strong support for evolving data.

What matters for schema drift

Look for tools that support:

• Schema evolution: add/drop columns, type changes, nested fields
• Backward/forward compatibility
• Quarantine of bad records
• Versioned schemas
• Automated contract checks
• Late-binding or flexible ingestion of semi-structured data

What matters for retries

Look for:

• Idempotent writes
• Checkpointing
• Dead-letter queues
• Exponential backoff
• Partial reruns
• Exactly-once or at-least-once semantics depending on need

My shortlist by scenario

• Best overall for resilience: Kafka + Schema Registry + Debezium + Dagster/Temporal + Iceberg/Delta
• Best for simpler batch pipelines: Airbyte + dbt + Airflow + Snowflake
• Best for heavy schema drift + scale: Databricks + Delta Lake + Auto Loader + Airflow/Dagster
• Best managed option: Fivetran + dbt + Snowflake/BigQuery

If you tell me:

1. batch or streaming,
2. your source systems,
3. cloud provider, and
4. volume/latency needs,

I can recommend a more exact stack.