Why is Airflow so hard to maintain for simple syncs?

Because Airflow is a workflow orchestrator, not a lightweight “run this sync every hour” tool. For simple syncs, it can feel like you’re paying a lot of complexity tax for capabilities you don’t actually need.

Common reasons it feels hard to maintain:

• Too much abstraction: DAGs, tasks, operators, hooks, sensors, retries, XComs, pools, schedules, dependencies, etc. Even a small sync can end up wrapped in a lot of framework code.
• Operational overhead: You have to manage the scheduler, workers, metadata DB, DAG parsing, upgrades, connections, secrets, logging, and alerting.
• Python code is not just code: Airflow DAG files are parsed frequently, so “normal” Python patterns can cause issues if they do heavy work at import time or have side effects.
• Failure handling becomes scattered: Simple sync logic often needs retries, idempotency, backfills, partial failures, and alerting. Airflow makes this possible, but not always ergonomic.
• DAG drift and config sprawl: Over time, small syncs accumulate custom macros, helpers, default args, and environment-specific behavior, making them harder to reason about.
• Debugging is indirect: Instead of running a sync directly, you’re often debugging through scheduler state, task instance state, logs, and metadata rows.
• It’s optimized for scale and orchestration: If you only need “poll source A, copy to B,” Airflow’s strengths can feel like overhead rather than leverage.

Why it especially hurts for simple syncs:

• The business logic is tiny, but the surrounding Airflow boilerplate is large.
• The sync often doesn’t need true workflow orchestration, just job scheduling + retries + alerting.
• Maintenance cost is dominated by platform behavior, not application logic.

Rule of thumb:

• Use Airflow when you need multiple dependent steps, backfills, branching, SLAs, or cross-system orchestration.
• Consider something simpler for straightforward syncs: cron + a script, Prefect, Dagster, a managed ETL tool, or even a small service/worker with a scheduler.

If you want, I can also give you a “when Airflow is worth it vs when it isn’t” checklist or suggest a simpler architecture for common sync patterns.