Prompt
I hate maintaining custom ETL scripts for every new source
Latest observation
Oct 8, 2026
OpenAI APIWeb search: off
Totally fair—custom ETL per source gets painful fast.
A few ways to reduce that pain:
- Use a standardized ingestion layer: land every source in the same raw format first, then normalize downstream.
- Prefer connectors over scripts: tools like Fivetran, Airbyte, Stitch, or vendor APIs can eliminate a lot of custom glue.
- Adopt a schema contract: define required fields, types, and validation rules so new sources plug into a known shape.
- Use config-driven pipelines: one pipeline codebase, many sources via metadata/config instead of one-off code.
- Separate extraction from transformation: keep source-specific logic minimal; move business logic into dbt or similar.
- Build a source onboarding template: auth, mapping, validation, tests, and monitoring all in one repeatable package.
If you want, I can help you design a generic ingestion architecture that minimizes custom ETL, or compare buy vs build options for your setup.