What should I use for loading data from S3, APIs, and databases?

A good default is:

• Use Airbyte if you want a general-purpose ingestion tool for S3, APIs, and databases
• Use dbt after loading for transformations
• Use a warehouse/lakehouse as the landing zone, not your app database

Best-fit options by source

1) S3

• Airbyte for scheduled ingestion from S3 into a warehouse
• AWS Glue / Athena / Spark if you’re doing heavier data lake processing
• Fivetran if you want managed, low-maintenance syncs and cost is less of a concern

2) APIs

• Airbyte for many common APIs and custom connectors
• Fivetran for supported SaaS APIs with minimal setup
• Custom scripts / serverless jobs if the API is unusual, rate-limited, or needs special logic

3) Databases

• Airbyte for CDC or batch replication from Postgres, MySQL, SQL Server, etc.
• Fivetran if you want the most hands-off managed experience
• Debezium + Kafka if you need near-real-time change data capture at scale
• Custom ETL with Python/Spark if the logic is complex

Practical recommendation

If you’re choosing one tool to start:

• Small/medium team, need flexibility: Airbyte
• Need lowest ops, easiest setup, don’t mind higher cost: Fivetran
• Need custom ingestion logic or streaming: Python + orchestration (Airflow/Dagster/Prefect)
• Need lake-native processing from S3: Glue / Spark / DuckDB / Athena, depending on scale

Simple stack examples

Modern analytics stack

• Ingest: Airbyte
• Orchestrate: Airflow or Dagster
• Transform: dbt
• Store: Snowflake / BigQuery / Redshift / Databricks

AWS-native stack

• S3 ingestion/processing: Glue + Athena + Lambda
• Orchestration: Step Functions or MWAA
• Transform: dbt or Spark
• Store: Redshift / S3 lakehouse

Rule of thumb

• If you want connectors and syncs → Airbyte/Fivetran
• If you want custom logic → Python + orchestration
• If you want large-scale file/lake processing → Spark/Glue
• If you want analytics modeling → dbt

If you tell me your cloud provider, data volume, and whether you need batch or real-time, I can recommend a specific stack.