I'm building a data pipeline for Postgres to BigQuery, what should I use?

If you’re moving data from Postgres → BigQuery, the best tool depends on whether you want simple batch syncs, near-real-time CDC, or a fully managed ELT pipeline.

Good options

1) Airbyte

Best if you want an open-source / flexible connector-based approach.

• Supports Postgres and BigQuery well
• Can do full refresh or incremental sync
• Easy to self-host or use cloud
• Good if you may add more sources later

Choose Airbyte if: you want control, extensibility, and lower vendor lock-in.

---

2) Fivetran

Best if you want the least operational overhead.

• Very easy setup
• Strong CDC/incremental replication from Postgres
• Reliable managed service
• Automatically handles schema changes well

Choose Fivetran if: you want “set it and forget it” and are okay paying more.

---

3) Google Cloud Datastream + Dataflow / BigQuery

Best if you want a Google-native architecture.

• Datastream can capture Postgres changes
• BigQuery integration is strong
• Good for CDC-style replication
• More engineering work than SaaS ELT tools

Choose this if: you’re already on GCP and want tighter cloud-native control.

---

4) dbt + scheduled loads

Best for transformation, not extraction.

• dbt is great after data lands in BigQuery
• Not a replication tool by itself
• Usually paired with Airbyte/Fivetran/Datastream

Choose dbt if: you need transformations, modeling, and tests in BigQuery.

---

What I’d recommend

If you want the easiest production setup:

• Fivetran + dbt

If you want open-source and flexibility:

• Airbyte + dbt

If you want streaming/CDC in GCP:

• Datastream → BigQuery + dbt

Practical recommendation

For most teams:

• Start with Airbyte if budget and flexibility matter
• Use Fivetran if reliability and speed to production matter most
• Use dbt for modeling in BigQuery regardless

Questions to choose the right one

1. Do you need real-time / CDC or is hourly/daily sync enough?
2. How much operational overhead can you tolerate?
3. Are you already using GCP?
4. Do you need just replication, or also transformation and modeling?

If you want, I can give you a recommended stack based on:

• your data volume,
• freshness requirements,
• budget,
• and whether you’re on GCP/AWS/Azure.