Prompt
Why did my PostgreSQL replica fall behind under load?
Latest observation
A PostgreSQL replica usually falls behind under load because it can’t receive, replay, or keep up with WAL as fast as the primary produces it. The common causes fall into a few buckets:
1) The primary is generating WAL faster than the replica can apply it
Heavy write load on the primary can produce a lot of WAL:
- bulk inserts/updates/deletes
- vacuum/cleanup activity
- index builds
- checkpoint pressure
- large transactions
If the replica’s CPU, disk, or network can’t keep up, lag grows.
2) The replica is overloaded by read traffic
If you’re using the replica for reporting or application reads, those queries can compete with WAL replay for:
- CPU
- memory
- disk I/O
- cache
Long-running queries can also delay replay of conflicting changes if the replica is configured to wait rather than cancel queries.
3) Disk I/O on the replica is too slow
Replay is often limited by storage latency:
- slow EBS / network-attached storage
- saturated disks
- fsync/checkpoint contention
- poor random write performance
Even if the network is fine, the replica may simply not be able to write changes quickly enough.
4) Network latency or bandwidth limits
If WAL shipping is delayed because of:
- low bandwidth
- packet loss
- high latency
- cross-region replication
then the replica can only fall further behind during spikes.
5) WAL replay conflicts with replica queries
On hot standby replicas, replay may need to wait for queries to finish, or queries may be canceled depending on settings like:
max_standby_streaming_delayhot_standby_feedback
If queries are long-lived, replay can lag.
6) A single large transaction caused a burst
A huge transaction can create a lot of WAL at once. The replica can’t apply it until commit, so it may look fine and then suddenly lag badly.
7) Checkpoints / autovacuum / maintenance churn
These can increase WAL volume and I/O:
- checkpoint spikes
- aggressive autovacuum
- table rewrites
- schema changes
8) Replica CPU bottleneck
WAL apply is not free. If the replica is:
- underprovisioned
- running other services
- doing heavy query parsing/execution
then apply throughput drops.
How to tell which one it is
Check these first:
Replication lag
SELECT
now() - pg_last_xact_replay_timestamp() AS replay_delay;
WAL receive vs replay status
SELECT
status,
receive_start_lsn,
received_lsn,
latest_end_lsn,
latest_end_time
FROM pg_stat_wal_receiver;
Replica activity
SELECT pid, state, query_start, wait_event_type, wait_event, query
FROM pg_stat_activity
WHERE backend_type = 'client backend';
Conflicting queries / replay delays
Look for long-running queries on the replica and settings like:
max_standby_streaming_delayhot_standby_feedback
System bottlenecks
Check:
- CPU utilization
- disk latency / IOPS
- network throughput
- memory pressure / swapping
Typical fixes
- Reduce write burstiness on the primary
- Scale up replica CPU / IOPS
- Move replica storage to faster disks
- Limit or optimize read queries on the replica
- Tune replay conflict settings
- Avoid huge transactions where possible
- Improve network throughput if replication is remote
- Add more replicas or use a dedicated reporting replica
If you want, I can help you diagnose it from actual PostgreSQL metrics or logs. If you paste:
pg_stat_replicationpg_stat_wal_receiver- replica CPU/disk/network stats
- and any relevant log lines
I can help narrow down the exact cause.
Brands