Prompt
How do I troubleshoot a service that times out only under load?
Latest observation
When a service only times out under load, the goal is to find what becomes scarce first: CPU, threads, connections, locks, downstream dependencies, or request/queue capacity. A good troubleshooting approach is to reproduce the issue, measure where latency grows, and then follow the bottleneck.
1) Confirm the failure mode
First, determine exactly what “times out” means:
- Client-side timeout or server-side timeout?
- Does the request actually finish late, or is it dropped/queued?
- Is it all requests, or only certain endpoints / tenants / payload sizes?
Useful checks:
- Compare p50 / p95 / p99 latency under load.
- Look at error codes: 429, 503, 504, connection timeout, read timeout.
- Check whether timeouts happen only after a certain request rate or concurrent connection level.
2) Reproduce with controlled load
Use a load test that ramps up gradually:
- Increase RPS and separately increase concurrency
- Keep request patterns realistic
- Test one instance if possible to isolate the bottleneck
This helps distinguish:
- Throughput limit: service can handle only so many requests/sec
- Concurrency limit: service degrades when too many requests are in flight
3) Check the classic bottlenecks
CPU saturation
Symptoms:
- CPU near 100%
- Increased latency across many endpoints
- Run queue / load average rises
What to inspect:
- CPU profiles
- Hot code paths
- Excessive serialization/deserialization
- Regex, encryption, compression, JSON processing
Memory pressure / GC
Symptoms:
- Latency spikes
- Frequent GC pauses
- High heap usage or swapping
What to inspect:
- GC logs / pause times
- Memory allocations
- Object churn
- Paging/swapping
Thread/worker pool exhaustion
Symptoms:
- Requests queue up waiting for workers
- Thread pool queue length grows
- Active threads max out
What to inspect:
- Web server worker pool
- Background job pools
- Async executor pools
- Thread dumps to see blocked/waiting threads
Connection pool exhaustion
Very common for database or HTTP clients. Symptoms:
- Requests wait for DB connection
- Timeouts increase with load
- Pool stats show maxed-out connections
What to inspect:
- DB pool size, wait time, checkout time
- HTTP client pool limits
- Slow connection acquisition
Lock contention / synchronization
Symptoms:
- CPU may not be maxed, but throughput stalls
- Many threads blocked on the same lock
What to inspect:
- Thread dumps
- Contention hot spots
- Shared caches, mutexes, synchronized sections
4) Check downstream dependencies
Often the service itself is fine, but a dependency slows down under load:
- Database
- Cache
- Message broker
- External API
- File/object storage
Look for:
- Increased query latency
- Lock waits in DB
- Slow queries
- Rate limiting by downstream services
- Retry storms amplifying load
Important: retries can make things worse under stress. If the service retries aggressively, it can create a positive feedback loop.
5) Inspect queueing and backpressure
Timeouts under load often mean requests are being queued somewhere:
- Load balancer
- Reverse proxy
- App server
- Thread pool
- DB pool
- Message queue
Things to measure:
- Queue depth
- Queue wait time
- In-flight requests
- Backpressure signals
- Request admission rate
If queues are unbounded, latency can grow until everything times out. A bounded queue with clear rejection is often safer than “infinite waiting.”
6) Correlate latency with resource metrics
At the same timestamp, compare:
- CPU
- Memory/GC
- Disk I/O
- Network I/O
- Open file descriptors
- Thread counts
- DB pool usage
- Cache hit rate
If possible, break down latency into:
- Time in app
- Time waiting for DB
- Time waiting for downstream calls
- Time in queue
- Time in serialization/network
7) Use tracing and profiling
Distributed tracing is very useful if requests cross multiple services:
- Find the slow span
- Identify where time is spent
- Compare normal vs overloaded traces
Profiling can reveal:
- Hot functions
- Allocation hotspots
- Lock contention
- Excessive logging
8) Check for load-specific bugs
Some issues only appear under load:
- Race conditions
- Deadlocks
- Cache stampedes
- N+1 query patterns
- Burstiness from scheduled jobs
- Log volume overwhelming disk or CPU
9) Validate timeout settings
Sometimes the service is “slow,” but the timeout is too aggressive or mismatched:
- Client timeout shorter than server processing time
- Load balancer idle timeout too low
- DB timeout too low/high
- Retry policy too aggressive
Make sure timeouts are aligned across:
- Client
- Proxy/load balancer
- App
- Downstream dependencies
10) Mitigations while debugging
If production is affected:
- Reduce incoming traffic temporarily
- Scale out horizontally if applicable
- Disable expensive features
- Increase pool sizes carefully
- Add rate limiting
- Reduce retries
- Add circuit breakers
- Shed load instead of timing out everything
A practical workflow
- Reproduce the issue in a staging-like environment
- Increase load gradually and note when latency starts rising
- Check metrics: CPU, memory, threads, pools, downstream latency
- Inspect traces and thread dumps at the point of slowdown
- Identify the first saturated resource
- Fix the bottleneck and retest
If you want, I can also give you a step-by-step troubleshooting checklist for a specific stack, like:
- Java/Spring
- Node.js
- Go
- Python/Django/FastAPI
- Kubernetes
- Nginx/Envoy
- PostgreSQL/MySQL backend