Prompt
How do I troubleshoot an increase in request latency?
Latest observation
Oct 23, 2026
OpenAI APIWeb search: off
To troubleshoot an increase in request latency, work from the top down so you can isolate where time is being spent.
1) Confirm the symptom
- Check whether latency increased for:
- all requests or only some endpoints
- a specific region / AZ / tenant / customer segment
- only p50, or p95/p99 as well
- Compare against:
- deployment timeline
- traffic changes
- dependency incidents
- infrastructure changes
2) Break latency into components
Look at where the time is going:
- client/network time
- load balancer / ingress time
- app processing time
- database queries
- cache calls
- downstream API calls
- queueing / thread pool waits
- serialization / response size
If you have tracing, compare spans before and after the regression.
3) Check recent changes
Common causes are:
- new deployment or config change
- database migration or schema change
- increased logging/telemetry overhead
- feature flag rollout
- dependency version change
- autoscaling / capacity reduction
- certificate/DNS/network changes
Rollback or disable the suspected change if possible.
4) Inspect infrastructure health
Look for signs of saturation:
- CPU pegged or throttling
- memory pressure / GC pauses
- disk I/O latency
- network packet loss / retransmits
- container restarts
- low worker/thread pool availability
- request queue buildup
5) Examine dependencies
Latency often comes from one slow dependency:
- database slow queries, lock contention, connection pool exhaustion
- cache misses or cache instability
- downstream service latency or errors
- message broker lag
- DNS resolution delays
Check dependency dashboards and logs.
6) Review application behavior
Look for:
- N+1 queries
- inefficient loops or serialization
- blocking I/O on critical paths
- excessive retries/timeouts
- large payloads
- cold starts / cache warmup issues
- contention on locks or shared resources
7) Compare good vs bad periods
Use a diff approach:
- same endpoint, before/after
- same request type, slow vs fast
- same server, before/after
- same dependency, healthy vs slow
This often reveals the bottleneck quickly.
8) Verify with targeted tests
- replay a representative request
- add timing around suspected code paths
- run load tests with controlled traffic
- query dependency latency directly
- inspect traces for the slowest spans
9) Mitigate quickly
Possible short-term fixes:
- rollback bad deploy
- scale up replicas / workers
- increase connection pools carefully
- disable expensive features
- reduce logging volume
- raise cache hit rate
- add retries only if safe and bounded
- optimize the slow query or hot path
10) Prevent recurrence
After fixing:
- add alerts on p95/p99 latency and dependency latency
- instrument key spans
- set SLOs and error budgets
- capture request IDs for tracing
- run load tests before releases
If you want, I can give you:
- a production incident checklist,
- a step-by-step latency debugging flowchart, or
- a cloud/service-specific runbook for AWS, Kubernetes, or a web app stack.
Brands