Prompt
What should I use to find bottlenecks in a distributed system
Latest observation
To find bottlenecks in a distributed system, use observability tools that cover metrics, logs, and tracing—with distributed tracing being the most directly useful for pinpointing where requests slow down across services.
What to use
-
Distributed tracing
Best for following a request end-to-end and identifying which service, call, or dependency is causing latency.- Tools: OpenTelemetry, Jaeger, Zipkin, Datadog APM, New Relic
-
Metrics monitoring
Good for spotting high-level bottlenecks like CPU, memory, disk I/O, network saturation, queue depth, error rate, and latency.- Tools: Prometheus + Grafana, Datadog, CloudWatch, Stackdriver/Cloud Monitoring
-
Centralized logging
Useful for investigating specific failures or slow paths after you’ve identified a suspicious component.- Tools: ELK/EFK stack, Splunk, Loki
Best practice
Use all three together:
- Metrics tell you something is wrong.
- Tracing tells you where it is wrong.
- Logs tell you why it is wrong.
Common bottleneck indicators
- High request latency
- Queue buildup
- Increased retries/timeouts
- CPU/memory exhaustion
- Slow database queries
- Network bottlenecks
- Lock contention / thread starvation
- Hot partitions / uneven load
If you want a simple starting point
Install OpenTelemetry for tracing and Prometheus + Grafana for metrics. That combination is a strong baseline for identifying bottlenecks in most distributed systems.
If you want, I can also suggest a tool stack based on your environment (Kubernetes, AWS, Java microservices, etc.).