How do I monitor microservices performance in Kubernetes?

To monitor microservices performance in Kubernetes, it helps to track four layers: cluster, nodes/pods, services, and application behavior.

1) Collect core metrics

Start with metrics from:

• Kubernetes control plane: API server latency, scheduler health, etcd
• Nodes: CPU, memory, disk I/O, network
• Pods/containers: CPU throttling, memory limits/usage, restarts, OOM kills
• Application metrics: request rate, latency, error rate, queue depth, business KPIs

Common stack:

• Prometheus for metrics collection and storage
• Grafana for dashboards
• kube-state-metrics for Kubernetes object state
• node-exporter for node-level metrics

2) Instrument each microservice

Expose application metrics in a format Prometheus can scrape, usually via /metrics.

Track the main RED metrics:

• Rate: requests per second
• Errors: failed requests, 5xxs, exceptions
• Duration: latency percentiles like p95/p99

Also useful:

• Dependency call latency
• DB query time
• Cache hit/miss rate
• Circuit breaker state
• Retry counts
• Queue backlog

3) Add distributed tracing

Metrics tell you something is slow; tracing tells you where.

Use:

• OpenTelemetry for instrumentation
• Jaeger, Tempo, or Zipkin for trace storage and visualization

This is especially useful for:

• Microservice call chains
• Identifying slow downstream dependencies
• Seeing where time is spent across services

4) Centralize logs

Use logs alongside metrics and traces:

• Loki + Grafana, or
• ELK/EFK stack: Elasticsearch/OpenSearch + Fluent Bit/Fluentd + Kibana/OpenSearch Dashboards

Include:

• Correlation IDs / trace IDs
• Request IDs
• Error stack traces
• Structured JSON logs

5) Set alerts on SLOs, not just resource usage

Alert on user-impacting signals:

• p95/p99 latency above threshold
• Error rate above threshold
• Saturation: CPU throttling, memory pressure, queue buildup
• Pod crash loops or frequent restarts

Examples:

• “API error rate > 1% for 5 minutes”
• “p95 latency > 500ms for 10 minutes”
• “Pod restarts > 3 in 15 minutes”
• “CPU throttling > 20%”

6) Use Kubernetes-native tooling

Helpful tools:

• kubectl top pods/nodes for quick checks
• Metrics Server for HPA and basic resource metrics
• HPA/VPA for scaling based on load
• Service mesh observability if you use one, like Istio or Linkerd

7) Build dashboards per service

For each microservice dashboard, show:

• Request rate
• Success/error rate
• Latency p50/p95/p99
• CPU/memory usage
• Restarts and OOMs
• Downstream dependency health
• Pod count / replica status

8) A practical reference setup

A common production setup is:

• Prometheus + Grafana
• kube-state-metrics
• node-exporter
• OpenTelemetry Collector
• Jaeger/Tempo
• Loki or ELK
• Alerts via Alertmanager to Slack/PagerDuty/email

9) Key best practices

• Use structured logs
• Propagate trace IDs across services
• Monitor p95/p99, not just averages
• Separate infrastructure and application dashboards
• Keep alerts actionable and low-noise
• Tag metrics with service, namespace, pod, version, and environment

If you want, I can also give you:

1. a recommended open-source monitoring stack for Kubernetes, or
2. a step-by-step setup guide with Prometheus and Grafana.