kubernetes monitoring for production

For production Kubernetes monitoring, you want coverage across cluster health, application performance, infrastructure, and alerting—not just “is the cluster up.”

What to monitor

1) Cluster-level

• API server: request latency, error rates, saturation
• etcd: latency, leader changes, disk sync latency, DB size
• Scheduler / controller manager: queue depth, leader health, reconcile errors
• Node health: CPU, memory, disk, filesystem, network, node conditions
• Kubelet / cAdvisor: pod/container resource usage, restarts, OOM kills
• Cluster events: failed scheduling, evictions, image pull errors, etc.

2) Workload-level

• Pod restarts / crash loops
• Deployment health: desired vs available replicas
• HPA behavior: scaling events, maxed-out replicas
• Resource requests vs usage
• Latency, throughput, errors for each service
• SLOs: availability, p95/p99 latency, error budgets

3) Infrastructure-level

• Nodes: CPU, RAM, disk, network, filesystem pressure
• Persistent volumes: capacity, IOPS, latency, errors
• Ingress / load balancer: request rate, 4xx/5xx, latency
• Container runtime: image pull failures, runtime errors

---

Common production stack

A widely used setup is:

• Prometheus for metrics collection
• Alertmanager for alert routing
• Grafana for dashboards
• Loki for logs
• Tempo or Jaeger for tracing
• kube-state-metrics for Kubernetes object state
• node-exporter for node metrics
• cAdvisor / kubelet metrics for container metrics
• metrics-server for autoscaling and basic resource metrics

---

Recommended alerts

Start with alerts that are actionable and low-noise:

Critical

• Node NotReady
• Pod CrashLoopBackOff
• Deployment unavailable replicas > 0
• etcd unhealthy / leader issues
• API server elevated 5xx
• Disk nearly full on nodes / volumes
• High pod restart rate
• OOM kills
• Ingress 5xx spike

Warning

• High memory/CPU usage sustained
• PVC usage above threshold
• Pod pending too long
• HPA at max replicas
• Frequent evictions
• Certificate expiration approaching

---

Best practices

• Use SLIs/SLOs, not just raw infrastructure metrics
• Alert on symptoms and causes
• Avoid alert spam: tune thresholds and use grouping/inhibition
• Separate prod vs non-prod
• Label everything: namespace, service, cluster, environment, team
• Set retention based on compliance and troubleshooting needs
• Test alerting with deliberate failures
• Secure monitoring endpoints and limit RBAC
• Monitor from outside the cluster too to detect networking / DNS / ingress issues

---

Minimal production setup

If you want a practical baseline:

1. Install Prometheus + Alertmanager + Grafana
2. Add kube-state-metrics and node-exporter
3. Enable kubelet/container metrics
4. Add blackbox exporter for synthetic probing
5. Add Loki for logs
6. Define alerts for nodes, workloads, and storage
7. Build dashboards for:
   • Cluster overview
   • Node health
   • Namespace/workload health
   • API server / etcd
   • Ingress / service latency

---

Good tools by category

• Metrics: Prometheus, Thanos, VictoriaMetrics, Grafana Mimir
• Logs: Loki, Elasticsearch/OpenSearch
• Tracing: Jaeger, Tempo, OpenTelemetry
• K8s state: kube-state-metrics
• Node metrics: node-exporter
• Synthetic checks: blackbox-exporter, Grafana synthetic monitoring

---

If you want, I can give you either:

1. a production-ready Prometheus/Grafana stack architecture,
2. a Helm-based install guide, or
3. a list of exact Prometheus alert rules for Kubernetes.