Prompt
kubernetes monitoring and alerting
Latest observation
Nov 16, 2026
OpenAI APIWeb search: off
Kubernetes monitoring and alerting usually has 4 parts:
- Metrics collection – CPU, memory, disk, network, pod/container health
- Log collection – application and cluster logs
- Tracing – request flow across services
- Alerting – notify when something is unhealthy or close to failing
Common Kubernetes monitoring stack
1) Metrics
Most common setup:
- Prometheus – collects and stores metrics
- Grafana – dashboards and visualization
- kube-state-metrics – exposes Kubernetes object state like deployments, pods, nodes, HPA, etc.
- node-exporter – node-level metrics
- cAdvisor / kubelet metrics – container usage metrics
2) Alerting
- Alertmanager – sends alerts from Prometheus to email, Slack, PagerDuty, Opsgenie, etc.
3) Logging
- Loki + Promtail + Grafana or
- EFK/ELK stack: Elasticsearch, Fluentd/Fluent Bit, Kibana
4) Tracing
- OpenTelemetry
- Jaeger or Tempo
What to monitor in Kubernetes
Cluster-level
- Node CPU, memory, disk, filesystem pressure
- Node readiness / not ready
- Pod count and scheduling failures
- API server latency and errors
- etcd health
- Control plane component health
Workload-level
- Pod restarts
- CrashLoopBackOff
- OOMKilled
- CPU/memory requests vs limits
- Deployment availability
- HPA scaling behavior
- Container CPU throttling
- Persistent volume usage
- Ingress/controller errors
Application-level
- HTTP error rate
- Latency p95/p99
- Request throughput
- Queue depth
- Business KPIs
Important alert types
Critical alerts
- Node down
- Pod crash loops
- Deployment unavailable
- Disk almost full
- API server unavailable
- etcd unhealthy
Warning alerts
- High memory usage
- High CPU usage
- Pod restarts increasing
- Pending pods
- HPA maxed out
- Certificate expiration soon
Example Prometheus alert rules
Node down
groups:
- name: kubernetes-alerts
rules:
- alert: NodeDown
expr: up{job="node-exporter"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Node is down"
description: "Node {{ $labels.instance }} has been unreachable for 5 minutes."
Pod restarting too much
- alert: PodCrashLooping
expr: increase(kube_pod_container_status_restarts_total[5m]) > 3
for: 10m
labels:
severity: warning
annotations:
summary: "Pod is restarting frequently"
High memory usage
- alert: NodeMemoryHigh
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 10m
labels:
severity: warning
Best practices
- Use requests and limits properly for all containers
- Monitor both cluster health and application SLIs
- Avoid too many noisy alerts; group and tune thresholds
- Use burn rate alerts for SLO-based monitoring
- Keep dashboards simple:
- Cluster overview
- Namespace/workload overview
- Node health
- Application latency/error rate
- Alert on symptoms, not just raw resource usage
- Set retention and storage sizing for Prometheus carefully
- Secure metrics endpoints and access to Grafana/Alertmanager
Recommended starting setup for most teams
If you want a practical baseline:
- Prometheus + Alertmanager + Grafana
- kube-state-metrics
- node-exporter
- Loki for logs
- OpenTelemetry for tracing
Minimal deployment options
If you want quick installation:
- kube-prometheus-stack Helm chart
- bundles Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics
If you want, I can also give you:
- a full production monitoring architecture for Kubernetes
- Helm install steps
- Prometheus alert rules for common Kubernetes issues
- Grafana dashboard recommendations