How do I set up alerts for Kubernetes pod crashes

To set up alerts for Kubernetes pod crashes, the most common approach is:

1. Collect pod restart / crash metrics
2. Create alert rules
3. Send alerts to Slack, PagerDuty, email, etc.

The exact setup depends on your monitoring stack, but the usual best practice is Prometheus + Alertmanager.

---

1) What to alert on

Common signals for pod crashes:

• Pod restart count increases
• Container is in CrashLoopBackOff
• Pod is not ready
• Container terminated unexpectedly
• High rate of restarts over time

Useful Kubernetes metrics often come from:

• kube-state-metrics
• cAdvisor / node-exporter
• kubelet

---

2) Example Prometheus alert rules

Alert: container restarting too much

groups:
- name: kubernetes-pod-alerts
  rules:
  - alert: PodCrashLooping
    expr: rate(kube_pod_container_status_restarts_total[5m]) > 0
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Pod is restarting: {{ $labels.namespace }}/{{ $labels.pod }}"
      description: "Container {{ $labels.container }} in pod {{ $labels.pod }} is restarting frequently."

Alert: container in CrashLoopBackOff

groups:
- name: kubernetes-pod-alerts
  rules:
  - alert: PodCrashLoopBackOff
    expr: kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "CrashLoopBackOff in {{ $labels.namespace }}/{{ $labels.pod }}"
      description: "Container {{ $labels.container }} is in CrashLoopBackOff."

Alert: pod not ready

groups:
- name: kubernetes-pod-alerts
  rules:
  - alert: PodNotReady
    expr: kube_pod_status_ready{condition="true"} == 0
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Pod not ready: {{ $labels.namespace }}/{{ $labels.pod }}"

---

3) If using kube-prometheus-stack

If you installed monitoring via Helm (kube-prometheus-stack), you can add alert rules using a PrometheusRule object.

Example:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-crash-alerts
  namespace: monitoring
spec:
  groups:
  - name: pod-alerts
    rules:
    - alert: PodCrashLoopBackOff
      expr: kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "CrashLoopBackOff in {{ $labels.namespace }}/{{ $labels.pod }}"
        description: "Container {{ $labels.container }} is in CrashLoopBackOff."

Apply it with:

kubectl apply -f pod-crash-alerts.yaml

---

4) Configure Alertmanager notifications

Alertmanager routes alerts to destinations like Slack or email.

Slack example

route:
  receiver: slack

receivers:
- name: slack
  slack_configs:
  - channel: '#alerts'
    send_resolved: true
    api_url: 'https://hooks.slack.com/services/...'

---

5) Recommended alert thresholds

To reduce noise:

• Use for: to avoid transient spikes
• Alert on CrashLoopBackOff as critical
• Alert on restarts only if they continue for several minutes
• Exclude known batch jobs or short-lived pods if needed

Example:

• CrashLoopBackOff for 5m → critical
• restart rate > 0 for 10m → warning
• pod not ready for 10m → warning

---

6) Alternative tools

If you’re not using Prometheus, you can also use:

• Datadog
• Grafana Cloud
• New Relic
• Dynatrace
• Loki/Promtail for log-based alerts

For log-based alerts, you’d look for patterns like:

• CrashLoopBackOff
• Back-off restarting failed container
• application fatal errors / stack traces

---

7) Best practice checklist

• Monitor restarts, readiness, and CrashLoopBackOff
• Add for: to prevent flapping alerts
• Route critical alerts to paging
• Include namespace/pod/container in annotations
• Ignore expected restarts for Jobs/CronJobs where appropriate

---

If you want, I can give you:

1. a ready-to-apply YAML for kube-prometheus-stack, or
2. a setup for Slack/email alerts specifically.