How do I monitor ephemeral Kubernetes pods that keep restarting?

To monitor ephemeral Kubernetes pods that keep restarting, focus on capturing state quickly, tracking restarts, and shipping logs/metrics outside the pod before it disappears.

1) Watch restarts and pod status

Use kubectl to see which pods are flapping:

kubectl get pods -A -w
kubectl get pods -A -o wide
kubectl describe pod <pod-name> -n <namespace>

Look for:

• RestartCount
• CrashLoopBackOff
• OOMKilled
• Error, Evicted, ImagePullBackOff

To sort by restart count:

kubectl get pods -A \
  --sort-by='.status.containerStatuses[0].restartCount'

2) Grab logs before they vanish

For restarting containers, the previous crashed instance’s logs are usually most useful:

kubectl logs <pod-name> -n <namespace> --previous

If there are multiple containers:

kubectl logs <pod-name> -n <namespace> -c <container-name> --previous

If the pod restarts too fast, use:

kubectl logs -f <pod-name> -n <namespace>

3) Use events to understand why it’s restarting

Events often tell you more than logs:

kubectl get events -n <namespace> --sort-by=.lastTimestamp

Or:

kubectl describe pod <pod-name> -n <namespace>

This can reveal:

• probes failing
• resource limits exceeded
• scheduling issues
• node pressure
• image pull failures

4) Centralize logs outside the pod

Ephemeral pods are hard to inspect locally, so forward logs to a central system:

• Fluent Bit / Fluentd
• Vector
• Loki
• Elasticsearch / OpenSearch
• Cloud logging (CloudWatch, Stackdriver, Azure Monitor)

Best practice:

• log to stdout/stderr
• avoid relying on files inside the container
• ship logs from the node/cluster, not the pod itself

5) Add metrics and alerts

Track restart-related metrics from:

• kube-state-metrics
• Prometheus
• Grafana
• Alertmanager

Useful metrics/alerts:

• container restarts increasing quickly
• pod in CrashLoopBackOff
• high memory usage near limits
• liveness/readiness probe failures
• pod pending/running transitions

Example PromQL:

increase(kube_pod_container_status_restarts_total[5m]) > 0

6) Enable termination details

Make sure pods exit with useful information:

• non-zero exit codes
• clear application logs
• graceful shutdown handling
• dump stack traces on fatal errors

Also inspect:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState}'

This can show exit code and reason from the last crash.

7) Increase observability for short-lived jobs

If these are Jobs/CronJobs or short-lived pods:

• add structured JSON logs
• emit logs early in startup
• use sidecar or node-level log shipping
• expose Prometheus metrics if the pod lives long enough
• add startup probes to avoid premature restarts

8) Use ephemeral debugging if needed

If the pod is gone too quickly, use:

• kubectl debug to attach an ephemeral container
• temporarily increase terminationGracePeriodSeconds
• scale down restart loops by fixing probe timing or resource limits

Example:

kubectl debug -it pod/<pod-name> -n <namespace> --image=busybox

9) Common causes of repeated restarts

• application crash
• bad config or secret
• failing liveness probe
• memory limit too low
• CPU starvation
• dependency unavailable
• OOMKilled
• init container failure

Practical monitoring setup

A solid approach is:

1. Prometheus + kube-state-metrics for restart and pod health metrics
2. Loki/ELK/Cloud logging for centralized logs
3. Alertmanager/Grafana alerts on restart spikes and CrashLoopBackOff
4. kubectl describe/logs --previous for immediate triage
5. Events collection for scheduling/probe/node issues

If you want, I can also give you:

• a Prometheus alert rule for pod restarts,
• a Grafana dashboard query set,
• or a kubectl one-liner to quickly identify crashing pods.