Prompt
How do I handle node failures with Kubernetes self-healing?
Latest observation
Kubernetes self-healing helps recover from node failures, but the exact behavior depends on what failed and how your workloads are configured. Here’s the practical way to handle it.
What Kubernetes does automatically
When a node fails, Kubernetes will typically:
- Mark the node NotReady after missed heartbeats
- Evict pods from the failed node after a timeout
- Reschedule pods onto healthy nodes if they’re managed by controllers like:
- Deployments
- ReplicaSets
- StatefulSets
- DaemonSets
- Jobs
If your app is run as a standalone Pod, Kubernetes won’t recreate it elsewhere. Use a controller.
Best practices for self-healing node failures
1. Run workloads with controllers
Use a Deployment for stateless apps, StatefulSet for stateful apps, etc.
Example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: my-app:1.0
This lets Kubernetes recreate pods on other nodes if one node dies.
2. Use multiple nodes and spread replicas
Don’t place all replicas on one node. Use:
- Pod anti-affinity
- topologySpreadConstraints
- multiple node pools / availability zones
Example anti-affinity:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: my-app
topologyKey: kubernetes.io/hostname
3. Configure readiness and liveness probes
These help Kubernetes detect unhealthy containers and restart them.
- livenessProbe: restart container if it hangs
- readinessProbe: remove pod from service endpoints if it’s not ready
Example:
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
4. Make storage resilient
If pods use persistent volumes:
- Use storage that survives node loss
- Ensure your CSI driver supports reattachment/recovery
- Avoid local node storage for critical data unless replicated elsewhere
5. Set reasonable pod disruption behavior
Use:
- PodDisruptionBudget to keep enough replicas available during voluntary disruptions
- priorityClassName for important workloads
Example PDB:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: my-app-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: my-app
6. Make sure the cluster can replace lost capacity
If a node fails permanently, Kubernetes can only reschedule pods if there’s room elsewhere.
Ensure:
- Cluster autoscaler is enabled, if using cloud infrastructure
- Enough spare capacity exists across nodes
- Node pools are redundant
7. Tune node failure detection and eviction behavior
Kubernetes detects node failure through the node controller and eviction manager. The timing is configurable, but defaults are often fine for most clusters.
If you need faster failover, review:
- Node lease/heartbeat intervals
node-monitor-periodpod-eviction-timeout
These are cluster-level settings and should be adjusted carefully.
Typical recovery flow
- Node stops responding
- Control plane marks node
NotReady - Pods on that node are considered failed
- Controllers create replacement pods
- Scheduler places them on healthy nodes
- Service traffic shifts once readiness probes pass
Operational checklist
- Use Deployments/StatefulSets, not naked Pods
- Run multiple replicas
- Spread pods across nodes/AZs
- Configure readiness/liveness probes
- Use durable storage for stateful apps
- Add PodDisruptionBudgets
- Enable autoscaling or maintain spare capacity
- Monitor node conditions and pod rescheduling events
If you want, I can also give you:
- a sample deployment manifest with self-healing best practices, or
- a step-by-step node failure troubleshooting guide.
Brands