Prompt
Kubernetes pods not restarting after node failure
Latest observation
If a pod is not restarting after a node failure, the key point is that Pods are usually not “restarted” on the same node—they are rescheduled onto another node by a controller.
Common reasons this happens
1) The pod is not managed by a controller
If the pod was created directly (not via Deployment, ReplicaSet, StatefulSet, DaemonSet, Job), Kubernetes will not recreate it after node loss.
Check:
kubectl get pod <pod-name> -o yaml | grep -i ownerReferences -A5
Fix: Create the workload via a Deployment/StatefulSet/etc.
2) The node is not marked NotReady / eviction hasn’t happened yet
Kubernetes waits before evicting pods from an unhealthy node.
Typical timing:
- Node becomes
NotReady - After
node-monitor-grace-periodandpod-eviction-timeout, the control plane evicts pods - Then new pods are scheduled elsewhere
This can take minutes.
Check node status:
kubectl get nodes
kubectl describe node <node-name>
3) No healthy nodes available
If all other nodes are full, tainted, or incompatible, the pod can’t be scheduled.
Check pending pods:
kubectl get pods -A | grep Pending
kubectl describe pod <pod-name>
Look at the events section for scheduling failures.
4) Pod has scheduling constraints
Things like:
nodeSelectornodeAffinitypodAffinitytolerationstopologySpreadConstraints- hostPort conflicts
can prevent rescheduling.
Check:
kubectl describe pod <pod-name>
5) Storage is blocking rescheduling
If the pod uses a volume that can only attach to one node at a time, the new pod may be stuck waiting for detach/attach.
Common with:
- cloud block storage
- ReadWriteOnce PVCs
Check events:
kubectl describe pod <pod-name>
kubectl describe pvc <pvc-name>
6) Cluster autoscaler is not enabled
If the remaining nodes don’t have capacity, Kubernetes won’t create new nodes unless cluster autoscaling is configured.
What to look at first
Run these:
kubectl get pods -o wide
kubectl get nodes
kubectl describe pod <pod-name>
kubectl get events --sort-by=.metadata.creationTimestamp
Also check whether the pod has an owner:
kubectl get pod <pod-name> -o jsonpath='{.metadata.ownerReferences}'
Important behavior note
A pod on a failed node may show as Unknown or remain until Kubernetes times it out. The replacement pod is created only if the pod belongs to a controller and the scheduler can place it somewhere else.
If you want, I can help you debug your specific case if you paste:
kubectl describe pod <pod>kubectl get nodeskubectl get pod <pod> -o yaml | sed -n '/ownerReferences:/,/spec:/p'
Brands