Prompt
Kubernetes deployment failing after node drain
Latest observation
Oct 19, 2026
OpenAI APIWeb search: off
A node drain can evict pods and expose issues in your Deployment/cluster that were hidden before. The most common causes are:
Likely reasons
-
Pod can’t reschedule
- Not enough CPU/memory on remaining nodes
- Node selectors / affinity / tolerations too restrictive
- PV is tied to the drained node
- PodDisruptionBudget too strict
-
Readiness/liveness failures after restart
- App depends on local cache or node-local state
- Probes are too aggressive
- Startup time is longer than probe thresholds
-
Stateful or storage-related problem
- Local PV, hostPath, or node-local volume can’t move
- Volume attach/mount failure
-
Image or config issue surfaced during restart
- Old pod kept running on drained node, new one fails to start
- Missing secret/configmap/env var
What to check first
Run these:
kubectl get pods -n <ns> -o wide
kubectl describe pod <pod-name> -n <ns>
kubectl get events -n <ns> --sort-by=.lastTimestamp
kubectl describe deploy <deploy-name> -n <ns>
kubectl get pdb -n <ns>
kubectl get nodes
Look for:
PendingImagePullBackOffFailedSchedulingReadiness probe failedMountVolume.SetUp failed0/… nodes are availablenode(s) were unschedulable
Common fixes
- Relax scheduling constraints
- Remove overly strict
nodeAffinity,nodeSelector,tolerations
- Remove overly strict
- Add capacity
- More nodes or larger nodes
- Fix storage
- Use network-attached storage instead of
hostPathor local PV
- Use network-attached storage instead of
- Adjust probes
- Increase
initialDelaySeconds,timeoutSeconds,failureThreshold
- Increase
- Review PDB
- Ensure it allows at least one pod disruption if needed
If you want, paste:
kubectl describe pod <pod>kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -30- your Deployment YAML
and I can help pinpoint the exact cause.
Brands