How do I monitor container health and node issues in Kubernetes?

To monitor container health and node issues in Kubernetes, you usually combine:

1. Kubernetes health probes for containers
2. Cluster/node metrics for infrastructure health
3. Logging and alerts for failures and trends
4. Observability tooling like Prometheus/Grafana, kube-state-metrics, and node-exporter

1) Monitor container health with probes

Kubernetes provides three main probes:

Readiness probe

Checks whether a container is ready to receive traffic.

• If it fails, the pod stays running but is removed from service endpoints.
• Useful for startup dependencies like DB connections.

Liveness probe

Checks whether a container is alive.

• If it fails, Kubernetes restarts the container.
• Useful for detecting deadlocks or hung processes.

Startup probe

Checks whether a container has finished starting.

• Prevents liveness/readiness from killing slow-starting containers too early.

Example:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

startupProbe:
  httpGet:
    path: /startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

2) Check pod/container status

Useful commands:

kubectl get pods -A
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>
kubectl get events -A --sort-by=.metadata.creationTimestamp

Look for:

• CrashLoopBackOff
• ImagePullBackOff
• Error
• OOMKilled
• Pending
• probe failures in events

3) Monitor node health

Nodes can have issues like resource exhaustion, disk pressure, memory pressure, or network problems.

Check node status:

kubectl get nodes
kubectl describe node <node-name>

Important node conditions:

• Ready
• MemoryPressure
• DiskPressure
• PIDPressure
• NetworkUnavailable

Also inspect:

• kubelet logs
• container runtime logs
• system logs on the node

4) Collect metrics with Prometheus

Prometheus is commonly used to scrape metrics from:

• kube-state-metrics for Kubernetes object state
• node-exporter for host/node metrics
• cAdvisor/kubelet for container CPU/memory usage

Typical alerts:

• Pod restarts increase
• Container CPU or memory near limits
• Node not ready
• Disk usage high
• OOM kills
• High pod eviction count

5) Visualize with Grafana

Grafana dashboards help you see:

• Node CPU/memory/disk
• Pod health and restarts
• Workload saturation
• Cluster capacity
• API server and kubelet health

6) Set up alerts

Alert on:

• Pod restart loops
• Liveness/readiness probe failures
• Node NotReady
• Disk pressure
• Memory pressure
• Persistent volume full or near full
• High error rates in app logs

7) Use logs and tracing

For application-level health:

• Centralize logs with Loki, Elasticsearch/OpenSearch, etc.
• Use distributed tracing (Jaeger, Tempo, OpenTelemetry) to identify upstream/downstream failures

8) Common practical setup

A common stack is:

• Prometheus for metrics
• Grafana for dashboards
• Alertmanager for alerts
• kube-state-metrics for Kubernetes object state
• node-exporter for node metrics
• Loki for logs

Quick checklist

• Add liveness/readiness/startup probes to containers
• Monitor pod events and restart counts
• Track node conditions and resource usage
• Install Prometheus + Grafana + Alertmanager
• Create alerts for NotReady, OOMKilled, high restarts, disk/memory pressure
• Review logs for root cause analysis

If you want, I can also give you:

• a Prometheus alert rules example
• a Grafana dashboard list
• or a minimal Kubernetes monitoring stack YAML/Helm setup