Here’s a practical way to troubleshoot Kubernetes service discovery when pods can’t resolve or reach services.

1) Confirm the Service exists and has endpoints

A Service without endpoints usually means the selector doesn’t match any pods, or the pods aren’t Ready.

kubectl get svc -A
kubectl describe svc <service-name> -n <namespace>
kubectl get endpoints <service-name> -n <namespace>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service-name>

What to look for:

• ClusterIP is present
• Selector matches your pod labels
• Endpoints are populated
• If endpoints are empty, check pod labels and readiness

Check pods:

kubectl get pods -n <namespace> --show-labels
kubectl describe pod <pod-name> -n <namespace>

2) Verify DNS resolution from inside a pod

Service discovery usually means DNS via CoreDNS.

Run a debug pod:

kubectl run -it --rm dns-test --image=busybox:1.36 --restart=Never -- sh

Inside it:

nslookup <service-name>
nslookup <service-name>.<namespace>.svc.cluster.local

If that fails:

• DNS not working
• Pod’s DNS config may be wrong
• CoreDNS may be unhealthy

3) Check CoreDNS health

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system deploy/coredns
kubectl describe deployment -n kube-system coredns
kubectl get svc -n kube-system kube-dns

Look for:

• CrashLoopBackOff
• Timeouts talking to the API server
• Config errors in CoreDNS Corefile

You can inspect the Corefile:

kubectl -n kube-system get configmap coredns -o yaml

4) Verify pod DNS settings

Check what DNS the pod actually has:

kubectl exec -n <namespace> <pod-name> -- cat /etc/resolv.conf

Expected:

• Nameserver points to cluster DNS service IP
• Search domains include the namespace and svc.cluster.local

If not:

• Pod spec may override dnsPolicy or dnsConfig
• For normal pods, use dnsPolicy: ClusterFirst

5) Test service connectivity directly by ClusterIP

If DNS resolves but traffic still fails, test the Service IP.

Get the ClusterIP:

kubectl get svc <service-name> -n <namespace>

From a pod:

wget -qO- http://<cluster-ip>:<port>
# or
nc -vz <cluster-ip> <port>

If DNS works but ClusterIP doesn’t:

• kube-proxy issue
• CNI/network policy issue
• service port/targetPort mismatch
• application not listening on the expected port

6) Check service ports and targetPort

A common mistake is incorrect port mapping.

kubectl describe svc <service-name> -n <namespace>

Confirm:

• port is what clients use
• targetPort matches the container’s listening port
• Protocol is correct (TCP/UDP)

Also check the app inside the pod:

kubectl exec -n <namespace> <pod-name> -- ss -lntp

7) Check network policies

NetworkPolicies can block service traffic even when DNS is fine.

kubectl get netpol -A
kubectl describe netpol -n <namespace> <policy-name>

Look for:

• Default-deny ingress/egress
• Missing allow rules for client and server namespaces/pods
• DNS egress blocked to kube-dns/CoreDNS

8) Check kube-proxy and CNI

If services don’t route correctly, kube-proxy or the CNI may be broken.

kubectl get pods -n kube-system -o wide | egrep 'kube-proxy|cilium|calico|weave|flannel'
kubectl logs -n kube-system <kube-proxy-pod>

Signs of trouble:

• kube-proxy not running on nodes
• iptables/IPVS rules not being programmed
• CNI pods failing

On nodes, if you have access:

iptables -t nat -L -n | grep KUBE
ipvsadm -Ln

9) Confirm the pod is Ready

Services only route to Ready pods by default.

kubectl get pod -n <namespace>
kubectl describe pod -n <namespace> <pod-name>

Check:

• Readiness probe failures
• Container not ready
• App startup issues

10) Look for namespace/name mismatches

Remember service discovery names are namespace-scoped.

Examples:

• myservice works only in same namespace
• Cross-namespace should use:

  myservice.otherns.svc.cluster.local
  

11) If using headless services

Headless services behave differently.

Check:

kubectl get svc <service-name> -n <namespace> -o yaml

If clusterIP: None, DNS returns pod IPs directly. Make sure:

• Endpoints exist
• Client understands multiple A records
• StatefulSet DNS names are correct

12) Use dig, curl, and tcpdump if needed

For deeper debugging:

kubectl exec -it <pod> -- sh
apk add bind-tools curl  # if image allows
dig <service-name>
curl -v http://<service-name>:<port>

If node-level access is available:

• tcpdump on node or pod network to see DNS queries and service traffic

---

Quick checklist

If service discovery fails, check in this order:

1. Service exists
2. Endpoints exist
3. Pod labels/selectors match
4. Pod is Ready
5. DNS works from inside pod
6. CoreDNS is healthy
7. NetworkPolicy isn’t blocking
8. kube-proxy/CNI are healthy
9. Service port/targetPort are correct

If you want, I can also give you:

• a step-by-step debugging command sequence
• a decision tree for DNS vs routing problems
• or help you troubleshoot a specific error message.