Service and DNS failures

A hop-by-hop checklist for "the app is not reachable": DNS, the Service, its selector and targetPort, EndpointSlices, readiness, NetworkPolicies and kube-proxy.

The official competency is Troubleshoot services and networking, and it usually arrives as one sentence: the frontend cannot reach the backend, fix it. The Pods are Running, nothing is crashing, and the fault sits somewhere between the client and the container. You find it fastest by testing each hop in order instead of guessing, so this lesson gives you that order and the exact symptom each fault produces.

A client calls http://orders. Its resolver expands the name with the search domains from /etc/resolv.conf and asks CoreDNS, which answers with the Service's ClusterIP. The packet goes to that virtual IP; kube-proxy's rules on the client's node rewrite it to one Pod IP and the Service's targetPort, picked from the Service's EndpointSlices. The EndpointSlice controller fills those slices with Pods that match the selector and are Ready. Finally the CNI delivers the packet, and any NetworkPolicy selecting the destination Pod decides whether it is allowed in.

# 1. Does the Service exist, and what does it select and forward to?
kubectl -n shop get svc orders -o wide
kubectl -n shop get svc orders -o jsonpath='{.spec.selector}{"\n"}{.spec.ports}{"\n"}'

# 2. Do Pods match that selector exactly, and are they Ready?
kubectl -n shop get pods -l app=orders -o wide --show-labels

# 3. Did the selector produce endpoints? (Endpoints is deprecated since v1.33; read EndpointSlices.)
kubectl -n shop get endpointslices -l kubernetes.io/service-name=orders -o wide

# 4. Which port does the container actually listen on?
kubectl -n shop get pod -l app=orders -o jsonpath='{.items[0].spec.containers[*].ports}{"\n"}'

# 5. Is any NetworkPolicy selecting the backend Pods?
kubectl -n shop get netpol
kubectl -n shop describe netpol

# 6. Test from inside the cluster, by name and by Pod IP, with a short timeout.
kubectl -n shop run probe --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl -n shop exec probe -- wget -qO- -T 3 http://orders.shop.svc.cluster.local/
kubectl -n shop exec probe -- wget -qO- -T 3 http://<pod-ip>:8080/

Read the client's error before you read any YAML. On this lab (kube-proxy in iptables mode), a Service with no endpoints answers Connection refused, because kube-proxy installs a reject rule for it. A wrong targetPort also gives Connection refused, but from the Pod itself, and the EndpointSlice is populated. A NetworkPolicy drop gives no answer at all, so wget waits out its -T timeout. Refused with empty endpoints means selector or readiness; refused with endpoints means port; a hang means policy or the network.

Selectors, targetPort and readiness

• —Selector. A Service selector is an exact, case-sensitive AND of every pair it lists: tier: backend does not match tier: Backend, and one extra key the Pods lack matches nothing. Compare it with kubectl get pods --show-labels. The Deployment's own metadata.labels are irrelevant; only spec.template.metadata.labels end up on Pods.
• —targetPort. port is what clients dial on the ClusterIP; targetPort is where the packet lands in the Pod. Leave targetPort out and it defaults to port, the classic Connection refused when the app listens on 8080 and the Service says 80. A named targetPort follows the container if its port number changes.
• —Readiness. A Pod whose readiness probe fails stays in the EndpointSlice with ready: false and receives no Service traffic. Three unready Pods behind a correct Service look exactly like a broken selector from the client's side; the EndpointSlice tells them apart.

# In the Pod template
ports:
  - name: http
    containerPort: 8080
---
# In the Service
ports:
  - name: web
    port: 80          # what clients call
    targetPort: http  # resolved per Pod to its containerPort named http

Policies, DNS, kube-proxy and NodePort

• —NetworkPolicy. Once any policy with policyTypes: [Ingress] selects a Pod, only traffic some policy allows gets in. Check that spec.podSelector picks the backend, that from matches the client's labels (and namespace, when namespaceSelector is used), and that ports lists the container port. Policies see the packet after the Service rewrote it, so a rule for port 80 does nothing for an app on 8080.
• —DNS. Test with the full name, nslookup <svc>.<ns>.svc.cluster.local, from busybox:1.36. NXDOMAIN means the name or namespace is wrong; no servers could be reached means CoreDNS is down, unreachable or blocked. If the client's namespace has an egress policy, it must allow UDP and TCP 53 to CoreDNS, or every lookup times out.
• —kube-proxy. It runs as a DaemonSet in kube-system and programs each node's rules (iptables mode on this lab). A missing or crashing kube-proxy Pod on one node breaks ClusterIPs only for clients on that node, while Pod IPs keep working.
• —NodePort. The same Service, also opened on every node at spec.ports[].nodePort (30000 to 32767). Test with curl -s -m 3 http://<node-internal-ip>:<nodePort>/ using an IP from kubectl get nodes -o wide. On the lab run it inside a node (docker exec cka-worker curl …), because the node IPs live on Docker's network. With externalTrafficPolicy: Local only nodes that run a backend Pod answer.

# Resolver config the kubelet wrote into the Pod
kubectl -n shop exec probe -- cat /etc/resolv.conf
# search shop.svc.cluster.local svc.cluster.local cluster.local / nameserver 10.96.0.10 / options ndots:5

# Use the FQDN with busybox nslookup; it is the unambiguous test
kubectl -n shop exec probe -- nslookup orders.shop.svc.cluster.local
kubectl -n shop exec probe -- nslookup kubernetes.default.svc.cluster.local

# CoreDNS: Pods, Service, logs and its config
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system get svc kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
kubectl -n kube-system get configmap coredns -o yaml

# kube-proxy: one Ready Pod per node, and its logs
kubectl -n kube-system get ds kube-proxy
kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide
kubectl -n kube-system logs ds/kube-proxy --tail=20

# The rules it wrote for a Service, on a node (exam: ssh <node>, then sudo iptables-save)
docker exec cka-worker iptables-save | grep 'shop/orders'

# NodePort from inside a node
kubectl -n shop get svc orders -o jsonpath='{.spec.ports[0].nodePort}'; echo
docker exec cka-worker curl -s -m 3 http://localhost:<nodePort>/