Service and DNS failures

A step-by-step check for "the app is not reachable": test each stop between the client and the Pod, in order.

This exam skill is called Troubleshoot services and networking. The task is usually one sentence: the frontend cannot reach the backend, fix it. The Pods are Running and nothing crashes. The fault is somewhere between the client and the app. Do not guess. Check each stop on the path in order. The first stop that fails is your fault.

The path of one request

Say a client Pod calls http://orders. Before the request reaches the app, it passes these stops:

Read the error first

The client's error tells you which stop to look at, so read it before any YAML. These are the errors wget prints in a busybox Pod on this lab:

• Refused, with an empty slice: on this lab kube-proxy runs in iptables mode. For a Service with no Pods, it adds a rule that rejects the call at once.
• Refused, with Pods listed: the request reaches a Pod, but nothing listens on that port, so the Pod refuses it.
• A hang: a NetworkPolicy drops the request and sends no answer at all, so the client waits until its timeout.

The checklist as commands

Here is the same check as commands, for a Service orders in namespace shop. Run them top to bottom and stop at the first surprise.

# 1. Does the Service exist, and what does it select and forward to?
kubectl -n shop get svc orders -o wide
kubectl -n shop get svc orders -o jsonpath='{.spec.selector}{"\n"}{.spec.ports}{"\n"}'

# 2. Do Pods match that selector exactly, and are they Ready?
kubectl -n shop get pods -l app=orders -o wide --show-labels

# 3. Did the selector find Pods? The EndpointSlice lists them.
kubectl -n shop get endpointslices -l kubernetes.io/service-name=orders -o wide

# 4. Which port does the container say it listens on?
kubectl -n shop get pod -l app=orders -o jsonpath='{.items[0].spec.containers[*].ports}{"\n"}'

# 5. Is any NetworkPolicy selecting the backend Pods?
kubectl -n shop get netpol
kubectl -n shop describe netpol

# 6. Test from inside the cluster, by name and by Pod IP, with a short timeout.
kubectl -n shop run probe --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl -n shop exec probe -- wget -qO- -T 3 http://orders.shop.svc.cluster.local/
kubectl -n shop exec probe -- wget -qO- -T 3 http://<pod-ip>:8080/

Selector, targetPort and readiness

These three cause most Service faults. The first two tasks below test the selector and the targetPort.

• Selector. Every label in the selector must match a Pod label exactly, so tier: backed finds no Pod with tier: backend. Compare it with kubectl get pods --show-labels.
• targetPort. port is what clients call, and targetPort is the port the app listens on inside the Pod. If you leave targetPort out, it copies port. It can also be the container port's name, such as http. Services shows how.
• Readiness. A Pod that fails its readiness probe stays in the EndpointSlice as not ready, and gets no traffic.

NetworkPolicy, DNS and kube-proxy

• NetworkPolicy. Once a policy selects a Pod for ingress, only traffic that a policy allows gets in. Check that podSelector picks the backend, that from matches the client's labels, and that ports lists the container port. The policy sees the request after the Service has changed port 80 into 8080. So a rule for port 80 does nothing for an app on 8080.
• DNS. Test with the full name, <svc>.<ns>.svc.cluster.local, from a busybox Pod. An egress policy that does not allow port 53 also breaks every lookup. CoreDNS and service discovery explains DNS errors and fixes.
• kube-proxy. It runs one Pod per node and does the Service forwarding. So a broken kube-proxy Pod breaks Service IPs only for clients on its node.
• NodePort. The same Service, also opened on a port of every node. Services shows how to test one.

# The Pod's DNS settings
kubectl -n shop exec probe -- cat /etc/resolv.conf

# Look up the full name, then a name that always exists
kubectl -n shop exec probe -- nslookup orders.shop.svc.cluster.local
kubectl -n shop exec probe -- nslookup kubernetes.default.svc.cluster.local

# CoreDNS: its Pods, then their logs
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50

# kube-proxy: one Running Pod per node
kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide