CoreDNS and service discovery

How Services and Pods get DNS names, what a Pod's resolv.conf does with them, how the Corefile is put together, and how to debug a lookup that fails.

Understand and use CoreDNS is its own competency, and DNS also sits underneath every "the app cannot reach the database" troubleshooting task. CoreDNS is just a Deployment in kube-system behind a Service called kube-dns, configured by one ConfigMap. Knowing the naming scheme lets you write a lookup without thinking. Knowing the Corefile lets you change it without breaking the cluster.

The names CoreDNS answers

cluster.local is the cluster domain, set by kubelet's clusterDomain and the kubernetes plugin's zone. It is the default on the exam too, but if a lookup fails everywhere, check it: grep clusterDomain /var/lib/kubelet/config.yaml on a node.

What a Pod's resolver does

kubectl -n nw-dns exec client -- cat /etc/resolv.conf
# search nw-dns.svc.cluster.local svc.cluster.local cluster.local
# nameserver 10.96.0.10
# options ndots:5

kubectl -n kube-system get svc kube-dns      # 10.96.0.10: the ClusterIP of CoreDNS

kubelet writes this file. nameserver is the kube-dns ClusterIP (kubelet's clusterDNS). The search list is why a bare inventory works inside nw-dns and inventory.nw-dns works from any namespace. ndots:5 means any name with fewer than five dots is tried with each search suffix before being tried as is. So api.example.com costs three failed cluster lookups first. A trailing dot (api.example.com.) skips the search list.

dnsPolicy and dnsConfig

apiVersion: v1
kind: Pod
metadata:
  name: resolver
  namespace: nw-dnscfg
spec:
  dnsPolicy: ClusterFirst
  dnsConfig:
    searches:
      - corp.internal          # appended after the cluster search domains
    options:
      - name: ndots
        value: "2"             # value is a string
      - name: timeout
        value: "2"
  containers:
    - name: busybox
      image: busybox:1.36
      command: ["sleep", "3600"]

dnsConfig and dnsPolicy are immutable on a running Pod. Change them in the manifest and recreate the Pod (kubectl replace --force -f pod.yaml), or change the Deployment template and let it roll.

The Corefile

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}

• —.:53 is a server block for every zone on port 53. You can add more blocks, such as corp.internal:53 { forward . 10.0.0.53 }, to send one domain to a specific DNS server (a stub domain).
• —kubernetes answers cluster.local plus reverse lookups from the API (Services, EndpointSlices, Pods). pods insecure enables the a-b-c-d.ns.pod records.
• —forward . /etc/resolv.conf sends everything else to the resolvers of the node CoreDNS runs on. Replace the path with IPs to use specific upstreams.
• —cache 30 caches answers for up to 30 seconds. Remember this when a change "does not work" for half a minute.
• —hosts (not present by default) serves static entries; add fallthrough so other names still reach the next plugin.
• —reload rereads the Corefile when the ConfigMap changes, which takes up to a minute or two. loop stops CoreDNS if it detects it is forwarding to itself. health and ready back the Pod's probes.

# Keep a copy first. Every Pod in the cluster depends on this file.
kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/cka/Corefile.bak

kubectl -n kube-system edit cm coredns      # e.g. add, inside .:53 { … }:
#    hosts {
#       10.10.10.10 legacy-db.corp.internal
#       fallthrough
#    }

# Apply now instead of waiting for reload, then make sure both replicas come back
kubectl -n kube-system rollout restart deploy coredns
kubectl -n kube-system rollout status deploy coredns --timeout=120s
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20     # a syntax error shows up here

# Roll back if it went wrong
kubectl -n kube-system create cm coredns --from-file=Corefile=/tmp/cka/Corefile.bak --dry-run=client -o yaml | kubectl replace -f -
kubectl -n kube-system rollout restart deploy coredns

Debugging a lookup

kubectl -n nw-dns run client --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl -n nw-dns wait --for=condition=Ready pod/client --timeout=60s

kubectl -n nw-dns exec client -- nslookup inventory.nw-dns.svc.cluster.local   # one Service
kubectl -n nw-dns exec client -- nslookup inventory-pods.nw-dns.svc.cluster.local   # headless: one line per Pod
kubectl -n nw-dns exec client -- nslookup kubernetes.default.svc.cluster.local      # the API: always exists
kubectl -n nw-dns exec client -- nslookup 10.96.0.1                                 # reverse lookup

# Is CoreDNS itself healthy?
kubectl -n kube-system get deploy coredns
kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=30

Learn what broken DNS looks like, because the message names the layer. NXDOMAIN means CoreDNS answered: the name is wrong, the namespace is wrong, or the Service does not exist. ;; connection timed out; no servers could be reached means nothing answered on port 53: CoreDNS is down or has no endpoints, or a NetworkPolicy blocks egress to it. Applications then report bad address or could not resolve host, and every connection that uses a name fails while connections by IP still work.