CoreDNS and service discovery

How Services and Pods get DNS names, how a Pod looks them up, and how to change the Corefile safely.

Inside a cluster, apps find each other by name. Your frontend calls http://api, and DNS turns api into the Service's IP. CoreDNS is the DNS server that answers. It runs as a Deployment in kube-system, behind a Service called kube-dns, and one ConfigMap holds its settings. The exam topic is Understand and use CoreDNS.

The names CoreDNS answers

Every Service gets a name made of its own name, its namespace and svc.cluster.local. For Service inventory in namespace nw-dns, that is inventory.nw-dns.svc.cluster.local. In the table, an A record is a DNS answer that maps a name to an IP.

What a Pod's resolver does

The kubelet writes a file called /etc/resolv.conf into every Pod. It tells the Pod where to send lookups, and how to finish short names.

kubectl -n nw-dns exec client -- cat /etc/resolv.conf
# search nw-dns.svc.cluster.local svc.cluster.local cluster.local
# nameserver 10.96.0.10
# options ndots:5

kubectl -n kube-system get svc kube-dns      # 10.96.0.10: the ClusterIP of CoreDNS

• nameserver 10.96.0.10 is the ClusterIP of the kube-dns Service. Every lookup goes there.
• search lists the endings tried for a short name. That is why inventory works inside nw-dns, and inventory.nw-dns works from any namespace.
• ndots:5 means a name with fewer than five dots gets each search ending tried first. So api.example.com costs three failed cluster lookups before the real one. A dot at the end (api.example.com.) skips the search list.

dnsPolicy and dnsConfig

Two Pod fields change that file. dnsPolicy picks where lookups go. dnsConfig adds your own settings on top.

apiVersion: v1
kind: Pod
metadata:
  name: resolver
  namespace: nw-dnscfg
spec:
  dnsPolicy: ClusterFirst      # dnsPolicy and dnsConfig are immutable: change the manifest and
  dnsConfig:                   # recreate the Pod (kubectl replace --force -f), or edit a Deployment template
    searches:
      - corp.internal          # appended after the cluster search domains
    options:
      - name: ndots
        value: "2"             # value is a string
      - name: timeout
        value: "2"
  containers:
    - name: busybox
      image: busybox:1.36
      command: ["sleep", "3600"]

The Corefile

CoreDNS reads its settings from a file called the Corefile, stored in the coredns ConfigMap in kube-system. Each line inside the block turns on one plugin. Here is the one in this lab:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}

You do not need every line. These are the ones exam tasks touch:

• kubernetes answers names in cluster.local, from the Services and Pods in the API.
• forward . /etc/resolv.conf sends every other name to the DNS servers of the node CoreDNS runs on.
• cache 30 keeps answers for up to 30 seconds. Here it skips cluster.local names, so only outside names can lag behind a change.
• hosts is not there by default. It serves fixed name-to-IP entries. Always add fallthrough, so names it does not list still reach the next plugin.
• reload rereads the Corefile when the ConfigMap changes. That can take a minute or two, so restart CoreDNS instead of waiting.

Every Pod in the cluster depends on this file, and one typo breaks DNS everywhere. So change it in four steps: back it up, edit it, restart CoreDNS, and check the logs.

# Keep a copy first. Every Pod in the cluster depends on this file.
kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/cka/Corefile.bak

kubectl -n kube-system edit cm coredns      # e.g. add, inside .:53 { … }:
#    hosts {
#       10.10.10.10 legacy-db.corp.internal
#       fallthrough
#    }

# Apply now instead of waiting for reload, then make sure both replicas come back
kubectl -n kube-system rollout restart deploy coredns
kubectl -n kube-system rollout status deploy coredns --timeout=120s
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20     # a syntax error shows up here

# Roll back if it went wrong
kubectl -n kube-system create cm coredns --from-file=Corefile=/tmp/cka/Corefile.bak --dry-run=client -o yaml | kubectl replace -f -
kubectl -n kube-system rollout restart deploy coredns

Running a lookup

To test DNS, run nslookup from a client Pod in the namespace you care about. The busybox:1.36 image has it.

kubectl -n nw-dns run client --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl -n nw-dns wait --for=condition=Ready pod/client --timeout=60s

kubectl -n nw-dns exec client -- nslookup inventory.nw-dns.svc.cluster.local   # one Service
kubectl -n nw-dns exec client -- nslookup inventory-pods.nw-dns.svc.cluster.local   # headless: one line per Pod
kubectl -n nw-dns exec client -- nslookup kubernetes.default.svc.cluster.local      # the API: always exists

When a lookup fails, the error message tells you which part is broken. Service and DNS failures walks through each case.