Control plane failures

Where the control plane lives on a kubeadm node, what each part's failure looks like from kubectl, and how to debug with crictl and the kubelet's log when kubectl itself does not work.

This lesson and the next cover two exam topics: Troubleshoot clusters and nodes and Troubleshoot cluster components.

In a typical task, you ssh to the control plane node and find a file with one wrong character. You fix it and wait for the part to come back. These tasks look scary, because kubectl may not work when you start. But a kubeadm control plane is just four files in one folder, and every tool you need is on the node.

Where the control plane lives

A quick recap from Control plane and node components: the four control plane parts are static Pods. The kubelet runs them from files in /etc/kubernetes/manifests, the folder named by staticPodPath in /var/lib/kubelet/config.yaml. In kubectl, each shows up as a read-only mirror Pod named <part>-<node>, such as kube-apiserver-cka-control-plane. So to fix a part, edit its file, never the mirror Pod.

grep staticPodPath /var/lib/kubelet/config.yaml
ls -l /etc/kubernetes/manifests
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

# From anywhere, while the API server works
kubectl -n kube-system get pods -o wide
kubectl -n kube-system get pod kube-apiserver-cka-control-plane -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.source}'   # file

Each part reads its settings from flags in its file. These are the ones that break most often:

Symptoms by part

• kube-apiserver down. Every kubectl command fails with The connection to the server <host>:6443 was refused, or with an i/o timeout. Pods already running keep serving traffic, but nothing can be created, scaled or deleted. This is the one failure you must debug entirely on the node.
• kube-scheduler down. New Pods stay Pending with NODE <none> and no events at all, not even FailedScheduling. Nothing is looking at them. kubectl -n kube-system get pods shows the scheduler Pod in CrashLoopBackOff or Error, or missing.
• kube-controller-manager down. The API accepts every change, but nothing acts on it. A new Deployment shows 0/3, with no ReplicaSet and no Pods. Scaling does nothing, Jobs never start, and a node you shut down is never marked NotReady.
• etcd down. The API server runs, but cannot read or write its data. kubectl hangs, or says etcdserver: request timed out or context deadline exceeded. On the node, crictl ps -a shows the etcd container Exited or missing.

Debugging while kubectl works

kubectl -n kube-system get pods
kubectl -n kube-system describe pod kube-scheduler-cka-control-plane | tail -15
kubectl -n kube-system logs kube-scheduler-cka-control-plane
kubectl -n kube-system logs kube-controller-manager-cka-control-plane --previous
kubectl get --raw '/readyz?verbose'             # one [+] ok or [-] failed line per check, etcd included

A mirror Pod has events and logs like any other Pod, so describe and logs usually show the typo. The table in What goes wrong in a manifest below lists which typo gives which message. Whatever you find, fix it in the manifest file.

Debugging when kubectl does not work

When the API server is down, kubectl cannot reach anything. Two tools still work, because they run on the node and do not need the API server:

In crictl ps -a, read two columns. STATE says Running or Exited. ATTEMPT counts restarts: if it keeps climbing, the part is crash-looping. If a part has no container at all, it never started. The kubelet could not pull its image or could not read its file.

crictl ps -a                                   # every container, including Exited ones
crictl ps -a --name kube-apiserver
crictl logs <container-id>                     # the component's own output; add --tail 50
crictl pods                                    # sandboxes, including ones with no container
journalctl -u kubelet --since '10 min ago' --no-pager | grep -iE 'apiserver|manifest|static|failed' | tail -20
journalctl -u kubelet -f                       # watch while you fix
systemctl status kubelet --no-pager
cat /etc/kubernetes/manifests/kube-apiserver.yaml

On the lab, run those after docker exec -it cka-control-plane bash. On the exam, run them after ssh <control-plane-node>, and sudo -i if you are not root. Either way, type exit before the next question. kubectl on the node has no kubeconfig, so it fails with connection refused to localhost:8080. That looks like a second outage, but it is not.

What goes wrong in a manifest

A file moved out of the folder makes its part stop, too. And never keep a backup copy inside the folder: the kubelet runs every file there, as Control plane and node components warns.

cd /etc/kubernetes/manifests
grep -n 'image:' *.yaml                                   # tags agree? (etcd has its own)
grep -o '/etc/kubernetes/[a-z./-]*' kube-apiserver.yaml | sort -u | xargs ls -ld    # every path exists?
diff /tmp/kube-apiserver.yaml.bak kube-apiserver.yaml     # if you (or the task) made a backup

Fix, then wait: the rhythm

The kubelet watches the folder, so it notices a saved file within seconds. It stops the old container and starts a new one. After an API server fix, allow up to a minute before kubectl answers. The server must start, pass its own health checks and reconnect to etcd. Watch with crictl ps on the node, or with watch kubectl -n kube-system get pods once kubectl answers.

cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak   # outside the directory
vim /etc/kubernetes/manifests/kube-apiserver.yaml                               # fix the one character
journalctl -u kubelet -f | grep -i apiserver                                    # Ctrl-C when it says the pod was created
crictl ps --name kube-apiserver                                                 # Running, ATTEMPT stable
exit
kubectl get nodes                                                               # back in business