Control plane failures

Where the control plane lives on a kubeadm node, what each component's failure looks like from kubectl, and how to debug with crictl and the kubelet journal when kubectl itself is gone.

This lesson and the next cover two competencies of the domain verbatim: Troubleshoot clusters and nodes and Troubleshoot cluster components. On the exam these are the tasks where you ssh to a control-plane node, find a manifest with one wrong character, fix it and wait for the component to come back. They look frightening because kubectl may be dead when you start. They are not: a kubeadm control plane is four files in one directory, and everything you need to read them is on the node.

Where the control plane lives

kubeadm runs kube-apiserver, kube-controller-manager, kube-scheduler and etcd as static Pods: the kubelet on the control-plane node reads their manifests from staticPodPath (set in /var/lib/kubelet/config.yaml, normally /etc/kubernetes/manifests) and runs them itself, with no scheduler or controller involved. For each one it creates a read-only mirror Pod in kube-system named <component>-<node> so you can see it in kubectl. Deleting a mirror Pod does nothing lasting; editing it is rejected. The file is the source of truth: change the file and the kubelet restarts the component; move the file out of the directory and the component stops.

grep staticPodPath /var/lib/kubelet/config.yaml
ls -l /etc/kubernetes/manifests
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

# From anywhere, while the API server works
kubectl -n kube-system get pods -o wide
kubectl -n kube-system get pod kube-apiserver-cka-control-plane -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.source}'   # file

Symptoms by component

• —kube-apiserver down. Every kubectl command fails with The connection to the server <host>:6443 was refused or Unable to connect to the server: … i/o timeout. Pods already running keep serving traffic; kubelets keep their last state; nothing can be created, scaled or deleted. This is the one failure you must debug entirely on the node.
• —kube-scheduler down. New Pods stay Pending with NODE <none> and no events, not even FailedScheduling, because nothing is looking at them. Existing Pods are unaffected. kubectl -n kube-system get pods shows the scheduler Pod in CrashLoopBackOff, Error, or missing.
• —kube-controller-manager down. A new Deployment shows 0/3 with no ReplicaSet and no Pods (kubectl get rs is empty); scaling does nothing; Jobs never start; a node you shut down is never marked NotReady; PVCs are not bound; a new namespace never gets its default ServiceAccount. The API accepts every change and nothing acts on it.
• —etcd down. The API server process is up but has nowhere to read or write: kubectl returns etcdserver: request timed out, context deadline exceeded, or hangs; kubectl get --raw /readyz?verbose shows [-]etcd failed. The API server's log is full of connection errors to 127.0.0.1:2379, and the etcd container itself is Exited or missing in crictl ps -a.

Debugging while kubectl works

kubectl -n kube-system get pods
kubectl -n kube-system describe pod kube-scheduler-cka-control-plane | tail -15
kubectl -n kube-system logs kube-scheduler-cka-control-plane
kubectl -n kube-system logs kube-controller-manager-cka-control-plane --previous
kubectl get --raw /readyz?verbose | tail -5     # per-check health of the API server, including etcd
kubectl get --raw /livez?verbose | tail -3

A mirror Pod in CrashLoopBackOff with an executable file not found message is a typo in command; ErrImagePull is a typo in image; a container that exits at once with unknown flag or no such file or directory in its logs is a typo in an argument or a path. All three are fixed in the manifest, never in the mirror Pod.

Debugging when kubectl does not work

When the API server is down you have the container runtime and the kubelet's journal. crictl talks to containerd directly and lists what is really running on the node, including containers that exited. Its output has a STATE column and an ATTEMPT column: a component whose ATTEMPT keeps climbing is crash-looping; a component with no container at all was never started, which means the kubelet could not pull its image or could not parse its manifest.

crictl ps -a                                   # every container, including Exited ones
crictl ps -a --name kube-apiserver
crictl logs <container-id>                     # the component's own output; add --tail 50
crictl pods                                    # sandboxes, including ones with no container
journalctl -u kubelet --since '10 min ago' --no-pager | grep -iE 'apiserver|manifest|static|failed' | tail -20
journalctl -u kubelet -f                       # watch while you fix
systemctl status kubelet --no-pager
cat /etc/kubernetes/manifests/kube-apiserver.yaml

On the lab, run those after docker exec -it cka-control-plane bash; on the exam, after ssh <control-plane-node> (and sudo -i if the prompt is not root). Either way, type exit before the next question: kubectl on the node has no kubeconfig and will fail with connection refused to localhost:8080, which looks like a second outage and is not.

What goes wrong in a manifest

• —Image tag typo. crictl ps -a shows no container for the component; the kubelet journal says Failed to pull image with the bad reference. Once the API server is up, the mirror Pod shows ErrImagePull. Compare the tag with the other three manifests: all four components share the Kubernetes version, except etcd.
• —Command or flag typo. The container starts and exits within a second. crictl logs prints Error: unknown flag: --etcd-serverss or exec: "kube-schedulerr": executable file not found. ATTEMPT climbs.
• —Wrong path in a flag. --kubeconfig, --client-ca-file, --etcd-certfile, --tls-private-key-file: the component logs open /etc/kubernetes/pki/…: no such file or directory. Check each path exists on the host and is mounted into the container by a hostPath volume.
• —Wrong hostPath. A volume whose path points at a file that does not exist. With type: FileOrCreate the kubelet silently creates an empty file, so the component starts and then fails on an empty kubeconfig or certificate: invalid configuration: no configuration has been provided, or a TLS error.
• —YAML that does not parse. A broken indentation or a stray character. The kubelet logs Could not process manifest file with the path and skips it, so the component simply disappears: no mirror Pod, no container, nothing in crictl ps -a. The journal is the only place this is written.
• —Manifest moved or renamed. Same effect as a parse error. The kubelet reads every file in the directory, so a backup copy saved inside it is read as a second Pod with the same name. Keep backups in /tmp or /root.

cd /etc/kubernetes/manifests
grep -n 'image:' *.yaml                                   # tags agree? (etcd has its own)
grep -o '/etc/kubernetes/[a-z./-]*' kube-apiserver.yaml | sort -u | xargs ls -ld    # every path exists?
diff /tmp/kube-apiserver.yaml.bak kube-apiserver.yaml     # if you (or the task) made a backup

How the kubelet reports it, and the fix-and-wait rhythm

The kubelet watches the manifest directory with inotify and also rescans it every 20 seconds (fileCheckFrequency), so a saved file is noticed within seconds. It kills the old container, creates the new one, and logs what it does with the static pod's name. After an API server fix, allow up to a minute before kubectl answers: the server needs to come up, pass its own health checks and reconnect to etcd. Watch with crictl ps on the node, or watch kubectl -n kube-system get pods from the student terminal once it responds.

cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak   # outside the directory
vim /etc/kubernetes/manifests/kube-apiserver.yaml                               # fix the one character
journalctl -u kubelet -f | grep -i apiserver                                    # Ctrl-C when it says the pod was created
crictl ps --name kube-apiserver                                                 # Running, ATTEMPT stable
exit
kubectl get nodes                                                               # back in business