Worker node failures

What NotReady means, how to read a node's conditions, and the four things that take a worker down on the exam: the kubelet stopped, its config broken, its certificate or kubeconfig wrong, or the container runtime gone.

This lesson covers the exam topic Troubleshoot clusters and nodes. The part it is really about is the kubelet. A typical task says: node X is NotReady, fix it. You do the whole job on the node, over ssh, with three tools: systemctl, journalctl and a text editor. This lesson shows the order to use them in, and the files that matter.

NotReady and the node conditions

Every 10 seconds the kubelet sends a heartbeat to say it is alive. If the controller manager hears nothing for 50 seconds, it sets the node's Ready condition to Unknown. kubectl get nodes then shows NotReady. But NotReady comes in two kinds, and they send you to different places:

Here is every node condition, with what to check. For Ready, trouble is False or Unknown. For the other conditions, trouble is True:

kubectl get nodes -o wide
kubectl describe node cka-worker2 | sed -n '/^Conditions:/,/^Addresses:/p'
kubectl describe node cka-worker2 | grep -A3 '^Taints'
kubectl get pods -A -o wide --field-selector spec.nodeName=cka-worker2

The kubelet: stopped, crash-looping, or misconfigured

On the node, the first command is always systemctl status kubelet. Its Active line gives one of three answers:

• inactive (dead): someone stopped it. systemctl start kubelet brings the node back within seconds. Then run systemctl enable kubelet, so a reboot does not undo it. The exam often checks this.
• activating (auto-restart): the kubelet starts and exits, again and again. Its journal says why, and it is almost always a wrong path in its config file.
• active (running), but the node is still NotReady: the kubelet runs but cannot do its job. Again the journal says why: usually the API server address, a certificate, or the container runtime.

systemctl status kubelet --no-pager
journalctl -u kubelet -n 50 --no-pager          # the last 50 lines; the error repeats near the end
journalctl -u kubelet -f                         # follow while you fix; Ctrl-C to stop
systemctl start kubelet
systemctl enable kubelet
systemctl restart kubelet                        # after editing its configuration
systemctl is-active kubelet; systemctl is-enabled kubelet

Where the kubelet's configuration lives

Two files hold almost everything that breaks:

grep -nE 'staticPodPath|clientCAFile|containerRuntimeEndpoint|clusterDNS|cgroupDriver' /var/lib/kubelet/config.yaml
grep server: /etc/kubernetes/kubelet.conf

Here is the most common case, a broken config.yaml:

• Symptom: systemctl status kubelet says activating (auto-restart).
• Cause: the journal names it. For example: failed to run Kubelet: unable to load client CA file /etc/kubernetes/pki/ca.crtt: open …: no such file or directory. A YAML typo gives an error with a line number instead.
• Fix: run ls -l on the path it quotes, correct the file, then run systemctl restart kubelet.

An expired kubelet certificate

The kubelet proves who it is to the API server with a client certificate. Normally it renews that certificate by itself, before it expires. But if the node is off for longer than the certificate lasts, the certificate expires:

• Symptom: the kubelet is active, but the node stays NotReady. The journal shows Unauthorized or x509: certificate has expired.
• Check: openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates prints the certificate's dates.
• Fix: the node needs a new client certificate, through the CSR API, or it must join the cluster again. Certificates, kubeconfig and the CSR API covers certificates.

Container runtime down, disk full

What happens to the Pods on a NotReady node

• The first minutes: the containers keep running. About 50 seconds after the last heartbeat, the node is marked NotReady and gets a NoExecute taint: node.kubernetes.io/unreachable, or not-ready for Ready = False. The scheduler stops placing Pods there, and its Pods drop out of their Services.
• About 5 minutes later: every Pod has a default toleration for these taints that lasts 300 seconds. When it runs out, the Pods are deleted, and their Deployments create new ones on healthy nodes. The old Pods show Terminating until the kubelet comes back.
• Exceptions: DaemonSet Pods stay. Bare Pods are deleted and never come back. StatefulSet Pods are not replaced until the old one is really gone, so you must force-delete it.

If a sick node is still reachable, you do not have to wait: kubectl drain moves its Pods away now. Upgrading clusters and draining nodes teaches cordon and drain.

The node checklist on one screen