Worker node failures

What NotReady means, how to read node conditions, and the four things that take a worker down on the exam: the kubelet stopped, its configuration broken, its certificate or kubeconfig wrong, or the container runtime gone.

Troubleshoot clusters and nodes is the competency here, and the kubelet is the cluster component it is really about. A worker-node task on the exam reads: node X is NotReady, fix it. The whole job happens on the node, over ssh, with three tools: systemctl, journalctl and a text editor. This lesson gives you the order to use them in, the files that matter, and what the cluster does to the Pods on that node while you work.

NotReady and the node conditions

The kubelet posts its node status every 10 seconds and renews a Lease in kube-node-lease at the same rate. When the controller-manager sees neither for node-monitor-grace-period (50 seconds by default), it sets the Ready condition to Unknown, which kubectl get nodes prints as NotReady. So: Ready False means the kubelet is alive and reporting a problem; Ready Unknown means the kubelet is not reporting at all. The Reason and Message columns of describe node say which.

kubectl get nodes -o wide
kubectl describe node cka-worker2 | sed -n '/^Conditions:/,/^Addresses:/p'
kubectl describe node cka-worker2 | grep -A3 '^Taints'
kubectl get lease -n kube-node-lease                 # RENEW TIME stops moving for a dead kubelet (-o yaml)
kubectl get pods -A -o wide --field-selector spec.nodeName=cka-worker2

The kubelet: stopped, crash-looping, or misconfigured

On the node, the first command is always systemctl status kubelet. It has three answers. inactive (dead): someone stopped it; systemctl start kubelet brings the node back within seconds, and systemctl enable kubelet makes sure a reboot does not undo it (the exam often checks this). activating (auto-restart) with Result: exit-code: the kubelet starts and exits at once, over and over; the reason is in its journal, and it is almost always a path in its configuration. active (running) while the node is still NotReady: the kubelet runs but cannot do its job; again the journal says why, typically the API server address, a certificate, or the container runtime.

systemctl status kubelet --no-pager
journalctl -u kubelet -n 50 --no-pager          # the last 50 lines; the error repeats near the end
journalctl -u kubelet -f                         # follow while you fix; Ctrl-C to stop
systemctl start kubelet
systemctl enable kubelet
systemctl restart kubelet                        # after editing its configuration
systemctl is-active kubelet; systemctl is-enabled kubelet

Where the kubelet's configuration lives

systemctl cat kubelet | grep -E 'ExecStart|Environment'
cat /var/lib/kubelet/kubeadm-flags.env
grep -nE 'staticPodPath|clientCAFile|containerRuntimeEndpoint|clusterDNS|cgroupDriver' /var/lib/kubelet/config.yaml
grep server: /etc/kubernetes/kubelet.conf
ls -l /var/lib/kubelet/pki/

A broken config.yaml produces a journal line such as failed to run Kubelet: unable to load client CA file /etc/kubernetes/pki/ca.crtt: open …: no such file or directory, or a YAML unmarshal error with a line number. Read the path it quotes, ls -l it, fix the file, systemctl restart kubelet. A wrong staticPodPath does not stop a worker (it has no static Pods), but on the control plane it makes the whole control plane vanish, which is the previous lesson's problem. A wrong cgroupDriver or containerRuntimeEndpoint fails the kubelet's startup checks the same way; the message names the value.

Certificates and the kubelet's kubeconfig

/etc/kubernetes/kubelet.conf points at /var/lib/kubelet/pki/kubelet-client-current.pem, a symlink to the latest rotated certificate; with rotateCertificates: true the kubelet renews it before expiry. If the node has been off for longer than the certificate lived, or the link is broken, the journal shows Unauthorized or x509: certificate has expired while the kubelet is active and the node stays NotReady. Check expiry with openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates -subject. The repair on a kubeadm node is a fresh client certificate through the CSR API or re-joining the node, which the certificates lesson covers; on the exam, confirming the expiry and naming the cause is usually the hard part.

Container runtime down, disk full

The kubelet needs containerd through /run/containerd/containerd.sock. With containerd stopped, crictl ps fails with connect: no such file or directory, the kubelet journal repeats container runtime is down or PLEG is not healthy, and the node reports Ready False with reason KubeletNotReady. The fix is systemctl start containerd (and enable); the kubelet reconnects by itself. crictl reads its endpoint from /etc/crictl.yaml; if it complains about the socket while containerd runs, that file is pointing somewhere else.

DiskPressure makes the kubelet evict Pods (they show Evicted in kubectl get pods) and adds the taint node.kubernetes.io/disk-pressure:NoSchedule so nothing new lands there. On the node, df -h /var/lib/containerd /var/lib/kubelet shows which filesystem is full. crictl rmi --prune removes unused images, journalctl --vacuum-size=200M shrinks the journal, and finding the one Pod writing logs into an emptyDir is usually the real answer. The thresholds are evictionHard in config.yaml; the lab's kind nodes set them to 0% so a full laptop does not evict your Pods.

What happens to the Pods on a NotReady node

1. 01t = 0. The kubelet stops. Containers keep running under containerd; the Pods still answer traffic. Nothing in the cluster notices yet.
2. 02t ≈ 50s. The node controller marks Ready Unknown. It taints the node node.kubernetes.io/unreachable:NoExecute (or node.kubernetes.io/not-ready:NoExecute for Ready False). The scheduler stops placing Pods there.
3. 03t ≈ 50s to 5m50s. Nothing is evicted. Every Pod carries two default tolerations, not-ready and unreachable for tolerationSeconds: 300, added by the DefaultTolerationSeconds admission plugin. kubectl still lists the Pods as Running; the node controller only sets their Ready condition to false, which drops them out of their Services' endpoints.
4. 04t ≈ 5m50s. The tolerations expire and the Pods are deleted through the API. Their Deployments and ReplicaSets create replacements on healthy nodes. The originals show Terminating until the kubelet returns to confirm, or forever if it does not; kubectl delete pod --force --grace-period=0 removes the record.
5. 05Exceptions. DaemonSet Pods tolerate both taints indefinitely and stay. Bare Pods are deleted and never replaced. StatefulSet Pods are not replaced until the old one is confirmed gone, by design, so a StatefulSet on a dead node needs you to force-delete its Pod.

kubectl get pod <pod> -o jsonpath='{.spec.tolerations}' | tr ',' '\n'
# {"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300}
# {"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300}
kubectl get node cka-worker2 -o jsonpath='{.spec.taints}'

cordon and drain as a tool

When a node is sick but still reachable (a failing disk, a runtime you are about to restart), do not wait for the node controller: kubectl cordon marks it unschedulable at once, and kubectl drain evicts its Pods respecting PodDisruptionBudgets so their controllers recreate them elsewhere now, not in six minutes. Fix the node, then kubectl uncordon. The cluster-upgrades lesson uses the same commands for maintenance.

kubectl cordon cka-worker2
kubectl drain cka-worker2 --ignore-daemonsets --delete-emptydir-data --timeout=120s
# ... repair the node ...
kubectl uncordon cka-worker2
kubectl get nodes      # SchedulingDisabled must be gone

The node checklist on one screen