Upgrading clusters and draining nodes

Upgrade a kubeadm cluster one minor version at a time: the first control plane with kubeadm upgrade apply, then every other node drained, upgraded and returned to service.

Upgrades are the heart of the competency Manage the lifecycle of Kubernetes clusters. A kubeadm upgrade is a fixed sequence of about fifteen commands, and the exam punishes small slips in it: a package left unpinned, a kubelet never restarted, a node left cordoned. This lesson gives you the sequence and the reasoning behind it, then lets you practice the part kind can do for real: draining and returning a node.

Version skew decides the order

Two rules fall out of that table. The control plane goes first, because nothing may be newer than the API server. And kubeadm moves one minor version per upgrade: 1.36 to 1.37 works, 1.35 straight to 1.37 is refused, while skipping patch releases is fine. Remember that the VERSION column of kubectl get nodes shows each node's kubelet version. It changes only after you upgrade and restart the kubelet on that node, never because of kubeadm upgrade apply.

Upgrade the first control plane

# exam: ssh <control-plane>; sudo -i
# root may need: export KUBECONFIG=/etc/kubernetes/admin.conf

# 1. Point apt at the new minor version and find the exact package
sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-cache madison kubeadm | head -3          # e.g. 1.37.1-1.1

# 2. kubeadm first
apt-mark unhold kubeadm
apt-get install -y kubeadm=1.37.1-1.1
apt-mark hold kubeadm
kubeadm version -o short                     # v1.37.1

# 3. Plan, then apply exactly the version you installed
kubeadm upgrade plan
kubeadm upgrade apply v1.37.1                # -y skips the confirmation prompt

# 4. kubelet and kubectl, with the node drained
kubectl drain <control-plane> --ignore-daemonsets
apt-mark unhold kubelet kubectl
apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon <control-plane>
kubectl get nodes

kubeadm upgrade plan checks that the cluster is healthy and lists the versions you can move to. kubeadm upgrade apply pulls the new images and rewrites the static Pod manifests one component at a time, keeping backups under /etc/kubernetes/tmp and rolling a component back if its new Pod does not come up. It also renews the control plane certificates and upgrades the CoreDNS and kube-proxy add-ons. It does not touch the kubelet binary; that is the package step that follows.

Other control planes and the workers

# Additional control plane nodes: the same package steps, but instead of "upgrade apply":
kubeadm upgrade node

# Each worker. On the worker (ssh <worker>; sudo -i):
sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.37.1-1.1 && apt-mark hold kubeadm
kubeadm upgrade node

# Wherever kubectl has admin credentials:
kubectl drain <worker> --ignore-daemonsets --delete-emptydir-data

# Back on the worker:
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet

# And from the admin side again:
kubectl uncordon <worker>
kubectl get nodes

kubeadm upgrade node is the per-node half of the upgrade. On another control plane node it rewrites that node's static Pod manifests for the version the cluster now runs; on a worker it refreshes the kubelet configuration in /var/lib/kubelet/config.yaml. It takes no version argument because it reads the target from the cluster. Never run upgrade apply again on the other control planes. Do workers one at a time, or in batches small enough for your spare capacity and PodDisruptionBudgets.

cordon, drain, uncordon