Upgrading clusters and draining nodes

Upgrade a kubeadm cluster one minor version at a time: first the control plane with kubeadm upgrade apply, then every other node, drained, upgraded and put back to work.

An upgrade moves the cluster to a newer Kubernetes version, for example from 1.36 to 1.37. With kubeadm it is a fixed list of about fifteen commands. The exam punishes small slips in that list: a package not pinned to a version, a kubelet never restarted, a node left cordoned.

This lesson gives you the list and the reason for each step. Then you practice the part kind can do for real: draining a node and putting it back. It belongs to the exam topic Manage the lifecycle of Kubernetes clusters.

Version skew decides the order

During an upgrade, parts of the cluster run different versions for a while. This is called version skew. The rules say how far apart the versions may be, measured against the API server:

Example: the API server runs 1.37. A kubelet on 1.34 is allowed. A kubelet on 1.38 is not. Two rules follow from the table:

• The control plane goes first, because nothing may be newer than the API server.
• One minor version per upgrade. 1.36 to 1.37 works. 1.35 straight to 1.37 is refused. Skipping patch releases is fine, for example 1.36.2 straight to 1.37.1.

One more thing: the VERSION column of kubectl get nodes shows each node's kubelet version. It changes only after you upgrade and restart the kubelet on that node. kubeadm upgrade apply alone never changes it.

Upgrade the first control plane

Run these on the first control plane node, as root. The example moves from v1.36 to v1.37.1. The same steps repeat on every node, so learn their shape once:

# exam: ssh <control-plane>; sudo -i
# root may need: export KUBECONFIG=/etc/kubernetes/admin.conf

# 1. Point apt at the new minor version and find the exact package
sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-cache madison kubeadm | head -3          # e.g. 1.37.1-1.1

# 2. kubeadm first
apt-mark unhold kubeadm
apt-get install -y kubeadm=1.37.1-1.1
apt-mark hold kubeadm
kubeadm version -o short                     # v1.37.1

# 3. Plan, then apply exactly the version you installed
kubeadm upgrade plan
kubeadm upgrade apply v1.37.1                # -y skips the confirmation prompt

# 4. kubelet and kubectl, with the node drained
kubectl drain <control-plane> --ignore-daemonsets
apt-mark unhold kubelet kubectl
apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon <control-plane>
kubectl get nodes

kubeadm upgrade apply upgrades the control plane Pods, renews their certificates, and upgrades CoreDNS and kube-proxy. It does not touch the kubelet. That is step 4 in the block above.

Other control planes and the workers

Every other node gets the same package steps. The one difference: run kubeadm upgrade node instead of upgrade apply.

# Additional control plane nodes: the same package steps, but instead of "upgrade apply":
kubeadm upgrade node

# Each worker. On the worker (ssh <worker>; sudo -i):
sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.37.1-1.1 && apt-mark hold kubeadm
kubeadm upgrade node

# Wherever kubectl has admin credentials:
kubectl drain <worker> --ignore-daemonsets --delete-emptydir-data

# Back on the worker:
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet

# And from the admin side again:
kubectl uncordon <worker>
kubectl get nodes

kubeadm upgrade node reads the target version from the cluster, so it takes no version number. Never run upgrade apply a second time on the other control planes. Upgrade the nodes one at a time, so the rest of the cluster keeps your apps running.

cordon, drain, uncordon

Before you restart a node's kubelet, move its Pods somewhere else. Example: cka-worker2 runs three web Pods. kubectl drain cka-worker2 evicts them, and their Deployment starts new copies on the other workers. After the upgrade, kubectl uncordon cka-worker2 opens the node again.