Highly available control planes

Stacked versus external etcd, quorum math, a load balancer in front of the API servers, and the kubeadm flags that let additional control plane nodes join.

This lesson covers Implement and configure a highly-available control plane. The exam will not have you build a load balancer, but it can ask you to join another control plane node, to say how many etcd members may fail, or to find out how an existing cluster is laid out. All of that rests on one question: which pieces hold state, and which do not.

What has to be redundant

The API server is stateless. Run one on every control plane node, put a load balancer in front, and any instance can serve any request. The scheduler and the controller manager also run on every control plane node, but only one copy of each acts at a time: they elect a leader through a Lease object in kube-system, and the others wait to take over. etcd is the hard part, because it holds the state and needs a majority of its members to agree on every write.

Quorum is floor(n/2) + 1. A write commits only once a quorum has it, and a member cut off from the majority stops accepting writes rather than drift apart. Even sizes buy nothing: four members tolerate one failure, the same as three, with one more machine that can break, and two members are worse than one. Run three, or five if you must survive two failures; beyond that every write waits on more peers. When quorum is lost, the API server cannot write anything. Running Pods keep running, but nothing in the cluster can change.

Stacked or external etcd

Building it with kubeadm

# First control plane. The endpoint is the load balancer's DNS name (or VIP) and port.
sudo kubeadm init \
  --control-plane-endpoint "k8s-api.lab.internal:6443" \
  --upload-certs \
  --pod-network-cidr=192.168.0.0/16

# kubeadm prints two join commands. The control plane one adds two flags:
sudo kubeadm join k8s-api.lab.internal:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>

# Two hours later the uploaded certificates are gone. On an existing control plane:
sudo kubeadm init phase upload-certs --upload-certs                  # prints a new key
sudo kubeadm token create --print-join-command --certificate-key <new-key>

controlPlaneEndpoint is the address every kubeconfig, kubelet and joining node uses, and kubeadm adds it to the API server certificate. kubeadm cannot convert a cluster initialized without it into an HA one, so set it on day one even with a single node; this kind lab has cka-control-plane:6443. The load balancer passes TCP 6443 through to every API server (TLS ends at the API server) and health-checks each backend; HAProxy with keepalived, kube-vip and cloud load balancers all work. --upload-certs stores the shared CA, ServiceAccount and front-proxy keys (and the etcd CA when etcd is stacked) in the kubeadm-certs Secret, encrypted with the certificate key. Secret and key expire after two hours. Without them you copy those files to each new control plane by hand before joining it.

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.37.1
controlPlaneEndpoint: "k8s-api.lab.internal:6443"
networking:
  podSubnet: 192.168.0.0/16
etcd:
  external:
    endpoints:
      - https://10.0.0.21:2379
      - https://10.0.0.22:2379
      - https://10.0.0.23:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

# What kubeadm decided at init time
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep '^controlPlaneEndpoint:'
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A3 '^etcd:'

# Which etcd members the API server uses (one argument per line, so commas stay intact)
kubectl -n kube-system get pod -l component=kube-apiserver \
  -o jsonpath='{range .items[0].spec.containers[0].command[*]}{@}{"\n"}{end}' | grep -- --etcd-servers

# Control plane nodes, their Pods, and the current leaders
kubectl get nodes -l node-role.kubernetes.io/control-plane
kubectl -n kube-system get pods -l tier=control-plane -o wide
kubectl -n kube-system get lease kube-scheduler kube-controller-manager \
  -o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity