Highly available control planes

Stacked or external etcd, quorum math, a load balancer in front of the API servers, and the kubeadm flags that add more control plane nodes.

A highly available (HA) control plane keeps working when one control plane machine dies. You get it by running the control plane on three machines instead of one. Example: your cluster has one control plane node and its disk fails. Running apps keep going, but nobody can deploy, scale or fix anything until it is back. With three control plane nodes, the other two carry on.

This lesson covers the exam topic Implement and configure a highly-available control plane. The exam will not make you build a load balancer. It can ask you to join another control plane node, or to say how many etcd members may fail. It can also ask how a cluster is laid out.

What has to be redundant

In an HA cluster, each control plane part runs on several machines. But the copies work together in different ways:

Quorum: why three members, not two or four

Quorum means a majority of the etcd members. etcd saves a write only once a quorum has it. The formula is floor(n/2) + 1: half the members, rounded down, plus one. Example: with 3 members, quorum is 2, so 1 member can fail.

• Even numbers add nothing. 4 members survive 1 failure, the same as 3, with one more machine that can break. 2 members are worse than 1.
• Run 3, or 5 if you must survive 2 failures. More members make every write wait for more machines.
• When quorum is lost, nothing can change. The API server cannot save anything. Running Pods keep running, but you cannot create, change or delete objects.

Stacked or external etcd

There are two ways to place etcd. Stacked: each control plane node also runs an etcd member, so 3 machines give you 3 API servers and 3 etcd members. This is what kubeadm builds by default. External: etcd runs on its own machines, so the same takes 6 machines.

Building it with kubeadm

You build an HA cluster with the same kubeadm init and kubeadm join as a normal one, plus four flags:

# First control plane. The endpoint is the load balancer's DNS name (or VIP) and port.
sudo kubeadm init \
  --control-plane-endpoint "k8s-api.lab.internal:6443" \
  --upload-certs \
  --pod-network-cidr=192.168.0.0/16

# kubeadm prints two join commands. The control plane one adds two flags:
sudo kubeadm join k8s-api.lab.internal:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>

# Two hours later the uploaded certificates are gone. On an existing control plane:
sudo kubeadm init phase upload-certs --upload-certs                  # prints a new key
sudo kubeadm token create --print-join-command --certificate-key <new-key>

controlPlaneEndpoint is the one address that every kubeconfig, kubelet and joining node uses. Set it on day one, even with a single control plane node: kubeadm cannot turn a cluster built without it into an HA one later. On this kind lab it is cka-control-plane:6443.

With external etcd you use a config file instead of flags. Only the etcd part differs:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
controlPlaneEndpoint: "k8s-api.lab.internal:6443"
etcd:
  external:
    endpoints:
      - https://10.0.0.21:2379
      - https://10.0.0.22:2379
      - https://10.0.0.23:2379
    # plus caFile, certFile and keyFile: the client certificate the API server uses for etcd

Reading an existing cluster's layout

Exam questions about HA are mostly about reading. kubeadm saved its choices in the kubeadm-config ConfigMap, and the API server's flags show which etcd members it uses:

# What kubeadm decided at init time
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep '^controlPlaneEndpoint:'
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A3 '^etcd:'

# Which etcd members the API server uses (one argument per line, so commas stay intact)
kubectl -n kube-system get pod -l component=kube-apiserver \
  -o jsonpath='{range .items[0].spec.containers[0].command[*]}{@}{"\n"}{end}' | grep -- --etcd-servers

# Control plane nodes and their Pods
kubectl get nodes -l node-role.kubernetes.io/control-plane
kubectl -n kube-system get pods -l tier=control-plane -o wide