etcd backup and restore

Take an etcd snapshot with etcdctl, check it with etcdutl, restore it into a new data directory and point the etcd static Pod at it without losing the control plane.

Architecture · 25%40 min2 tasks

After this lesson

01You can read etcd's endpoint and certificate paths from its static Pod manifest and pass them to etcdctl.
02You can save a snapshot and inspect it with etcdutl snapshot status.
03You can restore a snapshot into a new data directory, switch the etcd static Pod to it and confirm the API server serves the restored state.

etcd holds every object in the cluster, so a snapshot of it is the backup of cluster state, and taking and restoring one is part of Manage the lifecycle of Kubernetes clusters. Expect an etcd question on the CKA. It is good value, because the steps never change. On the exam etcdctl is installed on the control plane node and you reach it with ssh. On kind it is not, so this lesson also shows how to get the same tools onto the lab node.

Where etcdctl's flags come from

Read them from the manifest; do not type them from memorybash

# exam: ssh <control-plane>; sudo -i      lab: docker exec -it cka-control-plane bash
grep -E -- '--(listen-client-urls|trusted-ca-file|cert-file|key-file)=' /etc/kubernetes/manifests/etcd.yaml
#     - --cert-file=/etc/kubernetes/pki/etcd/server.crt
#     - --key-file=/etc/kubernetes/pki/etcd/server.key
#     - --listen-client-urls=https://127.0.0.1:2379,https://172.19.0.2:2379
#     - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

In etcd.yaml	Pass to etcdctl as	Value on this lab
`--listen-client-urls` (either URL)	`--endpoints`	`https://127.0.0.1:2379`
`--trusted-ca-file`	`--cacert`	`/etc/kubernetes/pki/etcd/ca.crt`
`--cert-file`	`--cert`	`/etc/kubernetes/pki/etcd/server.crt`
`--key-file`	`--key`	`/etc/kubernetes/pki/etcd/server.key`

kubeadm issues etcd's server certificate for client authentication too, so the server pair works as etcdctl's client identity; etcd/healthcheck-client.crt or the API server's apiserver-etcd-client.crt work as well. Two binaries share the job. etcdctl talks to a running member over the network and needs the endpoint and certificates. etcdutl works on files and needs neither. In etcd 3.6, which current Kubernetes ships, etcdctl snapshot only has save; status and restore live in etcdutl. etcd 3.5's etcdctl still accepts them with a deprecation warning, so run etcdctl version on the exam node and use what is installed. ETCDCTL_API=3 has been the default since etcd 3.4; setting it does no harm.

Take and check a snapshot

On the control plane node, as rootbash

mkdir -p /opt/cka
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/cka/etcd-snapshot.db
# Snapshot saved at /opt/cka/etcd-snapshot.db

etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table
# +----------+----------+------------+------------+---------+
# |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
# +----------+----------+------------+------------+---------+
# | 9a1eac1d |     9745 |       1010 |      15 MB |   3.6.0 |
# +----------+----------+------------+------------+---------+

# The same flags answer health and membership questions
etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
  member list -w table

Practicing on kind takes one extra step. kind does not put etcdctl on its nodes, and the etcd image has no shell, so kubectl exec … -- sh -c fails. You can call the binaries inside the etcd Pod directly, but that Pod only mounts /var/lib/etcd and /etc/kubernetes/pki/etcd, so any snapshot taken from there lands inside the live data directory, which is no place for a backup. The cleaner way is to copy the binaries the etcd container already ships onto the node and then work exactly as on the exam. The tasks below do that in their setup.

Two ways to run etcdctl on the kind labbash

# 1. Inside the etcd Pod: no shell in the image, so call etcdctl itself
kubectl -n kube-system exec etcd-cka-control-plane -- etcdctl \
  --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status -w table

# 2. Copy the binaries out of the running etcd container (/proc/<pid>/root is its filesystem)
docker exec cka-control-plane sh -c 'cp /proc/$(pgrep -xo etcd)/root/usr/local/bin/etcdctl /proc/$(pgrep -xo etcd)/root/usr/local/bin/etcdutl /usr/local/bin/'
docker exec cka-control-plane etcdctl version

Restore into a new data directory

Restore, then point the etcd static Pod at the resultbash

# 1. Restore into a NEW directory. etcdutl refuses one that is not empty.
etcdutl snapshot restore /opt/cka/etcd-snapshot.db --data-dir /var/lib/etcd-from-backup

# 2. Stop the API server while etcd switches data (the docs recommend it)
mv /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/

# 3. Change the etcd-data hostPath, and nothing else
vim /etc/kubernetes/manifests/etcd.yaml
#   volumes:
#   - hostPath:
#       path: /var/lib/etcd-from-backup      # was /var/lib/etcd
#       type: DirectoryOrCreate
#     name: etcd-data

# 4. Wait for the kubelet to recreate etcd (up to a minute), then start the API server again
crictl ps --name etcd
mv /etc/kubernetes/kube-apiserver.yaml /etc/kubernetes/manifests/

# 5. Check from wherever kubectl works
kubectl get nodes
kubectl get pods -A

Why only the hostPath: inside the container etcd keeps using --data-dir=/var/lib/etcd; you only change which host directory is mounted there. Changing --data-dir, the mountPath and the hostPath together also works, as long as all three agree. Change only the flag and etcd starts on an empty directory inside the container, which gives you a brand-new, empty cluster. A single-member restore needs no other flags. For a multi-member cluster, restore every member from the same snapshot and give each its own --name, --initial-cluster and --initial-advertise-peer-urls to match its manifest. For production clusters the etcd project also recommends --bump-revision with --mark-compacted, so clients holding watches (the API server, controllers) never see revisions go backwards.

In the docs · the tab you get on exam day

Operating etcd clusters for Kubernetes ↗search: “etcd backup restore”
PKI certificates and requirements ↗search: “pki certificates”
Create static Pods ↗search: “static pod”

Taskar-etcd-1

Architecture6%~7 min

context · kubectl config use-context kind-cka

Back up etcd and record the snapshot revision

On the control plane node cka-control-plane, take a snapshot of the cluster's etcd and save it on that node as /opt/cka/etcd-snapshot.db. Connect with the endpoint and TLS files the etcd static Pod is configured with. Then read the snapshot's revision with etcdutl and write only that number to /tmp/cka/ar-etcd-revision.txt on your workstation.

01
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-1/setup | bash
02
Solve it in your terminal
Work the way the exam makes you: kubectl, the docs, nothing else. Put any answer files under /tmp/cka/.
03
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-1/verify | bash
Counts toward your readiness score.
04
Clean up
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-1/cleanup | bash

Taskar-etcd-2

Architecture7%~12 min

context · kubectl config use-context kind-cka

Restore etcd from a snapshot

Someone deleted the ConfigMap app-config from namespace ar-etcd. A snapshot taken before the deletion is on cka-control-plane at /opt/cka/etcd-before.db. Restore etcd from that snapshot into the directory /var/lib/etcd-from-backup and make the cluster's etcd run from it. When you are done the API server must be serving again and app-config must exist.

Setup breaks part of your lab on purpose. Run the cleanup step when you finish, or reset the lab.

01
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-2/setup | bash
02
Solve it in your terminal
Work the way the exam makes you: kubectl, the docs, nothing else. Put any answer files under /tmp/cka/.
03
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-2/verify | bash
Counts toward your readiness score.
04
Clean up
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-2/cleanup | bash

Finished reading?

0 of 2 tasks passed. Readiness counts tasks, so run them before you move on.

Mark complete, next lesson →