etcd backup and restore

Back up etcd with etcdctl, check the snapshot with etcdutl, then restore it into a new folder and point etcd at it.

40 min 2 hands-on tasksArchitecture · 25% of the exam

Key ideas in this lesson · tap one for a simple explanation

What you will be able to do

Read etcd's address and certificate paths from its static Pod file and pass them to etcdctl.
Save a snapshot and check it with etcdutl snapshot status.
Restore a snapshot into a new data directory, switch the etcd static Pod to it and confirm the cluster serves the restored state.

is the cluster's database. It stores every object: every , and . A snapshot is a copy of that whole database in one file, so it is your backup of the cluster. Example: someone deletes a by mistake. If you took a snapshot this morning, you can restore it. The ConfigMap is back, and so is everything else, exactly as it was this morning.

Expect an etcd question on the CKA. It is good value, because the steps never change. It belongs to the exam topic Manage the lifecycle of Kubernetes clusters. On the exam the etcd tools are already installed on the , and you reach it with ssh.

Backup and restore

etcd

the live database

/var/lib/etcd

Snapshot file

etcdctl snapshot save

/opt/cka/etcd-snapshot.db

New data folder

etcdutl snapshot restore

/var/lib/etcd-from-backup

etcd again

etcd.yaml now points at the new folder

Save copies the live database into one file. Restore unpacks that file into a new folder. Then you point etcd at the new folder.

etcd, etcdctl and etcdutl

Three names that look alike. One is the database, and two are tools that work on it:

Name	What it is	What it needs	Example
etcd	The database itself. On clusters it runs as a on the control plane.	Its data folder, `/var/lib/etcd`	`/etc/kubernetes/manifests/etcd.yaml`
etcdctl	A client that talks to a running etcd over the network.	etcd's address and certificates	`etcdctl snapshot save`
etcdutl	A tool that works on etcd's files directly, even when etcd is not running.	Only the file's path	`etcdutl snapshot status`, `etcdutl snapshot restore`

Where etcdctl's flags come from

etcdctl needs four things to reach etcd: its address and three certificate files. Never type them from memory. Copy them from etcd's static Pod file:

Read them from the manifest; do not type them from memory· Terminal

# exam: ssh <control-plane>; sudo -i      lab: docker exec -it cka-control-plane bash
grep -E -- '--(listen-client-urls|trusted-ca-file|cert-file|key-file)=' /etc/kubernetes/manifests/etcd.yaml
#     - --cert-file=/etc/kubernetes/pki/etcd/server.crt
#     - --key-file=/etc/kubernetes/pki/etcd/server.key
#     - --listen-client-urls=https://127.0.0.1:2379,https://<node-ip>:2379
#     - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

In etcd.yaml	Pass to etcdctl as	Value on this lab
`--listen-client-urls` (either URL)	`--endpoints`	`https://127.0.0.1:2379`
`--trusted-ca-file`	`--cacert`	`/etc/kubernetes/pki/etcd/ca.crt`
`--cert-file`	`--cert`	`/etc/kubernetes/pki/etcd/server.crt`
`--key-file`	`--key`	`/etc/kubernetes/pki/etcd/server.key`

The certificate pair is etcd's own server certificate. kubeadm makes it valid for clients too, so etcdctl can log in to etcd with it.

Take and check a snapshot

Lab only: copy etcdctl and etcdutl onto the kind node· Terminal

# Copy the tools out of the running etcd container (/proc/<pid>/root is its filesystem)
docker exec cka-control-plane sh -c 'cp /proc/$(pgrep -xo etcd)/root/usr/local/bin/etcdctl /proc/$(pgrep -xo etcd)/root/usr/local/bin/etcdutl /usr/local/bin/'
docker exec cka-control-plane etcdctl version

Now save the snapshot, then check that the file is valid. Run these on the control plane node, as root:

On the control plane node, as root· Terminal

mkdir -p /opt/cka
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/cka/etcd-snapshot.db
# Snapshot saved at /opt/cka/etcd-snapshot.db

etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table
# +----------+----------+------------+------------+---------+
# |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
# +----------+----------+------------+------------+---------+
# | 9a1eac1d |     9745 |       1010 |      15 MB |   3.6.0 |
# +----------+----------+------------+------------+---------+

Restore into a new data directory

Restoring takes three moves. Unpack the snapshot into a new folder. Point etcd at that folder. Wait for the control plane to come back. Stop the during the switch, so nothing writes to etcd half way.

Step 2 below parks the API server's file in /etc/kubernetes, outside the folder. Never park it, or any backup, inside that folder: the runs every file there (see Control plane and node components).

Restore, then point the etcd static Pod at the result· Terminal

# 1. Restore into a NEW directory. etcdutl refuses one that is not empty.
etcdutl snapshot restore /opt/cka/etcd-snapshot.db --data-dir /var/lib/etcd-from-backup

# 2. Stop the API server while etcd switches data (the docs recommend it)
mv /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/

# 3. Change the etcd-data hostPath, and nothing else
vim /etc/kubernetes/manifests/etcd.yaml
#   volumes:
#   - hostPath:
#       path: /var/lib/etcd-from-backup      # was /var/lib/etcd
#       type: DirectoryOrCreate
#     name: etcd-data

# 4. Wait for the kubelet to recreate etcd (up to a minute), then start the API server again
crictl ps --name etcd
mv /etc/kubernetes/kube-apiserver.yaml /etc/kubernetes/manifests/

# 5. Check from wherever kubectl works
kubectl get nodes
kubectl get pods -A

Which line to change in etcd.yaml

Folder on the node

the hostPath: change this

/var/lib/etcd-from-backup

Folder in the container

the mountPath: leave it

/var/lib/etcd

etcd

the flag: leave it

--data-dir=/var/lib/etcd

Only the hostPath changes. etcd still reads /var/lib/etcd inside its container, but now finds the restored data there.

Be careful with the --data-dir flag. If you change only the flag, etcd starts on an empty folder inside the container, and you get a brand-new, empty cluster. Changing the flag, the mountPath and the hostPath together also works, as long as all three agree. Changing only the hostPath is the simplest.

In the official docs· open on exam day too

Operating etcd clusters for Kubernetessearch: etcd backup restore
PKI certificates and requirementssearch: pki certificates
Create static Podssearch: static pod

Hands-on task 1 of 2ar-etcd-1

~7 min6% weightArchitecture

Back up etcd and record the snapshot revision

On the control plane node cka-control-plane, take a snapshot of the cluster's etcd and save it on that node as /opt/cka/etcd-snapshot.db. Connect with the endpoint and TLS files the etcd static Pod is configured with. Then read the snapshot's revision with etcdutl and write only that number to /tmp/cka/ar-etcd-revision.txt on your workstation.

Cluster:kubectl config use-context kind-cka

1
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-1/setup | bash
2
Solve it in your terminal
Use kubectl and the Kubernetes docs, like on exam day. Save any answer files under /tmp/cka/.
3
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-1/verify | bash
Got FAIL? Read which check failed, fix it, run it again.
4
Clean up (optional)
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-1/cleanup | bash

Hands-on task 2 of 2ar-etcd-2

~12 min7% weightArchitecture

Restore etcd from a snapshot

Someone deleted the ConfigMap app-config from namespace ar-etcd. A snapshot taken before the deletion is on cka-control-plane at /opt/cka/etcd-before.db. Restore etcd from that snapshot into the directory /var/lib/etcd-from-backup and make the cluster's etcd run from it. When you are done the API server must be serving again and app-config must exist.

Cluster:kubectl config use-context kind-cka

Setup breaks part of your lab on purpose, like the exam does. Run the clean-up step when you finish.

1
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-2/setup | bash
2
Solve it in your terminal
Use kubectl and the Kubernetes docs, like on exam day. Save any answer files under /tmp/cka/.
3
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-2/verify | bash
Got FAIL? Read which check failed, fix it, run it again.
4
Clean up (optional)
curl -fsSL https://codewithmuh.com/labs/cka/t/ar-etcd-2/cleanup | bash

Finished this lesson?

0 of 2 tasks passed. Your readiness counts tasks, so run them before you move on.

Complete and continue