kubectl debug, ephemeral containers and crictl

How to debug when the image has no shell: ephemeral containers, debug copies, node debugging Pods and crictl.

Many production images are distroless: they hold the app and nothing else. No shell, no ps, no wget. So kubectl exec into them fails with executable file not found. kubectl debug fixes that: it brings a second image that has the tools. The exam expects you to use it quickly, and to clean up what it leaves behind.

When the API server itself is broken, kubectl cannot help at all. Then crictl, run on the node, is the fallback.

Which tool, when

Ephemeral containers

An ephemeral container is a temporary container you add to a Pod that is already running. The Pod is not restarted. Use one to look inside an app whose image has no tools.

• It is limited. It has no ports, probes or resource settings, and it is never restarted.
• It stays. You cannot change or remove it. It stays in spec.ephemeralContainers until the Pod is deleted.
• --target=app puts it in the same process list as container app. Then ps shows the app's processes, and /proc/1/root is the app's filesystem.
• Pass --profile=general, as the examples do. It gives the debug container the rights it needs to look at the app's processes.

# Interactive, the usual way: you land in a busybox shell next to the app's processes
kubectl -n shop debug -it api-0 --image=busybox:1.36 --target=app -c dbg --profile=general -- sh
#   ps                      # the app's processes, PID 1 is its entrypoint
#   ls /proc/1/root/etc     # the app container's filesystem
#   wget -qO- -T 3 http://localhost:8080/healthz

# Non-interactive: start it with a long-running command, then exec into it
kubectl -n shop debug api-0 --image=busybox:1.36 --target=app -c dbg --profile=general -- sleep 3600
kubectl -n shop exec api-0 -c dbg -- ps

# What it added to the Pod
kubectl -n shop get pod api-0 -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
kubectl -n shop get pod api-0 -o jsonpath='{.status.ephemeralContainerStatuses[?(@.name=="dbg")].state}{"\n"}'

Debugging a copy

Some containers crash before you can look inside. For those, make a copy of the Pod with one change, for example a command that only sleeps. The original Pod is not touched.

# Same Pod plus a busybox container, sharing one process namespace
kubectl -n shop debug api-0 --copy-to=api-debug --image=busybox:1.36 --share-processes -- sleep 3600

# Same Pod, but container app gets a different image
kubectl -n shop debug api-0 --copy-to=api-debug --set-image=app=busybox:1.36

# Same Pod, but container app runs a different command (it crashed on start)
kubectl -n shop debug api-0 --copy-to=api-debug --container=app -- sh -c 'sleep 3600'

kubectl -n shop get pod api-debug
kubectl -n shop delete pod api-debug        # the copy is yours to clean up

The copy is a plain Pod with a new name. No controller manages it, so nothing deletes it for you. It also drops the original's labels and probes, so no Service sends it traffic. To find why a container crashes, start a copy that only sleeps. Then kubectl exec into it and run the real program by hand, to see its error.

Debugging a node

kubectl debug node/cka-worker --image=busybox:1.36 starts a Pod on that node. Through it you can look at the node as if you had logged in:

• It shares the node's processes and network, so ps shows the kubelet.
• The node's whole filesystem is mounted at /host. chroot /host makes the shell use the node's own programs.
• The Pod is named node-debugger-cka-worker-<suffix> and lands in the current namespace. Pass -n so you know where it went.
• Exiting the shell does not delete it. It stays as Completed until you delete it.
• Add --profile=sysadmin only when you need a privileged container.

kubectl debug node/cka-worker -n shop -it --image=busybox:1.36 --profile=general -- sh
#   cat /host/etc/hostname
#   ps | grep kubelet                   # you see the node's processes
#   ls /host/var/log/pods
#   chroot /host                        # from here on, the node's own programs
kubectl -n shop get pods | grep node-debugger
kubectl -n shop delete pod node-debugger-cka-worker-x7k2p

# Non-interactive variant
kubectl debug node/cka-worker -n shop --image=busybox:1.36 --profile=general -- sleep 600
kubectl -n shop exec node-debugger-cka-worker-x7k2p -- cat /host/etc/hostname

crictl on the node

crictl talks to the container runtime directly, without the API server. You run it in a shell on the node. These three commands find a Pod's containers and run a command in one:

# exam: ssh <node>, then sudo -i    lab: docker exec -it cka-worker bash
crictl pods --name api-0                      # Pod sandbox ID and state
crictl ps --pod <pod-id>                      # running containers of that Pod
crictl exec <container-id> cat /etc/hostname  # only works if the image has the binary

crictl can also list stopped containers and read their logs. Control plane failures covers those commands.