kubectl debug, ephemeral containers and crictl

Getting a shell where there is none: ephemeral debug containers, debugging a copy of a Pod, node debugging Pods with the host at /host, and crictl on the node.

35 min 2 hands-on tasksTroubleshooting · 30% of the exam

What you will be able to do

Attach an ephemeral container to a running Pod, share the target container's processes and inspect it without restarting the Pod.
Debug a crashing Pod through a modified copy and clean the copy up afterwards.
Open a node debugging Pod, read the host's files and processes through it, and delete it when you are done.
List, inspect, read and exec into containers with crictl when the API gives you nothing useful.

Production images are increasingly distroless: no shell, no ps, no wget. kubectl exec into them fails with executable file not found, and on a node you may need tools the node image does not ship. kubectl debug solves both, and the exam expects you to drive it without fumbling, including the cleanup it leaves behind. When the API itself is the problem, crictl on the node is the fallback, and it is the subject of more than one control plane and node task.

Tool	What it gives you	Use it when
`kubectl exec`	A process inside an existing container, with that image's tools	The image has a shell and the container is running
`kubectl debug POD --image=… --target=C`	An ephemeral container in the same Pod, sharing C's process namespace	No shell in the image; the Pod must not restart
`kubectl debug POD --copy-to=NEW …`	A new Pod built from the original, with a changed image or command	The container crashes on start, or you need a different image
`kubectl debug node/NODE --image=…`	A Pod on that node in the host's PID, network and IPC namespaces, host root at `/host`	Node files, processes or network from inside the cluster
`crictl` on the node	The runtime's view: Pods, containers, logs, exec	The kubelet or API server is unhealthy, or you need the container's PID

Ephemeral containers

An ephemeral container is added to a running Pod through its ephemeralcontainers subresource, so the Pod is not recreated. It cannot have ports, probes or resources, it is never restarted, and it cannot be changed or removed: it stays in spec.ephemeralContainers until the Pod is deleted. --target=app puts it in the process namespace of container app, so ps shows the app's processes and /proc/1/root is the app's filesystem. The default general profile adds SYS_PTRACE to the debug container so that works. Pass --profile=general explicitly: older kubectl clients default to legacy.

Attach, use and inspect an ephemeral container· Terminal

# Interactive, the usual way: you land in a busybox shell next to the app's processes
kubectl -n shop debug -it api-0 --image=busybox:1.36 --target=app -c dbg --profile=general -- sh
#   ps                      # the app's processes, PID 1 is its entrypoint
#   ls /proc/1/root/etc     # the app container's filesystem
#   wget -qO- -T 3 http://localhost:8080/healthz

# Non-interactive: start it with a long-running command, then exec into it
kubectl -n shop debug api-0 --image=busybox:1.36 --target=app -c dbg --profile=general -- sleep 3600
kubectl -n shop exec api-0 -c dbg -- ps
kubectl -n shop attach -it api-0 -c dbg     # only if it runs a shell

# What it added to the Pod
kubectl -n shop get pod api-0 -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
kubectl -n shop get pod api-0 -o jsonpath='{.status.ephemeralContainerStatuses[?(@.name=="dbg")].state}{"\n"}'

Debugging a copy

--copy-to: a new Pod you are free to break· Terminal

# Same Pod plus a busybox container, sharing one process namespace
kubectl -n shop debug api-0 --copy-to=api-debug --image=busybox:1.36 --share-processes -- sleep 3600

# Same Pod, but container app gets a different image
kubectl -n shop debug api-0 --copy-to=api-debug --set-image=app=busybox:1.36

# Same Pod, but container app runs a different command (it crashed on start)
kubectl -n shop debug api-0 --copy-to=api-debug --container=app -- sh -c 'sleep 3600'

kubectl -n shop get pod api-debug
kubectl -n shop delete pod api-debug        # the copy is yours to clean up

The copy is an ordinary Pod with a new name, not managed by any controller. With the general profile it loses the original's labels (so Services do not send it traffic) and its probes, unless you pass --keep-labels or --keep-liveness; --same-node schedules it next to the original. Use it for a container that crashes before you can exec into it: start a copy whose command is sleep, exec in and run the real entrypoint by hand to see the error.

Debugging a node

kubectl debug node/cka-worker --image=busybox:1.36 creates a Pod named node-debugger-cka-worker-<suffix> in the current namespace, pinned to that node, with hostPID, hostNetwork and hostIPC, and the node's root filesystem mounted at /host. chroot /host puts you in the node's own userland. Exiting the shell does not remove anything: the Pod stays in Completed and you must delete it. Use -n so you know where it went, and add --profile=sysadmin only when you need a privileged container.

Node debugging Pod, then crictl on the node· Terminal

kubectl debug node/cka-worker -n shop -it --image=busybox:1.36 --profile=general -- sh
#   cat /host/etc/hostname
#   ps | grep kubelet                   # host PID namespace
#   ls /host/var/log/pods
#   chroot /host                        # node binaries and config from here on
kubectl -n shop get pods | grep node-debugger
kubectl -n shop delete pod node-debugger-cka-worker-x7k2p

# Non-interactive variant
kubectl debug node/cka-worker -n shop --image=busybox:1.36 --profile=general -- sleep 600
kubectl -n shop exec node-debugger-cka-worker-x7k2p -- cat /host/etc/hostname

# crictl, from a node shell (exam: ssh <node>, then sudo -i; lab: docker exec -it cka-worker bash)
crictl pods --name api-0                      # Pod sandbox ID and state
crictl ps --pod <pod-id>                      # running containers of that Pod
crictl ps -a --name app                       # include exited instances
crictl inspect <container-id> | grep -m1 '"pid"'
crictl logs --tail 20 <container-id>
crictl exec <container-id> cat /etc/hostname  # only works if the image has the binary
crictl images | grep busybox

In the official docs· open on exam day too

Debug Running Podssearch: debug running pods ephemeral
Ephemeral Containerssearch: ephemeral containers
Debugging Kubernetes Nodes With Kubectlsearch: kubectl debug node
Debugging Kubernetes nodes with crictlsearch: crictl

Hands-on task 1 of 2ts-debug-1

~6 min5% weightTroubleshooting

Debug a Pod that has no shell

Pod payroll in namespace tsx-debug runs an image without a shell, so kubectl exec into it fails. Without restarting or recreating the Pod, add a debug container named dbg using image busybox:1.36 that shares the process namespace of container app and keeps running with the command sleep 3600. Then, from dbg, write the command line of process ID 1 to /tmp/cka/payroll-pid1.txt.

Cluster:kubectl config use-context kind-cka

1
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-debug-1/setup | bash
2
Solve it in your terminal
Use kubectl and the Kubernetes docs, like on exam day. Save any answer files under /tmp/cka/.
3
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-debug-1/verify | bash
Got FAIL? Read which check failed, fix it, run it again.
4
Clean up (optional)
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-debug-1/cleanup | bash

Hands-on task 2 of 2ts-debug-2

~5 min4% weightTroubleshooting

Read a node file through a node debugging Pod

Using kubectl debug, start a node debugging Pod on node cka-worker in namespace tsx-node with image busybox:1.36. Through that Pod, read the node's file /etc/hostname and write its content to /tmp/cka/worker-hostname.txt. Delete the debugging Pod when you are done.

Cluster:kubectl config use-context kind-cka

1
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-debug-2/setup | bash
2
Solve it in your terminal
Use kubectl and the Kubernetes docs, like on exam day. Save any answer files under /tmp/cka/.
3
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-debug-2/verify | bash
Got FAIL? Read which check failed, fix it, run it again.
4
Clean up (optional)
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-debug-2/cleanup | bash

Finished this lesson?

0 of 2 tasks passed. Your readiness counts tasks, so run them before you move on.

Complete and continue