A troubleshooting method you can run under pressure

One fixed loop of commands that finds the cause of almost any broken app in minutes, a map from each Pod status to the place its cause is written, and the habit of fixing with one command instead of an editor.

Troubleshooting is 30% of the exam, the biggest part. Most of its tasks are not hard once you know where the cause is written. What costs time is wandering: running get pods three times, or opening kubectl edit before you know what to change. This lesson gives you one loop to run every time, and a map from each Pod status to the place that explains it.

The loop

Run the same steps, in the same order, on every broken object. Stop at the first step that names the cause.

1. 1.Read the question twice. Note the namespace, the object names, and what fixed means: Running, Ready, reachable through a Service, or a file written. That end state is what the grader checks.
2. 2.get. Run kubectl -n <ns> get all -o wide and read the STATUS, READY, RESTARTS and NODE columns. Note that all skips ConfigMaps, Secrets, PVCs, Ingresses and NetworkPolicies. Ask for those by name when the question mentions them.
3. 3.describe, read from the bottom. The Events at the end usually name the cause: a missing key, a bad image tag, a failing probe, or why no node fits.
4. 4.logs. What the app itself printed. A wrong command, a missing file or a refused connection shows up here. Logs, events and container output streams covers --previous and the other options.
5. 5.exec. If the container runs, look from the inside: env, ls /etc/config, wget -qO- localhost:8080. This shows what the Pod really sees.
6. 6.The node. Run kubectl get nodes. If the node is NotReady, the Pod is only a symptom: go to Worker node failures.
7. 7.The control plane. Run kubectl -n kube-system get pods. Pods Pending with no events, Deployments that create nothing, or kubectl refusing to connect point at a cluster part, not at the app. See Control plane failures.

ns=ts-method
kubectl -n $ns get all -o wide
kubectl -n $ns get events --sort-by=.lastTimestamp | tail -20   # or: kubectl -n $ns events --types=Warning
kubectl -n $ns describe pod <pod> | tail -25
kubectl -n $ns logs <pod> [-c <container>] [--previous]
kubectl -n $ns exec <pod> -- env
kubectl get nodes -o wide
kubectl -n kube-system get pods

What each status tells you

Stop at the first step that names a cause. The exam breaks one thing per task, so hunting for a second problem only costs time. Often step 2 is enough, because the STATUS column already says where to look:

The next lesson, Application failures, explains each of these statuses with an example.

Reading describe from the bottom

kubectl describe pod prints the settings first and the useful part last. So read it from the bottom up:

1. 1.Events. The From column says who complained: the scheduler or the kubelet. The Message is usually the cause, in plain words.
2. 2.State and Last State of each container, with its Reason and Exit Code.
3. 3.Conditions. PodScheduled False: no node yet. Initialized False: an init container has not finished. ContainersReady False: a probe fails or a container crashed.

Events are kept for only one hour. If the Pod broke yesterday, its events may be gone, but Last State and the logs are still there. To see only the problems in a whole namespace, run kubectl -n <ns> events --types=Warning.

Fix with a command, not an editor

kubectl edit is fine when you know the exact field. But it has two traps:

• You can save YAML with a wrong indent, then lose a minute reading the error.
• On a bare Pod most fields are locked, so the save is refused anyway.

So prefer a command that changes one field and nothing else. The grader reads only the result. It does not care how you got there.

kubectl -n $ns set image deploy/web nginx=nginx:1.27
kubectl -n $ns set resources deploy/web -c nginx --limits=memory=256Mi --requests=memory=128Mi
kubectl -n $ns patch deploy web -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"cka-worker"}}}}}'
kubectl -n $ns patch deploy web --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/port","value":80}]'

# A bare Pod with a wrong command, env, volume or probe cannot be patched in place: export, fix, recreate.
kubectl -n $ns get pod app -o yaml > app.yaml
vim app.yaml
kubectl -n $ns replace --force -f app.yaml      # add --grace-period=0 if the old Pod takes 30s to die

Most fields of a running Pod are locked. You can change its image, but a normal edit cannot change the command, args, env, volumes, probes, nodeSelector or resources. For those, save the Pod to a file, fix it, and recreate it with replace --force, as the last three lines above do.

When the Pod belongs to a Deployment, never fix the Pod. Change the Deployment's template, and the rollout replaces the Pods for you:

Worked example: a Deployment that never comes up

The question says: Deployment report in namespace ts-method has no available Pods. Fix it. Here are steps 2 and 3 of the loop, cut to the lines that matter:

$ kubectl -n ts-method get deploy,pods -o wide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/report   0/1     1            0           2m
NAME                          READY   STATUS    RESTARTS   AGE   IP       NODE
pod/report-677544647f-9m7nq   0/1     Pending   0          2m    <none>   <none>

$ kubectl -n ts-method describe pod report-677544647f-9m7nq | tail -2
  Type     Reason            Age   From               Message
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient memory. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

NODE is <none>, so the scheduler never placed the Pod. The event says why, node by node:

• The control plane said no because of its taint. That is normal: the taint keeps ordinary Pods off it.
• The two workers said no because of memory: Insufficient memory.

So the Pod asks for more memory than any worker has. Check the request, fix it with one command, and prove the result:

kubectl -n ts-method get deploy report -o jsonpath='{.spec.template.spec.containers[0].resources}'
# {"requests":{"memory":"64Gi"}}
kubectl describe node cka-worker | grep -A6 '^Allocatable'

kubectl -n ts-method set resources deploy report --requests=memory=64Mi
kubectl -n ts-method rollout status deploy/report --timeout=60s
kubectl -n ts-method get pods -o wide        # Running, 1/1, a NODE name

That is three commands to find the cause, one to fix it and one to prove it. Note who decides the right value: the question. If it had said the app needs 2Gi, you would set 2Gi. The events only tell you what is wrong.