A troubleshooting method you can run under pressure

A fixed loop of commands that finds the cause of almost any broken workload in minutes, a map from every Pod status to the place its cause is written, and the habit of fixing with a command instead of an editor.

Troubleshooting is 30% of the exam and its tasks carry the biggest weights. Few of them are hard once you know where the cause is written down. What costs candidates the exam is wandering: running get pods three times, reading a 200-line YAML from the top, or opening kubectl edit before knowing what to change. This lesson gives you a loop to run on autopilot and a map from every Pod status to the single place that explains it.

The loop

1. 01Read the question twice. Note the namespace, the object names, and what fixed means: Running, Ready, reachable through a Service, a file written. That last part is what the grader checks.
2. 02Get, wide, across the namespace. kubectl -n <ns> get all -o wide. The STATUS, READY, RESTARTS and NODE columns usually tell you which lesson of this module you are in. Remember that all omits ConfigMaps, Secrets, PVCs, Ingresses and NetworkPolicies; add them when the question mentions them.
3. 03Describe the broken object and read from the bottom. Events first, then the container's State and Last State, then Conditions. The Events section names the missing key, the bad tag, the probe that fails, or the reason nothing could be scheduled.
4. 04Logs. kubectl logs, with -c for a named container and --previous when the container restarted and is running again. This is where a wrong command, a missing file or a refused connection shows up.
5. 05Exec. If the container runs, look from the inside: env, ls /etc/config, wget -qO- localhost:8080. It settles arguments about what the Pod actually sees.
6. 06Node. kubectl get nodes and describe node. If the node is NotReady or under pressure, the Pod is a symptom; go to the worker-node lesson.
7. 07Control plane. kubectl -n kube-system get pods. Pods that stay Pending with no events, Deployments that create nothing, or kubectl refusing to connect point at a component, not at the application.

ns=ts-method
kubectl -n $ns get all -o wide
kubectl -n $ns get events --sort-by=.lastTimestamp | tail -20   # or: kubectl -n $ns events --types=Warning
kubectl -n $ns describe pod <pod> | tail -25
kubectl -n $ns logs <pod> [-c <container>] [--previous]
kubectl -n $ns exec <pod> -- env
kubectl get nodes -o wide
kubectl -n kube-system get pods

What each status is telling you

Stop at the first step of the loop that names a cause; the exam breaks one thing per task, and hunting for a second problem is how a six-minute task becomes a fifteen-minute one. Most of the time the first step is enough, because the STATUS column already points at the place to look. It is a kubectl summary built from the Pod's phase, its container states and their reason fields (kubectl get pod -o jsonpath='{.status.containerStatuses[0].state}' shows the raw material). Read the table once; after that, each word sends you to the right place.

Reading describe from the bottom

kubectl describe pod prints the spec first and the useful part last. Read Events first: the From column tells you whether the scheduler or the kubelet complained, and the Message is usually the cause verbatim. Then look at each container's State, Last State, Reason and Exit Code. Then Conditions: PodScheduled false means the scheduler has not placed it; Initialized false means an init container has not finished; ContainersReady false means a probe or a crash.

Events are kept for one hour by default. A task that says the Pod broke yesterday may have no events left; the container's Last State and the logs do not expire, so go there. For a whole namespace at once, kubectl -n <ns> events --types=Warning shows only the problems, in time order.

Logs: current, previous, and gone

kubectl logs <pod> reads the current container. While a container sits in CrashLoopBackOff there is no current container, so the kubelet serves the output of the last one that ran; plain logs already shows the crash. kubectl logs --previous is for the other case: the container restarted and is Running again, so the crash output belongs to the previous instance. The kubelet keeps only one dead instance per container by default, so if --previous answers unable to retrieve container logs, that instance is already gone and the plain command is the best you have.

kubectl -n $ns logs <pod>                      # current container, or the last crashed one
kubectl -n $ns logs <pod> --previous           # the instance before the current one
kubectl -n $ns logs <pod> -c <container>       # multi-container and init containers
kubectl -n $ns logs <pod> --all-containers --tail=50
kubectl -n $ns logs deploy/<name>              # one Pod of the Deployment, chosen for you
kubectl -n $ns logs -l app=<label> --prefix    # every matching Pod, each line prefixed

Fix with a command, not an editor

kubectl edit is fine when you know the exact field, but it hides two traps: you can save a YAML with an indentation error and spend a minute reading the validation message, and for a bare Pod most fields are immutable, so the save is rejected anyway. Prefer a command that changes one field and nothing else. The grader reads the resulting object; it has no opinion about how it got there.

kubectl -n $ns set image deploy/web nginx=nginx:1.27
kubectl -n $ns set env deploy/web APP_MODE=production
kubectl -n $ns set resources deploy/web -c nginx --limits=memory=256Mi --requests=memory=128Mi
kubectl -n $ns patch deploy web -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"cka-worker"}}}}}'
kubectl -n $ns patch deploy web --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/port","value":80}]'
kubectl -n $ns scale deploy web --replicas=3
kubectl -n $ns rollout undo deploy/web

# A bare Pod with a wrong command, env, volume or probe cannot be patched in place: export, fix, recreate.
kubectl -n $ns get pod app -o yaml > app.yaml
vim app.yaml
kubectl -n $ns replace --force -f app.yaml      # add --grace-period=0 if the old Pod takes 30s to die

On a running Pod you may change only the image, activeDeadlineSeconds, terminationGracePeriodSeconds and add tolerations. Everything else (command, args, env, volumes, probes, nodeSelector, resources) needs a new Pod. When the Pod belongs to a Deployment, change the Deployment's template instead and let the rollout replace the Pods; fixing a Pod the ReplicaSet owns is undone the moment it is recreated.

Worked example: a Deployment that never comes up

The question says: Deployment report in namespace ts-method has no available Pods. Fix it. Steps 2 and 3 of the loop, trimmed to the lines that matter:

$ kubectl -n ts-method get deploy,pods -o wide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/report   0/1     1            0           2m
NAME                          READY   STATUS    RESTARTS   AGE   IP       NODE
pod/report-677544647f-9m7nq   0/1     Pending   0          2m    <none>   <none>

$ kubectl -n ts-method describe pod report-677544647f-9m7nq | tail -2
  Type     Reason            Age   From               Message
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient memory. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

NODE is <none>, so the scheduler never placed it, and the event says why, node by node: the control plane rejects the Pod because of its taint, which is normal; the two workers reject it for memory. The request is bigger than any worker can offer. Confirm, fix with one command, and prove the end state:

kubectl -n ts-method get deploy report -o jsonpath='{.spec.template.spec.containers[0].resources}'
# {"requests":{"memory":"64Gi"}}
kubectl describe node cka-worker | grep -A6 '^Allocatable'

kubectl -n ts-method set resources deploy report --requests=memory=64Mi
kubectl -n ts-method rollout status deploy/report --timeout=60s
kubectl -n ts-method get pods -o wide        # Running, 1/1, a NODE name

Three commands of diagnosis, one of repair, one of proof. If the question had said the application needs 2Gi, the right fix would have been the same command with that value; the question defines what correct means, the events only tell you what is wrong.