Application failures

The catalog of ways a workload breaks on the exam, each as symptom, cause and fix, and four broken scenarios to run the method on.

Most troubleshooting tasks on the exam are application failures: something in a Pod template is wrong and the Pod shows it. There are only about ten patterns. This lesson is the catalog; the method lesson taught you how to walk it. For each pattern you get the symptom you will see, the cause behind it, and the fastest fix. The tasks at the end each break exactly one thing and describe only the symptom, the way the exam does.

Image problems

Symptom: ErrImagePull, then ImagePullBackOff as the kubelet backs off between attempts; or ContainerCreating with a Pulling image event that never completes. Cause: read the Failed event, which quotes the full reference and the registry's answer. not found or manifest unknown is a wrong name or tag. unauthorized or pull access denied is a private registry without credentials. no such host or i/o timeout is a registry hostname that does not resolve from the node.

# Wrong tag: change the template, the rollout replaces the Pods
kubectl -n $ns set image deploy/web nginx=nginx:1.27

# Private registry: a docker-registry Secret, referenced by the Pod or by its ServiceAccount
kubectl -n $ns create secret docker-registry regcred --docker-server=registry.example.com --docker-username=u --docker-password=p
kubectl -n $ns patch deploy web -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}}}'
kubectl -n $ns patch serviceaccount default -p '{"imagePullSecrets":[{"name":"regcred"}]}'   # applies to new Pods using it

CrashLoopBackOff

Symptom: RESTARTS climbs, STATUS alternates between Running, Error and CrashLoopBackOff with growing pauses. Cause: the process exits. Read kubectl logs first (it serves the last crashed instance while the container is in backoff), then the exit code in describe under Last State. Common on the exam: a misspelled command or argument, a required environment variable or config file that is missing, a port already in use, a dependency the app cannot reach, or a command that simply finishes (the container's job is done, so it exits 0 and is restarted forever).

kubectl -n $ns get pod worker                          # RESTARTS 5 (2m ago)
kubectl -n $ns logs worker                             # e.g. sh: sleeep: not found
kubectl -n $ns logs worker --previous                  # when the container is Running again
kubectl -n $ns describe pod worker | grep -A6 'Last State'
kubectl -n $ns get pod worker -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'

A bare Pod's command, args and env are immutable, so the fix is export, edit, replace --force. For a Deployment, fix the template and let the rollout do the recreation. If the kubelet reports StartError with executable file not found in $PATH instead of a crash, the command itself cannot be executed; the fix is the same.

Pending

Symptom: Pending with NODE <none>. Cause: the FailedScheduling event lists, per node, why the scheduler rejected it. Four phrases cover the exam. Insufficient cpu or Insufficient memory: the request exceeds what any node has allocatable; lower the request (kubectl set resources) unless the question fixes it. didn't match Pod's node affinity/selector: the nodeSelector or nodeAffinity names a label no node carries; correct the selector, or label a node only if the question asks you to. had untolerated taint(s): expected for the control plane; a problem only when it applies to every node, in which case add a toleration or remove the taint, whichever the question wants. unbound immediate PersistentVolumeClaims: the PVC is not Bound; fix the claim or provide a PV. A PVC in a WaitForFirstConsumer StorageClass is not a failure: it binds once the Pod is scheduled.

Pending with no event at all is different: nothing has looked at the Pod. That is the scheduler being down, and the control-plane lesson covers it.

ContainerCreating that never ends

Symptom: scheduled (NODE is set) but stuck in ContainerCreating. Cause: a FailedMount event: a ConfigMap or Secret volume whose source does not exist (secret "db-creds" not found), or a PVC that is not Bound. Create the missing object, or mark the volume optional: true if the question allows the app to run without it. The kubelet retries mounts with a growing backoff that reaches about two minutes, so after you create the object the Pod can take up to two minutes to start; recreate it if you are in a hurry. FailedCreatePodSandBox is the CNI failing on that node, which is a node problem, not a workload problem.

OOMKilled

Symptom: restarts with exit code 137 and Reason: OOMKilled in Last State. Cause: the container exceeded resources.limits.memory. The fix is a higher limit, or a lower memory footprint; the question usually tells you the number. On a Deployment, kubectl set resources deploy/x -c name --limits=memory=512Mi rolls out new Pods. On a bare Pod resources are immutable through a normal update (Kubernetes 1.33+ can resize them in place through the resize subresource, but recreating the Pod is faster and certain on the exam). Also check for a LimitRange in the namespace: it sets a default limit on Pods that declare none, and that default can be the thing that is too small.

Probes that kill healthy containers

Symptom: the app works, yet RESTARTS climbs and the events say Liveness probe failed: … connection refused or HTTP probe failed with statuscode: 404, followed by Container nginx failed liveness probe, will be restarted. Cause: the probe's port or path is wrong, or its initialDelaySeconds is shorter than the app's startup. A wrong readiness probe does not restart anything; the Pod stays Running with READY 0/1 and the Service stops sending it traffic. A failing startup probe kills the container before liveness ever runs. Fix the probe, not the app.

kubectl -n $ns get deploy api -o jsonpath='{.spec.template.spec.containers[0].livenessProbe}'; echo
kubectl -n $ns get deploy api -o jsonpath='{.spec.template.spec.containers[0].ports}'; echo
kubectl -n $ns patch deploy api --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/port","value":80}]'
kubectl -n $ns rollout status deploy/api --timeout=120s

Services that find no Pods

Symptom: the Pods are Running and Ready but curl to the Service times out or is refused. Cause: the Service's selector does not match the Pod labels, so the EndpointSlice is empty; or targetPort does not match the container's port, so there are endpoints but nothing listens there. kubectl -n <ns> get endpointslices tells you which. The service-and-DNS lesson in this module works through both, plus DNS and NetworkPolicy; the point here is only that Running Pods with a dead Service are a Service problem, not a Pod problem.

Init containers and environment typos

Init containers: Init:Error or Init:CrashLoopBackOff means an init container fails and the app containers have not started. kubectl logs <pod> -c <init-name> shows why; the usual exam causes are a wrong command or a dependency (a Service, a DNS name) it waits for that does not exist. Environment: a typo in a ConfigMap key or Secret name gives CreateContainerConfigError and an Event that names the key. A typo in the variable name gives nothing at all; the app starts and misbehaves, and kubectl exec <pod> -- env is how you catch it. Mark a reference optional: true only when the question says the app can run without it.

The catalog on one screen