Application failures

The usual ways an app breaks on the exam, each as symptom, cause and fix, and four broken scenarios to practice the method on.

Most troubleshooting tasks on the exam are app failures: one setting in a Pod is wrong, and the Pod shows it. There are only about ten patterns. For each one, this lesson gives the symptom, the cause and the fastest fix. You find them with the loop from the method lesson. The tasks at the end each break one thing, like the exam.

It also helps to know the order in which a Pod starts. The status tells you which step broke, and so where to look:

Image problems

• Symptom: ErrImagePull, then ImagePullBackOff, as the kubelet waits longer between tries. Or ContainerCreating with a Pulling image event that never ends.
• Cause: read the Failed event. It quotes the full image name and the registry's answer, as in the table below.
• Fix: correct the image in the Deployment, not in a Pod.

# Wrong tag: change the template, the rollout replaces the Pods
kubectl -n $ns set image deploy/web nginx=nginx:1.27

For a private registry, the Pod needs a login Secret listed under imagePullSecrets. ConfigMaps and Secrets shows how to create one.

CrashLoopBackOff

• Symptom: RESTARTS keeps climbing. STATUS switches between Running, Error and CrashLoopBackOff, with longer pauses each time.
• Cause: the process exits. Read kubectl logs first. Then read the Exit Code under Last State in describe; the table below decodes it.
• Common reasons: a misspelled command or argument, a missing environment variable or config file, or a service the app cannot reach. Or the command simply finishes: it exits 0, and the kubelet restarts it forever.

kubectl -n $ns get pod worker                          # RESTARTS 5 (2m ago)
kubectl -n $ns logs worker                             # e.g. sh: sleeep: not found
kubectl -n $ns logs worker --previous                  # when the container is Running again
kubectl -n $ns describe pod worker | grep -A6 'Last State'
kubectl -n $ns get pod worker -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'

Fix: correct the command, args or env. For a Deployment, change the template and the rollout recreates the Pods. A bare Pod must be recreated; see Bare Pods are mostly immutable below. If describe shows StartError with executable file not found in $PATH, the command could not even start. The fix is the same.

Pending

• Symptom: Pending, with NODE <none>.
• Cause: the scheduler found no node that fits. The FailedScheduling event says why each node was rejected. Four phrases cover the exam:

A PVC in a WaitForFirstConsumer StorageClass is not a failure: it waits for the Pod to be scheduled, then binds. And Pending with no event at all means nothing has looked at the Pod: the scheduler is down. Control plane failures covers that.

ContainerCreating that never ends

• Symptom: the Pod has a node, but stays in ContainerCreating.
• Cause: a FailedMount event. A ConfigMap or Secret volume points to an object that does not exist (secret "db-creds" not found), or a PVC is not Bound.
• Fix: create the missing object. Or mark the volume optional: true, if the question says the app can run without it.

After the fix, the Pod can take up to two minutes to start, because the kubelet retries the mount with a growing delay. If you are in a hurry, recreate the Pod.

OOMKilled

• Symptom: restarts with exit code 137, and Reason: OOMKilled under Last State.
• Cause: the container used more memory than its limit, resources.limits.memory.
• Fix: raise the limit; the question usually gives the number. On a Deployment, run kubectl set resources deploy/x -c name --limits=memory=512Mi. Recreate a bare Pod with the new limit.

Liveness probes that kill healthy containers

• Symptom: the app works, yet RESTARTS climbs. The events say Liveness probe failed: … connection refused or HTTP probe failed with statuscode: 404. Then: Container nginx failed liveness probe, will be restarted.
• Cause: the probe checks the wrong port or path. Or it starts checking before the app is up, because initialDelaySeconds is too short.
• Fix: fix the probe, not the app.

kubectl -n $ns get deploy api -o jsonpath='{.spec.template.spec.containers[0].livenessProbe}'; echo
kubectl -n $ns get deploy api -o jsonpath='{.spec.template.spec.containers[0].ports}'; echo
kubectl -n $ns patch deploy api --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/port","value":80}]'
kubectl -n $ns rollout status deploy/api --timeout=120s

Do not mix it up with a readiness probe. A failing readiness probe never restarts anything: the Pod shows READY 0/1 and gets no Service traffic. Probes, restart policies and disruption budgets explains the probe types.

Three more patterns, in short

• Pods Running and Ready, but the Service does not answer. That is a Service problem, not a Pod problem. Service and DNS failures covers it.
• Init:Error or Init:CrashLoopBackOff. An init container fails, so the app never starts. Read its logs with kubectl logs <pod> -c <init-name>. Pods, init containers and sidecars teaches init containers.
• A wrong ConfigMap key or Secret name. The Pod shows CreateContainerConfigError, and the event names the key. The second task of the method lesson is this case.

A typo in a variable name is sneakier. It gives no error at all: the app starts and misbehaves. kubectl exec <pod> -- env shows which variables the container really has.

The catalog on one screen