Probes, restart policies and disruption budgets

The three probes and what each one does when it fails, how readiness decides which Pods get traffic, and how a PodDisruptionBudget protects an app during a node drain.

Kubernetes can only fix what it notices. If a container crashes, the kubelet sees the process exit and restarts it. But what if your Django app is still running, yet stuck and answering nothing? A probe is a regular health check that the kubelet runs against a container, so it can notice that too.

On the exam, you may add probes to a workload, or fix a probe that checks the wrong place. You may also protect a workload from a node drain with a PodDisruptionBudget.

Three probes, three different questions

Example: your Django app takes 60 seconds to load. A startup probe gives it that time. After that, the readiness probe keeps it out of the Service while it cannot serve, and the liveness probe restarts it if it hangs.

Each probe uses one kind of check, called a handler. The common ones are httpGet (send an HTTP request), tcpSocket (open a port) and exec (run a command inside the container).

      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
              name: http
          startupProbe:
            httpGet:
              path: /
              port: http            # a named port works anywhere a port number does
            periodSeconds: 5
            failureThreshold: 24    # up to 24 x 5 = 120s to start
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: 80
            periodSeconds: 10
            failureThreshold: 3
        # Other handlers:
        #   exec:     { command: ["cat", "/tmp/healthy"] }   # exit code 0 = success

These timing fields work the same on all three probes:

• For a slow starter, use a startup probe, not a long initialDelaySeconds on liveness. The app gets time to start, and a later hang is still caught quickly.
• Keep liveness checks local. Say a liveness probe calls the database. When the database has a short outage, every replica fails its check and restarts at once: one outage becomes two.

Readiness is what Services and rollouts trust

A Pod that fails readiness stays Running, but shows READY 0/1. Two things follow:

• Services skip it. Its address stays in the Service's EndpointSlices, marked ready: false, so no traffic is sent to it.
• Rollouts wait for it. A Deployment counts only Ready Pods as available. So a rollout whose new Pods never become Ready stops, instead of replacing healthy Pods.

A wrong readiness port is one of the most common cases of it is running but nothing works. Trace it like this:

kubectl -n wl-probe get pods -l app=catalog                    # Running, READY 0/1
kubectl -n wl-probe describe pod -l app=catalog | grep -E 'Readiness|Liveness|Unhealthy' | head
#   Warning  Unhealthy  ...  Readiness probe failed: Get "http://10.244.x.x:8080/": dial tcp ... connection refused
kubectl -n wl-probe get endpointslices -l kubernetes.io/service-name=catalog -o yaml | grep -A1 conditions
kubectl -n wl-probe get deploy catalog -o jsonpath='{.spec.template.spec.containers[0].readinessProbe}'; echo

restartPolicy and CrashLoopBackOff

restartPolicy is set once per Pod and covers every container in it. Deployments, StatefulSets and DaemonSets accept only Always. Jobs use OnFailure or Never.

After a crash or a liveness kill, the kubelet restarts the container, and it waits a little longer each time. While it waits, the Pod shows CrashLoopBackOff. Finding out why is covered in Application failures.

PodDisruptionBudgets

Sometimes Pods are removed on purpose, for example when you drain a node to upgrade it. A PodDisruptionBudget (PDB) makes sure enough Pods of an app stay up while that happens. Example: your shop runs 3 cart Pods. A PDB with minAvailable: 2 lets a drain take away only one of them at a time.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cart-pdb
  namespace: wl-probe
spec:
  minAvailable: 2          # or maxUnavailable: 1; never both. Integers or percentages.
  selector:
    matchLabels:
      app: cart
# Imperative: kubectl -n wl-probe create pdb cart-pdb --selector=app=cart --min-available=2

What a PDB protects against

• It limits voluntary disruptions that go through the Eviction API: kubectl drain, a cluster autoscaler removing a node, and some upgrade tools.
• It does nothing when a node crashes.
• kubectl delete pod skips it completely.

How to read it. Run kubectl get pdb and look at ALLOWED DISRUPTIONS:

• 1 or more: a drain may evict that many of these Pods right now.
• 0: a drain of a node that runs one of these Pods will block.
• The trap: minAvailable equal to the replica count always leaves 0, so it blocks every drain.