Probes, restart policies and disruption budgets

Liveness, readiness and startup probes and what each one triggers, how readiness gates Service traffic and rollouts, how restartPolicy shapes recovery, and how a PodDisruptionBudget limits voluntary evictions.

This lesson is the core of Understand the primitives used to create robust, self-healing, application deployments. Controllers replace Pods that disappear; probes are how the kubelet notices a container that is still there but no longer useful. On the exam you will add probes to an existing workload, fix one that points at the wrong place, or protect a workload from a node drain with a PodDisruptionBudget.

Three probes, three different questions

      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
              name: http
          startupProbe:
            httpGet:
              path: /
              port: http            # a named port works anywhere a port number does
            periodSeconds: 5
            failureThreshold: 24    # up to 24 x 5 = 120s to start
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: 80
            periodSeconds: 10
            failureThreshold: 3
        # Other handlers:
        #   exec:     { command: ["cat", "/tmp/healthy"] }   # exit code 0 = success
        #   grpc:     { port: 9000 }                         # the app must implement the gRPC health service

Use a startup probe rather than a long initialDelaySeconds on liveness: it gives slow starters time without slowing down detection of a hang later. Keep liveness checks local. A liveness probe that calls a database restarts every replica at once when the database blips, turning one outage into two.

Readiness is what Services and rollouts trust

A Pod that fails readiness stays Running with READY 0/1, and its address is published in the Service's EndpointSlices with ready: false, so no traffic is sent to it. Deployments count only Ready Pods (held Ready for minReadySeconds) as available, so a rollout whose new Pods never become Ready stalls instead of replacing healthy Pods. A wrong readiness port is one of the most common it is running but nothing works scenarios.

kubectl -n wl-probe get pods -l app=catalog                    # Running, READY 0/1
kubectl -n wl-probe describe pod -l app=catalog | grep -E 'Readiness|Liveness|Unhealthy' | head
#   Warning  Unhealthy  ...  Readiness probe failed: Get "http://10.244.x.x:8080/": dial tcp ... connection refused
kubectl -n wl-probe get endpointslices -l kubernetes.io/service-name=catalog -o yaml | grep -A1 conditions
kubectl -n wl-probe get deploy catalog -o jsonpath='{.spec.template.spec.containers[0].readinessProbe}'; echo

restartPolicy and CrashLoopBackOff

restartPolicy belongs to the Pod and applies to every container: Always (the only value Deployments, StatefulSets and DaemonSets accept), OnFailure or Never (Jobs and bare Pods). After a liveness kill or a crash, the kubelet restarts the container with an exponentially growing delay; the Pod shows CrashLoopBackOff while it waits, and the delay resets once the container has run cleanly for a while. RESTARTS in kubectl get pods and Last State in kubectl describe pod tell you whether a probe or the process itself is to blame; kubectl logs --previous shows the output of the instance that died.

PodDisruptionBudgets

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cart-pdb
  namespace: wl-probe
spec:
  minAvailable: 2          # or maxUnavailable: 1; never both. Integers or percentages.
  selector:
    matchLabels:
      app: cart
# Imperative: kubectl -n wl-probe create pdb cart-pdb --selector=app=cart --min-available=2

A PDB limits voluntary disruptions that go through the Eviction API: kubectl drain, cluster autoscaler scale-downs, some upgrade tools. When an eviction would push the matching Pods below the budget, the API refuses it and kubectl drain keeps retrying until the budget allows it or its --timeout expires. It does nothing against node crashes, and kubectl delete pod bypasses it entirely. Read kubectl get pdb: ALLOWED DISRUPTIONS of 0 means a drain of a node running one of those Pods will block. A budget of minAvailable equal to the replica count therefore blocks drains forever.