StatefulSets and volumeClaimTemplates

How a StatefulSet gives each replica a stable name, a DNS entry and its own PersistentVolumeClaim, and what happens to those claims when you scale or delete it.

StatefulSets are where the three storage competencies meet: each replica gets a claim through a volumeClaimTemplate, the claim is satisfied by dynamic provisioning from a StorageClass, and the PV and PVC life cycle you learned in the last two lessons plays out once per replica. The exam uses them for "create a StatefulSet with N replicas and persistent storage" and for "scale it and check the volumes". Both are fast if you know the naming rules, because the grader checks names.

StatefulSet versus Deployment

Stable identity: the headless Service

A StatefulSet names a Service in spec.serviceName. Make it headless (clusterIP: None) and the DNS for each Pod becomes <pod>.<service>.<namespace>.svc.cluster.local, for example blog-0.blog.st-sts.svc.cluster.local, because the StatefulSet sets each Pod's hostname to its name and subdomain to the Service name. The Service itself does not need to exist for the Pods to start, only for those names to resolve, and its selector must match the Pod labels like any other Service. Clients that need "the primary" connect to blog-0 by name; clients that want any replica use the Service name, which returns every ready Pod IP.

apiVersion: v1
kind: Service
metadata:
  name: blog
  namespace: st-sts
spec:
  clusterIP: None
  selector:
    app: blog
  ports:
    - name: http
      port: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: blog
  namespace: st-sts
spec:
  serviceName: blog
  replicas: 2
  selector:
    matchLabels:
      app: blog
  template:
    metadata:
      labels:
        app: blog
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
              name: http
          volumeMounts:
            - name: content
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: content
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: standard
        resources:
          requests:
            storage: 100Mi

Ordered creation and podManagementPolicy

With the default podManagementPolicy: OrderedReady, Pod 0 is created and must be Running and Ready before Pod 1 starts; scaling down removes the highest ordinal first and waits for it to terminate before the next. That is what a database that elects a leader wants. Parallel launches and terminates all Pods at once and only affects scaling, not updates. Either way a Pod that is deleted comes back with the same name, the same claim and the same DNS entry, which is the whole promise.

volumeClaimTemplates and claim names

Each entry in volumeClaimTemplates is a PVC spec without a namespace. For every Pod the controller creates one claim per template named <template>-<statefulset>-<ordinal>: template content on StatefulSet blog gives content-blog-0, content-blog-1, and so on. The Pod template's volumeMounts refer to the template name as if it were a volume, so there is no volumes: entry for it. The claims are ordinary PVCs: the StorageClass provisions them, kubectl get pvc lists them, and they are bound to the Pod with that ordinal forever. volumeClaimTemplates cannot be changed on an existing StatefulSet; only replicas, ordinals, template, updateStrategy, minReadySeconds and persistentVolumeClaimRetentionPolicy are mutable. To change the template, delete the StatefulSet (with --cascade=orphan if the Pods must keep running) and recreate it.

kubectl apply -f blog.yaml
kubectl -n st-sts rollout status sts/blog --timeout=180s
kubectl -n st-sts get pods -l app=blog -o wide          # blog-0, blog-1
kubectl -n st-sts get pvc                                # content-blog-0, content-blog-1, both Bound
kubectl -n st-sts scale sts blog --replicas=3           # creates content-blog-2 then blog-2
kubectl -n st-sts get sts blog -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}'

What happens to the claims when you scale down or delete

By default, nothing. Scale from 3 to 2 and content-blog-2 stays Bound with its data, so scaling back up gives blog-2 its old disk back. Delete the StatefulSet and every claim survives, which is why a cluster accumulates orphaned claims after test runs. spec.persistentVolumeClaimRetentionPolicy (stable since v1.32) changes that per case: whenScaled and whenDeleted each take Retain (the default) or Delete. With whenScaled: Delete the claim for a removed ordinal is deleted once its Pod is gone; with whenDeleted: Delete all claims go with the StatefulSet. Whether the PV and the data then disappear is still the PV's reclaim policy, which the StorageClass set. The field is mutable, so you can add it to a running StatefulSet with a patch.

kubectl -n st-sts patch sts blog -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
kubectl -n st-sts scale sts blog --replicas=2
kubectl -n st-sts get pvc                                # content-blog-2 is gone after blog-2 terminates
# Cleaning up by hand when the policy is Retain:
kubectl -n st-sts delete sts blog
kubectl -n st-sts delete pvc -l app=blog                 # claims inherit the selector labels

updateStrategy and partition

updateStrategy.type is RollingUpdate by default: after a change to spec.template, Pods are replaced one at a time from the highest ordinal to the lowest, each waiting for the previous one to be Ready. OnDelete changes nothing until you delete a Pod yourself, which is how you control exactly when each replica moves. rollingUpdate.partition: N is the canary knob: only Pods with ordinal N or higher are updated, the rest keep the old template until you lower the partition to 0. Because the claims are bound to ordinals, none of this touches storage; blog-1 comes back on the new image with content-blog-1 still mounted.

kubectl -n st-sts patch sts blog -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":1}}}}'
kubectl -n st-sts set image sts/blog web=nginx:1.26     # only blog-1 (ordinal >= 1) is replaced
kubectl -n st-sts get pods -l app=blog -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
kubectl -n st-sts patch sts blog -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
kubectl -n st-sts rollout status sts/blog --timeout=180s