Autoscaling with the HPA

How the HorizontalPodAutoscaler turns metrics into a replica count, what it needs from metrics-server and from your Pod specs, how to shape scaling with behavior, and where vertical and cluster autoscaling fit.

This lesson covers Configure workload autoscaling. On the exam that means a HorizontalPodAutoscaler with exact numbers, sometimes with a behavior block that kubectl autoscale cannot generate. The grader reads the HPA object; it does not generate load and wait for scaling. What it can also read is whether your Pods request the resource the HPA targets, which is the step most people miss.

How the HPA decides

The HPA controller runs inside kube-controller-manager and reconciles every HPA every 15 seconds. For a CPU or memory target it reads Pod usage from the metrics.k8s.io API, which metrics-server serves (installed on this lab), divides by each Pod's request to get utilization, averages across Pods, and computes desired = ceil(current replicas × current value / target value). If the ratio is within 10% of 1 it does nothing. The result is clamped to minReplicas and maxReplicas and written to the target's scale subresource, so any Deployment, StatefulSet or ReplicaSet works.

Example: 4 replicas averaging 90% CPU against a 60% target gives ceil(4 × 90 / 60) = 6. Because utilization is measured against requests, a container without a CPU request makes the whole calculation impossible, and the HPA reports missing request for cpu. Once an HPA owns a Deployment, leave spec.replicas out of manifests you re-apply, or every apply fights the autoscaler.

kubectl get apiservice v1beta1.metrics.k8s.io           # Available True means metrics-server answers
kubectl -n wl-hpa top pods                                 # usage per Pod
kubectl -n wl-hpa get deploy api -o jsonpath='{.spec.template.spec.containers[*].resources}'; echo
kubectl -n wl-hpa set resources deploy/api -c nginx --requests=cpu=100m   # add the missing request (starts a rollout)

Create one

# Current kubectl (the exam's v1.37 included)
kubectl -n wl-hpa autoscale deployment api --min=2 --max=8 --cpu=60%
# Older kubectl only knows this form; newer releases still accept it with a deprecation warning
kubectl -n wl-hpa autoscale deployment api --min=2 --max=8 --cpu-percent=60

kubectl -n wl-hpa autoscale deployment api --min=2 --max=8 --cpu=60% --dry-run=client -o yaml > hpa.yaml

Recent kubectl replaced --cpu-percent with --cpu, which takes either a percentage (60%, a Utilization target) or a quantity (500m, an AverageValue target), and added --memory in the same style. If a flag is rejected, kubectl autoscale --help shows which form your kubectl knows. Anything beyond one CPU or memory target and min/max needs YAML.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker
  namespace: wl-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: worker
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization          # percent of the request
          averageUtilization: 75
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageValue         # absolute usage per Pod
          averageValue: 100Mi
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60 # act on the highest recommendation of the last 60s
      policies:
        - type: Pods
          value: 1                   # remove at most 1 Pod ...
          periodSeconds: 30          # ... per 30 seconds
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100                 # at most double per 15 seconds
          periodSeconds: 15

With several metrics the HPA computes a replica count for each and uses the largest. Besides Resource there are ContainerResource (one named container's usage, ignoring sidecars), Pods and Object (custom metrics) and External; those need a metrics adapter such as the Prometheus adapter. When a direction has several policies, selectPolicy: Max (the default) picks the one allowing the biggest change, Min the smallest, and Disabled turns that direction off.

Reading an HPA

kubectl -n wl-hpa get hpa
# NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS
# api    Deployment/api   cpu: <unknown>/60%   2         8         2        <- no metrics or no requests
kubectl -n wl-hpa describe hpa api | sed -n '/Conditions/,$p'
#   ScalingActive  False  FailedGetResourceMetric  ... missing request for cpu in container nginx ...

Three conditions tell the story. AbleToScale says whether the HPA can read and write the target's scale. ScalingActive is False when no metric can be computed: no metrics API, Pods too new to have metrics, or missing requests. ScalingLimited is True when the computed count was clamped by min, max or a policy. A fresh HPA shows <unknown> for up to a minute while metrics-server collects its first samples; if it persists, read the condition message.

Vertical Pod Autoscaler and Cluster Autoscaler

The Vertical Pod Autoscaler is a separate project from the Kubernetes autoscaler repository: a VerticalPodAutoscaler CRD plus a recommender, an updater and an admission webhook. It changes requests instead of replica counts, from recommendation-only mode up to modes that apply new values by recreating or resizing Pods. Do not point it and an HPA at the same CPU or memory metric; they will fight. Kubernetes itself can now resize a running Pod's CPU and memory in place through the Pod resize subresource, which VPA can use. The Cluster Autoscaler (or Karpenter) adds nodes when Pods are Pending for lack of resources and removes underused nodes after draining them, respecting PodDisruptionBudgets. Both need cloud node groups, so the exam treats them as concepts and kind has neither.