Monitoring resource usage

How metrics-server feeds kubectl top, what the CPU and memory columns actually measure, and how to compare live usage with requests, limits and node capacity.

The competency Monitor cluster and application resource usage is tested with short, file-answer tasks: which Pod in this namespace uses the most CPU, which node has the most memory, which container is the heavy one. The commands take seconds. The points are lost on reading the wrong column, sorting the wrong thing, confusing usage with requests, or writing pod/name when the grader wanted name.

kubectl top reads the resource metrics API, metrics.k8s.io/v1beta1. metrics-server serves it: every --metric-resolution (15 s here) it scrapes each kubelet's resource endpoint, keeps only the latest sample in memory and answers through the API server's aggregation layer. The same API feeds the HorizontalPodAutoscaler. There is no history and no per-request detail: a Pod appears a scrape or two after it starts, and its numbers vanish when it is gone. If metrics-server is missing or unhealthy, kubectl top answers Metrics API not available.

kubectl top nodes                                   # usage per node; % is of allocatable
kubectl top nodes --sort-by=memory                  # or --sort-by=cpu; highest first
kubectl top nodes --show-capacity                   # % of capacity instead of allocatable
kubectl top pods -n shop                            # one namespace
kubectl top pods -A --sort-by=cpu                   # every namespace, heaviest first
kubectl top pods -n shop --containers               # one line per container
kubectl top pods -n shop -l app=web --sort-by=memory
kubectl top pods -n shop --sum                      # adds a total row
kubectl top pod web-7d9c -n shop --containers       # a single Pod (one name only)

# The answer as a bare name, without a header
kubectl top pods -n shop --sort-by=cpu --no-headers | awk 'NR==1 {print $1}' > /tmp/cka/busiest.txt

# Is the pipeline healthy?
kubectl get apiservice v1beta1.metrics.k8s.io       # AVAILABLE must be True
kubectl -n kube-system get deploy metrics-server
kubectl -n kube-system logs deploy/metrics-server --tail=20
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/shop/pods | head -c 300

Usage, requests, limits and allocatable

Four different numbers get mixed up under pressure. Usage is what kubectl top measures. Requests are what the scheduler reserves: a node is full when the sum of requests reaches its allocatable, however idle top says it is. Limits cap usage: CPU above the limit is throttled, memory above it gets the container OOMKilled. Allocatable is the node's capacity minus what the kubelet reserves for the system and eviction. A Pod can use far more than it requests, up to its limit or, with no limit, up to what the node has spare; that Pod is the first candidate for eviction when the node runs short of memory.

# Requests and limits of every Pod in a namespace
kubectl -n shop get pods -o custom-columns='NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory'

# What the scheduler has handed out on a node (requests and limits, as % of allocatable)
kubectl describe node cka-worker | sed -n '/Allocated resources/,/Events/p'

# Capacity and allocatable of every node, side by side
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,CAP_MEMORY:.status.capacity.memory'

# The node with the most allocatable memory (the kubelet reports it in Ki)
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.allocatable.memory}{"\n"}{end}' | sort -k2 -n -r

Right after a Pod starts, kubectl top pod answers metrics not available yet for a few scrapes. In a script, retry instead of failing: for i in $(seq 1 30); do kubectl top pods -n shop --no-headers 2>/dev/null | grep -q . && break; sleep 3; done. On the exam the Pods have usually been running long before you arrive, but a Pod you just created for the task will not have numbers for the first 15 to 60 seconds.

metrics-server is not a monitoring system. Prometheus (often deployed as kube-prometheus-stack with Grafana and Alertmanager) scrapes far more metrics, keeps history, answers queries like p95 latency last week and fires alerts. None of that is on the exam cluster and you should not install it there: CKA tasks are answered with kubectl top, kubectl describe node and the object fields. Know the difference so you can explain why the HPA needs metrics-server but your on-call dashboards need Prometheus.