Monitoring resource usage

Where kubectl top gets its numbers, and how to tell usage apart from requests, limits and a node's allocatable size.

35 min 3 hands-on tasksTroubleshooting · 30% of the exam

Key ideas in this lesson · tap one for a simple explanation

What you will be able to do

Find the Pod, container or node using the most CPU or memory, and write the answer as the grader expects.
Tell usage, requests, limits and allocatable apart, and you know which command shows each one.
Check that metrics-server, the source of kubectl top, is working.

This exam skill is called Monitor cluster and application resource usage. The tasks are short: which uses the most CPU, which has the most memory. The commands take seconds. You lose points by reading the wrong number, or by writing pod/name when the grader wants just name.

Where the numbers come from

kubectl top gets its numbers from , which asks every for CPU and memory use, every 15 seconds on this lab. It keeps only the latest numbers, so there is no history, and a new Pod appears only after a scrape or two.

Where kubectl top gets its numbers

kubelet

measures each container on its node

metrics-server

keeps only the latest numbers

kubectl top and the HPA

read them through the API server

The HorizontalPodAutoscaler reads the same numbers. If metrics-server is down, kubectl top answers Metrics API not available.

kubectl top

kubectl top, and checking metrics-server behind it· Terminal

kubectl top nodes                                   # usage per node; % is of allocatable
kubectl top nodes --sort-by=memory                  # or --sort-by=cpu; highest first
kubectl top pods -n shop                            # one namespace
kubectl top pods -A --sort-by=cpu                   # every namespace, heaviest first
kubectl top pods -n shop --containers               # one line per container
kubectl top pods -n shop -l app=web --sort-by=memory
kubectl top pod web-7d9c -n shop --containers       # a single Pod (one name only)

# The answer as a bare name, without a header
kubectl top pods -n shop --sort-by=cpu --no-headers | awk 'NR==1 {print $1}' > /tmp/cka/busiest.txt

# Is metrics-server healthy?
kubectl get apiservice v1beta1.metrics.k8s.io       # AVAILABLE must be True
kubectl -n kube-system get deploy metrics-server
kubectl -n kube-system logs deploy/metrics-server --tail=20

A Pod that just started has no numbers for up to a minute: kubectl top says metrics not available yet. Wait and try again.

Column	What it is	Compare it with
CPU(cores)	CPU in use, in millicores: `250m` is a quarter of one CPU core	The CPU request and limit. Above its limit, the is slowed down.
MEMORY(bytes)	Memory in use (the working set). Files in a memory-backed volume count too.	The memory limit. Above it, the container is killed (OOMKilled).
CPU% / MEMORY% (nodes only)	Node usage as a share of the node's allocatable size	Not the Allocated resources in `kubectl describe node`: those are requests, not usage

Usage, requests, limits and allocatable

These four numbers sound alike and are easy to mix up under pressure. They mean different things:

Usage: what a container uses right now. kubectl top shows it.
Request: what the sets aside for the container. A node is full when its Pods' requests add up to its allocatable, even if kubectl top says it is idle.
Limit: the most a container may use. Above its CPU limit it is slowed down. Above its memory limit it is killed (OOMKilled).
Allocatable: the node's size, minus what is kept back for the system. It is a field on the Node: status.allocatable.

Example: a container requests 100Mi, has a 200Mi limit and uses 150Mi. It is above its request, which is allowed. If it tries to go past 200Mi, it is OOMKilled. Requests, limits, LimitRange and ResourceQuota teaches how to set these values.

Requests and capacity next to usage· Terminal

# Requests and limits of every Pod in a namespace
kubectl -n shop get pods -o custom-columns='NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory'

# What the scheduler has handed out on a node (requests and limits, as % of allocatable)
kubectl describe node cka-worker | sed -n '/Allocated resources/,/Events/p'

# Capacity and allocatable of every node, side by side
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,CAP_MEMORY:.status.capacity.memory'

# The node with the most allocatable memory (the kubelet reports it in Ki)
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.allocatable.memory}{"\n"}{end}' | sort -k2 -n -r

Full monitoring tools such as Prometheus keep history and send alerts, but the exam never needs them.

In the official docs· open on exam day too

Resource metrics pipelinesearch: resource metrics pipeline
kubectl topsearch: kubectl top
Resource Management for Pods and Containerssearch: resource requests limits
Tools for Monitoring Resourcessearch: monitoring resources

Hands-on task 1 of 3ts-top-1

~4 min3% weightTroubleshooting

Find the Pod burning CPU

Of the Pods with label app=batch in namespace tsx-top, find the one currently using the most CPU and write only its name to /tmp/cka/top-cpu-pod.txt. The Pods may be new; allow a minute for metrics to appear.

Cluster:kubectl config use-context kind-cka

1
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-1/setup | bash
2
Solve it in your terminal
Use kubectl and the Kubernetes docs, like on exam day. Save any answer files under /tmp/cka/.
3
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-1/verify | bash
Got FAIL? Read which check failed, fix it, run it again.
4
Clean up (optional)
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-1/cleanup | bash

Hands-on task 2 of 3ts-top-2

~3 min2% weightTroubleshooting

The node with the most allocatable memory

Write the name of the node with the most allocatable memory to /tmp/cka/top-mem-node.txt. If several nodes share the highest value, write each of them on its own line, sorted by name.

Cluster:kubectl config use-context kind-cka

1
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-2/setup | bash
2
Solve it in your terminal
Use kubectl and the Kubernetes docs, like on exam day. Save any answer files under /tmp/cka/.
3
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-2/verify | bash
Got FAIL? Read which check failed, fix it, run it again.
4
Clean up (optional)
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-2/cleanup | bash

Hands-on task 3 of 3ts-top-3

~5 min4% weightTroubleshooting

The Pod using more memory than it requested

In namespace tsx-req, one of the Pods labeled app=cache is using more memory than its container requests. Write only that Pod's name to /tmp/cka/over-request.txt. The Pods may be new; allow a minute for metrics to appear.

Cluster:kubectl config use-context kind-cka

1
Set up the scenario
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-3/setup | bash
2
Solve it in your terminal
Use kubectl and the Kubernetes docs, like on exam day. Save any answer files under /tmp/cka/.
3
Check your work
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-3/verify | bash
Got FAIL? Read which check failed, fix it, run it again.
4
Clean up (optional)
curl -fsSL https://codewithmuh.com/labs/cka/t/ts-top-3/cleanup | bash

Finished this lesson?

0 of 3 tasks passed. Your readiness counts tasks, so run them before you move on.

Complete and continue