Pod networking and CNI

How Pods get IP addresses and reach each other, the two address ranges of a cluster, and how to test a connection.

Every Pod in a cluster gets its own IP address, and any Pod can reach any other Pod at that address. This lesson shows who hands out those addresses, how traffic travels between nodes, and how to test it. The exam calls this Understand connectivity between Pods. It is rarely a task on its own, but every Service, NetworkPolicy and troubleshooting question builds on it.

The network model in three rules

Kubernetes sets three rules for the network. Every cluster must follow them.

• Every Pod gets its own IP address. All containers in one Pod share it, and they reach each other on localhost. So two containers in one Pod cannot both use port 8080.
• Any Pod can reach any other Pod at its real IP. A Pod on cka-worker calls a Pod on cka-worker2 directly. No address is rewritten on the way (no NAT), so the receiver sees the caller's real IP.
• Programs on a node can reach every Pod on that node. That is how the kubelet runs liveness and readiness probes.

CNI plugin and kube-proxy: two parts, two jobs

Kubernetes only writes these rules down. Two other parts make them work, and their jobs are easy to mix up:

So Pod-to-Pod is the CNI plugin, and Pod-to-Service is kube-proxy. Other CNI plugins you may meet are Cilium and Flannel. This lesson covers the CNI side; Services covers kube-proxy.

Two address ranges: Pod CIDR and Service CIDR

Every cluster has two separate ranges of IP addresses. Pods get their addresses from one, Services from the other. A range is written as a CIDR: 10.244.0.0/16 means every address that starts with 10.244..

# Node shell. Lab: docker exec. Exam: ssh <control-plane-node>, then sudo -i if needed.
docker exec cka-control-plane grep -- --cluster-cidr /etc/kubernetes/manifests/kube-controller-manager.yaml
docker exec cka-control-plane grep -- --service-cluster-ip-range /etc/kubernetes/manifests/kube-apiserver.yaml

# Each node's own slice of the Pod range (not the cluster's Pod CIDR)
kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR

No node shell? The same flags are visible through the API too, as Control plane and node components shows.

How the CNI plugin connects a Pod

When a new Pod starts on a node, the container runtime calls the CNI plugin. The plugin does three jobs:

1. 1.It plugs the Pod in. It creates a veth pair: a virtual network cable with two ends. One end becomes eth0 inside the Pod. The other end stays on the node; with Calico its name starts with cali.
2. 2.It picks an address. Its IPAM part (IP address management) takes a free IP from the pool. The plugin gives the Pod that IP, and adds a route on the node that points to the veth.
3. 3.It connects the nodes. It adds routes to the other nodes' Pod blocks. Calico here uses an IP-in-IP tunnel; many other setups use VXLAN. Both wrap the Pod's packet inside a packet between the two nodes.

When the Pod is deleted, the plugin frees its IP again. CNI, CSI and CRI shows where the plugin's files live on the node, and why a node without them stays NotReady.

docker exec -it cka-worker bash          # exam: ssh <node>
ip -br link | grep cali                  # host ends of the veth pairs, one per local Pod
ip route                                 # one /32 per local Pod, one route per remote node's block
#   10.244.89.67 dev caliad73dda8ffa scope link        (a local Pod)
#   10.244.157.128/26 via 172.19.0.3 dev tunl0 onlink  (another node's block, IP-in-IP)

Find a Pod, then reach it

Most networking tasks start with three facts about a Pod: its name, its IP and its node. These commands print them.

kubectl get pods -o wide                                     # IP and NODE columns
kubectl get pod web-0 -o jsonpath='{.status.podIP}{"\n"}'
kubectl get pod web-0 -o jsonpath='{.spec.nodeName}{"\n"}'
kubectl get pods -l tier=cache -o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName

# A Pod IP is not stable. Delete the Pod and its replacement gets a new one,
# which is the whole reason Services exist.

To test a connection, start a small client Pod and call the other Pod's IP from inside it. busybox:1.36 makes a good client: it is tiny, and it has wget and nslookup.

# Interactive: the one-liner you will type on the exam.
# --rm deletes the Pod afterwards; -T 3 stops wget waiting forever on a blocked port.
kubectl run tmp --image=busybox:1.36 --rm -it --restart=Never -- wget -qO- -T 3 http://10.244.89.67:8080/hostname

# Non-interactive (scripts, repeated checks): a long-lived client, then exec
kubectl run client --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl wait --for=condition=Ready pod/client --timeout=60s
kubectl exec client -- wget -qO- -T 3 http://10.244.89.67:8080/hostname
kubectl delete pod client --now