Pod networking and CNI

The Kubernetes network model, the two address ranges every cluster has, how a CNI plugin wires a Pod into the network, and how to prove two Pods can talk.

The curriculum line is Understand connectivity between Pods. It is rarely a task by itself. It is the knowledge you need in every Service, NetworkPolicy and troubleshooting question: which IP a packet goes to, who assigned it, and what sits between two Pods. Get the model straight here, and the rest of the networking module is just a series of objects built on top of it.

The network model in three rules

• —Every Pod gets its own IP address. All containers in a Pod share one network namespace, so they share that IP and reach each other on localhost. Two containers in one Pod cannot both bind port 8080.
• —Pods reach every other Pod directly, without NAT. A Pod on cka-worker connects to a Pod on cka-worker2 at the target's real Pod IP, and the target sees the caller's real Pod IP as the source.
• —Agents on a node reach every Pod on that node. kubelet's probes and any process on the host can connect to local Pod IPs. That is how liveness and readiness probes work.

Kubernetes defines these rules but does not implement them. The CNI plugin (Calico in this lab; Cilium, Flannel and others elsewhere) gives Pods their addresses and routes. kube-proxy (or a CNI that replaces it) implements Services on top of that flat Pod network. Keep the two jobs separate in your head: Pod-to-Pod is the CNI, Pod-to-Service is kube-proxy.

Two address ranges: Pod CIDR and Service CIDR

# Node shell. Lab: docker exec. Exam: ssh <control-plane-node>, then sudo -i if needed.
docker exec cka-control-plane grep -- --cluster-cidr /etc/kubernetes/manifests/kube-controller-manager.yaml
docker exec cka-control-plane grep -- --service-cluster-ip-range /etc/kubernetes/manifests/kube-apiserver.yaml

# The same flags from the API, no node shell needed (static Pods are mirrored as Pods)
kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
kubectl -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr

# Other places the values show up
kubectl get servicecidrs                                   # networking.k8s.io/v1 ServiceCIDR
kubectl -n kube-system get cm kubeadm-config -o yaml | grep -i subnet
kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR

Look closely at the lab and you will see a trap. The nodes carry podCIDRs like 192.168.1.0/24, yet the Pods on cka-worker have addresses like 192.168.220.43. Calico runs its own IPAM: it carves its IP pool (192.168.0.0/16 here) into /26 blocks and hands them to nodes as needed, ignoring spec.podCIDR. Simpler plugins such as Flannel or the host-local IPAM do use the node's podCIDR. When a question asks for "the Pod network CIDR of the cluster", it means the cluster-wide range, not one node's slice.

How a CNI plugin wires a Pod

1. 01The scheduler binds the Pod to a node. kubelet asks the container runtime (containerd, through CRI) to create the Pod sandbox, which owns a fresh network namespace.
2. 02The runtime reads the first config file in /etc/cni/net.d/ (lexical order) and runs the plugin binary named in it from /opt/cni/bin/ with the ADD command, passing the namespace path and Pod identity.
3. 03The plugin creates a veth pair: one end becomes eth0 inside the Pod, the other stays on the host (cali… interfaces for Calico; with the bridge plugin it is attached to a Linux bridge such as cni0).
4. 04The IPAM plugin (calico-ipam, host-local, …) picks a free address, and the plugin sets the Pod's IP, default route and a host route pointing at the veth.
5. 05For other nodes, the plugin programs routes to their Pod blocks: plain routes, BGP, or an overlay (Calico IP-in-IP here, VXLAN in many setups). When the Pod is deleted, the runtime calls DEL and the address goes back to the pool.

docker exec -it cka-worker bash          # exam: ssh <node>
ls /etc/cni/net.d/                       # 10-calico.conflist  calico-kubeconfig
cat /etc/cni/net.d/10-calico.conflist    # "type": "calico", "ipam": {"type": "calico-ipam"}, plus portmap
ls /opt/cni/bin/                         # calico calico-ipam host-local loopback portmap ...
ip -br link | grep cali                  # host ends of the veth pairs, one per local Pod
ip route                                 # one /32 per local Pod, one route per remote node's block
#   192.168.220.43 dev caliad73dda8ffa scope link
#   192.168.245.0/26 via 172.19.0.3 dev tunl0 onlink

kube-proxy: the Service layer

kubectl -n kube-system get ds kube-proxy
kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep -E '^(mode|clusterCIDR)'
# mode: iptables          (an empty mode also means iptables on Linux)

# The rules themselves, on any node
docker exec cka-worker iptables -t nat -L KUBE-SERVICES -n | head

Find a Pod's IP and node

kubectl get pods -o wide                                     # IP and NODE columns
kubectl get pod web-0 -o jsonpath='{.status.podIP}{"\n"}'
kubectl get pod web-0 -o jsonpath='{.spec.nodeName}{"\n"}'
kubectl get pod web-0 -o jsonpath='{.status.podIPs[*].ip}{"\n"}'   # all IPs on dual-stack
kubectl get pods -l tier=cache -o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName

# A Pod IP is not stable. Delete the Pod and its replacement gets a new one,
# which is the whole reason Services exist.

Prove connectivity with a client Pod

kubectl run tmp --image=busybox:1.36 --rm -it --restart=Never -- wget -qO- -T 3 http://192.168.220.43:8080/hostname
# --rm deletes the Pod when the command exits; -T 3 stops wget waiting forever on a blocked port

kubectl run client --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl wait --for=condition=Ready pod/client --timeout=60s
kubectl exec client -- wget -qO- -T 3 http://192.168.220.43:8080/hostname
kubectl exec client -- ip addr show eth0          # the client's own Pod IP
kubectl delete pod client --now