CNI, CSI and CRI

The three plug-in points for running containers, Pod networking and storage, and where to find each one on a node.

Kubernetes does not run containers, build Pod networks or attach disks by itself. It hands each job to a plug-in, through a standard interface. Think of a power socket: any device with the right plug works. That is why you can swap containerd for CRI-O, or Calico for another network plug-in, without changing Kubernetes.

• CRI (Container Runtime Interface): the kubelet asks a container runtime, such as containerd, to start and stop containers.
• CNI (Container Network Interface): the container runtime asks a network plug-in, such as Calico, to give each new Pod an IP address.
• CSI (Container Storage Interface): Kubernetes asks a storage driver to create volumes and mount them for Pods.

Exam tasks here are short. You find a value on a node and write it to a file. You use crictl where kubectl cannot reach. Or you say which interface a broken node or Pod points at.

CRI: how the kubelet reaches the runtime

The kubelet never starts containers itself. It asks the container runtime, through the CRI, over a socket file on the node. To find that socket, read containerRuntimeEndpoint in the kubelet's config file, /var/lib/kubelet/config.yaml.

• On this lab and most clusters the runtime is containerd, and the endpoint is unix:///run/containerd/containerd.sock.
• Older kubeadm clusters may set it as a flag instead: --container-runtime-endpoint in /var/lib/kubelet/kubeadm-flags.env.
• How containerd itself is set up, such as its cgroup driver, is part of Installing a cluster with kubeadm.

crictl: look below the API

crictl talks to the same socket as the kubelet. It shows what the runtime sees on one node, and it still works when the API server or the kubelet is down. Do not mix it up with kubectl:

A Pod sandbox is the runtime's home for one Pod. It holds the Pod's network, and the Pod's containers run inside it. crictl pods lists sandboxes, and crictl ps lists containers.

docker exec -it cka-worker bash                 # exam: ssh <node>, then sudo -i

grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
cat /var/lib/kubelet/kubeadm-flags.env           # extra kubelet flags kubeadm wrote

crictl pods --namespace kube-system              # pod sandboxes (POD ID)
crictl ps -a --name coredns                      # containers, including exited ones (CONTAINER)
crictl ps --name web --no-trunc -q               # full 64-character IDs only

Reading logs and debugging with crictl is taught in kubectl debug, ephemeral containers and crictl.

CNI: which network config a node uses

Each time a Pod starts, the container runtime calls the CNI plug-in to set up the Pod's network. Pod networking and CNI walks through the whole flow. For this lesson's task, three facts are enough:

• The runtime reads the config files in /etc/cni/net.d and uses only the first one in lexical order. Lexical means sorted character by character, like a dictionary. Only .conflist, .conf and .json files count.
• Each plug-in in that file has a type. The type is the name of a program in /opt/cni/bin.
• A leftover file such as 05-old.conf sorts before 10-calico.conflist, so it wins. New Pods then get the wrong network.

docker exec cka-worker ls -l /etc/cni/net.d       # 10-calico.conflist and calico-kubeconfig
docker exec cka-worker cat /etc/cni/net.d/10-calico.conflist   # plugins: "calico" (ipam "calico-ipam"), then "portmap"
docker exec cka-worker ls /opt/cni/bin            # calico, calico-ipam, host-local, loopback, portmap, ...

CSI: storage drivers

A CSI driver lets Kubernetes use a storage system, such as cloud disks, without that system's code being part of Kubernetes. A driver usually has two parts:

• A node plugin: a DaemonSet with one Pod per node. It registers with the kubelet and mounts volumes into Pods.
• A controller: a Deployment whose helper containers create, attach and resize volumes when claims ask for them.

A StorageClass names its driver in provisioner, for example ebs.csi.aws.com. StorageClasses and dynamic provisioning covers how claims use it.

kubectl get csidrivers
kubectl get csinode
kubectl get csinode cka-worker -o jsonpath='{.spec.drivers[*].name}'
kubectl get volumeattachments

From a symptom to an interface

When something breaks, the symptom tells you which interface to look at. The troubleshooting module shows the fixes, starting with Worker node failures.