Scheduling: selectors, affinity, taints and spread

How the scheduler picks a node, and the ways you can steer a Pod: nodeSelector, node affinity, pod anti-affinity, topology spread, taints and tolerations, and nodeName.

When you create a Pod, the scheduler picks the node it runs on. Most of the time you let it choose. Sometimes you need a say. Your Django app's Pods should run only on nodes with an SSD, or two copies of a database should never share one node.

On the exam, the grader can see exactly where your Pods landed. That cuts both ways: a Deployment running on the wrong node scores the same as one stuck in Pending. So always finish with kubectl get pods -o wide, which shows each Pod's node.

How the scheduler chooses a node

The scheduler watches for new Pods that have no node yet: their spec.nodeName is empty. For each one, it works in three steps:

1. 1.Filter: remove every node that cannot run the Pod. Examples are a node without enough CPU left to reserve, or a node with a taint the Pod does not tolerate.
2. 2.Score: rank the nodes that are left. Preferred rules, spreading, and images already on the node all add points.
3. 3.Bind: give the Pod to the best node by writing its name into spec.nodeName. The kubelet on that node then starts the Pod.

If no node passes the filter, the Pod stays Pending with a FailedScheduling event. The event names one reason per rejected node, for example: 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod anti-affinity rules.

Read it with kubectl describe pod. To see these events for every Pod, run kubectl get events --field-selector reason=FailedScheduling.

Here are the ways to steer a Pod. The names look alike, so keep them apart by what each one looks at:

nodeSelector, node affinity and nodeName

Nodes carry labels, just like Pods. For example, you might label one node disktype=ssd. There are two ways to send a Pod only to nodes with a given label:

• nodeSelector: simple. Every label you list must match exactly.
• nodeAffinity: more flexible. It has operators such as In and NotIn, and both hard rules (required) and soft ones (preferred).

    spec:
      nodeSelector:                 # every label must match (AND)
        pool: frontend
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:      # terms are ORed ...
              - matchExpressions:   # ... expressions inside one term are ANDed
                  - key: kubernetes.io/arch
                    operator: In    # In, NotIn, Exists, DoesNotExist, Gt, Lt
                    values: ["amd64", "arm64"]
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 80            # 1-100, added to the node's score
              preference:
                matchExpressions:
                  - key: disktype
                    operator: In
                    values: ["ssd"]

• If a Pod has both nodeSelector and nodeAffinity, a node must pass both.
• IgnoredDuringExecution in the long field names means: if a node's labels change later, Pods already running there stay.
• NotIn and DoesNotExist keep a Pod away from nodes with a label. That is often simpler than tainting them.

spec.nodeName skips the scheduler completely. The kubelet on that node just starts the Pod. The scheduler's checks never run, so NoSchedule taints are ignored: the kubelet only enforces NoExecute taints.

That makes nodeName the tool for manual scheduling, when the scheduler is down and Pods sit in Pending. You cannot add nodeName to an existing Pod; the API refuses the change. Delete the Pod, then create it again with nodeName set.

Pod anti-affinity and topology spread

Node rules look at node labels. Pod rules look at other Pods. Their most common use is to keep copies of one app apart, so that one broken node cannot take them all down.

Pod anti-affinity keeps a Pod off nodes that already run Pods with a given label. Here, each app=api Pod avoids nodes that already run one:

  template:
    metadata:
      labels:
        app: api
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: api
              topologyKey: kubernetes.io/hostname    # "same place" means same node
      containers:
        - name: api
          image: nginx:1.27

The topologyKey is a node label that says what "the same place" means. With kubernetes.io/hostname, each node is its own place. With topology.kubernetes.io/zone, each zone is one place. The rule matches Pods in the Pod's own namespace.

Topology spread is the gentler tool. Instead of "never together", it says "keep the numbers even". It sits in the same Pod spec, beside affinity:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule      # or ScheduleAnyway (a preference)
          labelSelector:
            matchLabels:
              app: api

With maxSkew: 1, 5 replicas on 2 nodes still all run, as 3 and 2. A required anti-affinity rule would leave 3 of them Pending.

There is also podAffinity, the opposite pull: run near Pods with a given label, for example in the same zone as a cache. It is written the same way, under affinity.podAffinity.

Taints and tolerations

A taint is a mark on a node that keeps Pods away. A toleration on a Pod says "I accept that mark". Example: you taint a node that has a GPU, so that only machine-learning Pods, which tolerate the taint, can use it.

Every taint has an effect, which says how strongly it keeps Pods away:

kubectl label node cka-worker pool=frontend
kubectl get nodes -L pool                        # show one label as a column
kubectl label node cka-worker pool-              # remove it

kubectl taint node cka-worker2 dedicated=batch:NoSchedule
kubectl get node cka-worker2 -o jsonpath='{.spec.taints}'; echo
kubectl taint node cka-worker2 dedicated=batch:NoSchedule-   # remove it (trailing minus)
kubectl describe node cka-control-plane | grep Taints        # node-role.kubernetes.io/control-plane:NoSchedule

      tolerations:
        - key: dedicated              # matches dedicated=batch:NoSchedule exactly
          operator: Equal
          value: batch
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists            # any value; no value field allowed
          effect: NoSchedule

Kubernetes also taints nodes by itself, for example with node.kubernetes.io/unreachable when a node stops reporting. Those taints are covered in Worker node failures.

A taint keeps Pods away, but a toleration only allows a Pod in. It never sends the Pod there. A Pod that tolerates dedicated=batch can still land on any node without a taint. So to keep a node for some Pods only, you need both halves:

On this shared lab, never use NoExecute. It evicts every Pod on the node that does not tolerate it, including other lessons' Pods.