Scheduling: selectors, affinity, taints and spread

How the scheduler filters and scores nodes, and every lever that steers a Pod: nodeName, nodeSelector, node affinity, pod affinity and anti-affinity, taints and tolerations, and topology spread.

This is the scheduling half of Configure Pod admission and scheduling (limits, node affinity, etc.). Scheduling tasks have a satisfying property: the grader can see exactly where your Pods landed. That cuts both ways. A Deployment that is Running on the wrong node scores the same as one that is Pending, so always finish with kubectl get pods -o wide.

How the scheduler chooses a node

kube-scheduler watches for Pods with an empty spec.nodeName. It first filters out nodes that cannot run the Pod: not enough unrequested CPU or memory, a nodeSelector or required affinity that does not match, a taint the Pod does not tolerate, a host port already taken, a volume in another zone. It then scores the survivors (spreading, preferred affinities, balanced allocation, images already present) and binds the Pod to the best one by setting nodeName. If no node survives the filter, the Pod stays Pending with a FailedScheduling event such as 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod anti-affinity rules. Read it with kubectl describe pod or kubectl get events --field-selector reason=FailedScheduling; it names one reason per rejected node.

nodeSelector, node affinity and nodeName

    spec:
      nodeSelector:                 # every label must match (AND)
        pool: frontend
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:      # terms are ORed ...
              - matchExpressions:   # ... expressions inside one term are ANDed
                  - key: kubernetes.io/arch
                    operator: In    # In, NotIn, Exists, DoesNotExist, Gt, Lt
                    values: ["amd64", "arm64"]
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 80            # 1-100, added to the node's score
              preference:
                matchExpressions:
                  - key: disktype
                    operator: In
                    values: ["ssd"]

When both nodeSelector and nodeAffinity are present, a node must satisfy both. IgnoredDuringExecution means a Pod already running stays put if the node's labels change later. NotIn and DoesNotExist give you anti-affinity to nodes, which is often simpler than tainting them.

spec.nodeName skips the scheduler entirely: the kubelet on that node just starts the Pod. No scheduler means no resource filtering (the kubelet rejects a Pod that does not fit, with a reason like OutOfcpu) and no NoSchedule check, because the kubelet only enforces NoExecute taints. That makes it the tool for manual scheduling when kube-scheduler is down and Pods sit Pending. You cannot patch nodeName onto an existing Pod; the API rejects the update. Delete the Pod and create it again with nodeName set, then go fix the scheduler (on kubeadm clusters, its static Pod manifest in /etc/kubernetes/manifests).

Pod affinity, anti-affinity and topology spread

  template:
    metadata:
      labels:
        app: api
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: api
              topologyKey: kubernetes.io/hostname    # "same place" means same node
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 50
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: cache
                topologyKey: topology.kubernetes.io/zone   # near the cache, same zone
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule      # or ScheduleAnyway (a preference)
          labelSelector:
            matchLabels:
              app: api
      containers:
        - name: api
          image: nginx:1.27

Pod affinity rules match other Pods' labels, within the Pod's own namespace unless you add namespaces or namespaceSelector. The topologyKey is a node label that defines a domain: per node with kubernetes.io/hostname, per zone with topology.kubernetes.io/zone. Topology spread is the gentler tool: maxSkew: 1 lets domains differ by at most one Pod, so it still schedules five replicas on two nodes, where a required anti-affinity would leave three Pending.

Taints and tolerations

kubectl label node cka-worker pool=frontend
kubectl get nodes -L pool                        # show one label as a column
kubectl label node cka-worker pool-              # remove it

kubectl taint node cka-worker2 dedicated=batch:NoSchedule
kubectl get node cka-worker2 -o jsonpath='{.spec.taints}'; echo
kubectl taint node cka-worker2 dedicated=batch:NoSchedule-   # remove it (trailing minus)
kubectl describe node cka-control-plane | grep Taints        # node-role.kubernetes.io/control-plane:NoSchedule

      tolerations:
        - key: dedicated              # matches dedicated=batch:NoSchedule exactly
          operator: Equal
          value: batch
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists            # any value; no value field allowed
          effect: NoSchedule
        - key: node.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 60       # stay 60s on an unreachable node, then be evicted

A taint repels; a toleration only permits. A Pod that tolerates dedicated=batch can still land on any untainted node, so dedicating nodes takes both halves: a taint to keep others off and a label plus nodeSelector or affinity to pull the right Pods on. Every Pod also gets not-ready and unreachable NoExecute tolerations of 300 seconds from the DefaultTolerationSeconds admission plugin, which is why Pods on a dead node take about five minutes to be replaced. On this shared lab, never use NoExecute: it evicts every Pod on the node that lacks a toleration, other lessons' Pods included.