Certificates, kubeconfig and the CSR API

Where kubeadm keeps the cluster PKI, how to check and renew certificates, how a kubeconfig is put together, and how to mint a user's credentials through the CertificateSigningRequest API.

Architecture · 25%45 min2 tasks

After this lesson

  • 01You can find any control plane certificate, read its subject, issuer, SANs and expiry with openssl, and renew it with kubeadm.
  • 02You can read and build a kubeconfig with kubectl config, including embedded credentials.
  • 03You can create a user with a private key and a CertificateSigningRequest, approve it and hand the user a working kubeconfig.

Every component of a kubeadm cluster proves who it is with an X.509 certificate, and every human reaches the cluster through a kubeconfig. The PKI that kubeadm init lays down is part of Create and manage Kubernetes clusters using kubeadm, and because certificates expire on a schedule, checking and renewing them is part of Manage the lifecycle of Kubernetes clusters. The exam also likes a practical variant: create credentials for a new user and give them a kubeconfig that works.

The PKI kubeadm builds

File in /etc/kubernetes/pkiWhat it isSigned by
ca.crt, ca.keyThe cluster CA. The API server accepts client certificates it signed (--client-ca-file), and the controller manager signs approved CSRs with it.Itself
apiserver.crtThe API server's serving certificate. Its SANs cover the node IP, the kubernetes Service IP, the kubernetes.default… names and the control plane endpoint.ca
apiserver-kubelet-client.crtThe API server's client identity toward kubelets (logs, exec, port-forward).ca
apiserver-etcd-client.crtThe API server's client identity toward etcd.etcd/ca
front-proxy-ca.crt, front-proxy-client.crtAuthenticates the API server to aggregated APIs such as metrics-server.front-proxy-ca
sa.key, sa.pubNot certificates: the key pair that signs and verifies ServiceAccount tokens.n/a
etcd/ca.crt, etcd/server.crt, etcd/peer.crt, etcd/healthcheck-client.crtetcd's own PKI: its serving, member-to-member and health-probe identities.etcd/ca

Next to pki/, kubeadm writes one kubeconfig per identity in /etc/kubernetes: controller-manager.conf, scheduler.conf, kubelet.conf and two admin files. admin.conf is user kubernetes-admin in group kubeadm:cluster-admins, which RBAC binds to cluster-admin. super-admin.conf is in system:masters, which bypasses RBAC entirely; keep it for emergencies. Leaf certificates are valid for one year and the CAs for ten. The kubelet's client certificate is the exception: the kubelet rotates it on its own, under /var/lib/kubelet/pki/.

Check expiry, inspect a certificate, renewbash
# exam: ssh <control-plane>; sudo -i      lab: docker exec -it cka-control-plane bash
kubeadm certs check-expiration

# Subject, issuer, expiry and SANs of any certificate
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -enddate
# subject=CN=kube-apiserver
# issuer=CN=kubernetes
# notAfter=Oct 17 09:51:49 2027 GMT
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text        # everything

# Renew one certificate, or all of them (on a real cluster; see below before you do)
kubeadm certs renew apiserver
kubeadm certs renew all

Renewal rewrites the files, including the certificates embedded in the kubeconfig files, but running control plane Pods keep the old certificate in memory until they restart. kubectl cannot restart a static Pod, so move its manifest out of /etc/kubernetes/manifests, wait about 20 seconds for the kubelet to stop it, and move it back. If your own kubeconfig is a copy of admin.conf, copy the renewed file again. kubeadm upgrade apply renews every certificate as a side effect, which is why clusters upgraded at least once a year never hit expiry.

Anatomy of a kubeconfig

Three lists and a pointeryaml
apiVersion: v1
kind: Config
clusters:                 # where: API server URL and the CA that signed its certificate
  - name: kind-cka
    cluster:
      server: https://127.0.0.1:52634
      certificate-authority-data: LS0tLS1CRUdJTi...     # base64 PEM, or certificate-authority: /path/ca.crt
users:                    # who: a client certificate, a token or an exec plugin
  - name: jane
    user:
      client-certificate-data: LS0tLS1CRUdJTi...
      client-key-data: LS0tLS1CRUdJTi...
contexts:                 # a named pairing of cluster and user, with an optional default namespace
  - name: jane@kind-cka
    context:
      cluster: kind-cka
      user: jane
      namespace: dev
current-context: jane@kind-cka
kubectl config, the subcommands you will usebash
kubectl config get-contexts
kubectl config current-context
kubectl config use-context kind-cka
kubectl config set-context --current --namespace=default     # default namespace of the current context
kubectl config view --minify                                 # current context only, secrets redacted
kubectl config view --minify --raw                           # with the embedded data

# Work on a separate file without touching ~/.kube/config
kubectl --kubeconfig=/tmp/cka/jane.kubeconfig config get-contexts
KUBECONFIG=~/.kube/config:/tmp/cka/jane.kubeconfig kubectl config get-contexts    # merged view

A new user through the CSR API

Kubernetes has no User object. A client certificate signed by the cluster CA is enough to authenticate: its Common Name becomes the username and each Organization becomes a group. The CertificateSigningRequest API gets such a certificate signed without anyone touching ca.key. You submit a CSR with signer kubernetes.io/kube-apiserver-client, an administrator approves it, and the controller manager signs it with the cluster CA. That signer is never approved automatically, unlike kubernetes.io/kube-apiserver-client-kubelet, which node bootstrapping uses. expirationSeconds requests a validity (600 at the least); the signer may shorten it.

Key, CSR, approval, certificate, kubeconfigbash
mkdir -p /tmp/cka && cd /tmp/cka
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane/O=developers" -out jane.csr

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane
spec:
  request: $(base64 < jane.csr | tr -d '\n')
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
    - client auth
EOF

kubectl get csr jane                  # CONDITION: Pending
kubectl certificate approve jane
kubectl get csr jane                  # CONDITION: Approved,Issued
kubectl get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
openssl x509 -in jane.crt -noout -subject -issuer -enddate

# A kubeconfig for jane in its own file. On a control plane node the CA is /etc/kubernetes/pki/ca.crt.
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
kubectl config set-cluster kind-cka --server="$SERVER" --certificate-authority=ca.crt --embed-certs=true --kubeconfig=jane.kubeconfig
kubectl config set-credentials jane --client-certificate=jane.crt --client-key=jane.key --embed-certs=true --kubeconfig=jane.kubeconfig
kubectl config set-context jane@kind-cka --cluster=kind-cka --user=jane --kubeconfig=jane.kubeconfig
kubectl config use-context jane@kind-cka --kubeconfig=jane.kubeconfig

kubectl --kubeconfig=jane.kubeconfig auth whoami
kubectl --kubeconfig=jane.kubeconfig get pods       # Forbidden: authenticated, not yet authorized

In the docs · the tab you get on exam day

Taskar-certs-1
Architecture3%~6 min

context · kubectl config use-context kind-cka

Inspect control plane certificates

Inspect this cluster's certificates on cka-control-plane. (1) Write the expiry date of the API server's serving certificate to /tmp/cka/ar-apiserver-expiry.txt, exactly as openssl prints it after notAfter= (for example Oct 17 09:51:49 2027 GMT). (2) Find the client certificate the API server presents to etcd and write the Common Name of the CA that issued it to /tmp/cka/ar-etcd-client-issuer.txt. (3) Save the complete output of kubeadm certs check-expiration to /tmp/cka/ar-cert-expiration.txt.
  1. 01

    Set up the scenario

    curl -fsSL https://codewithmuh.com/labs/cka/t/ar-certs-1/setup | bash
  2. 02

    Solve it in your terminal

    Work the way the exam makes you: kubectl, the docs, nothing else. Put any answer files under /tmp/cka/.

  3. 03

    Check your work

    curl -fsSL https://codewithmuh.com/labs/cka/t/ar-certs-1/verify | bash
    Counts toward your readiness score.
  4. 04

    Clean up

    curl -fsSL https://codewithmuh.com/labs/cka/t/ar-certs-1/cleanup | bash
Taskar-certs-2
Architecture7%~10 min

context · kubectl config use-context kind-cka

Create a user with the CSR API and a kubeconfig

Create credentials for a new user ar-dev in group ar-team. Generate a 2048-bit RSA private key at /tmp/cka/ar-dev.key and a certificate signing request for it. Submit the request as a CertificateSigningRequest named ar-dev for signer kubernetes.io/kube-apiserver-client, valid for 24 hours, approve it, and save the issued certificate as /tmp/cka/ar-dev.crt. Then create a self-contained kubeconfig at /tmp/cka/ar-dev.kubeconfig (CA, certificate and key embedded, not referenced by path) whose current context connects to this cluster as ar-dev with default namespace ar-certs. A RoleBinding that lets ar-dev read Pods in ar-certs already exists. Do not change your own kubeconfig.
  1. 01

    Set up the scenario

    curl -fsSL https://codewithmuh.com/labs/cka/t/ar-certs-2/setup | bash
  2. 02

    Solve it in your terminal

    Work the way the exam makes you: kubectl, the docs, nothing else. Put any answer files under /tmp/cka/.

  3. 03

    Check your work

    curl -fsSL https://codewithmuh.com/labs/cka/t/ar-certs-2/verify | bash
    Counts toward your readiness score.
  4. 04

    Clean up

    curl -fsSL https://codewithmuh.com/labs/cka/t/ar-certs-2/cleanup | bash

Finished reading?

0 of 2 tasks passed. Readiness counts tasks, so run them before you move on.