Certificates, kubeconfig and the CSR API

The cluster's certificates and how to renew them, what a kubeconfig holds, and how to give a new user a certificate through the CSR API.

A certificate is an ID card for a program or a person. Every part of a kubeadm cluster shows one to prove who it is. Example: when the API server connects to etcd, it shows its client certificate, and etcd checks that a CA it trusts signed it. A CA (certificate authority) issues these ID cards: whoever holds its key can sign new ones.

People reach the cluster through a kubeconfig: a file that says which cluster to talk to and which ID to show. This lesson covers both, plus a favorite exam task: create a new user and give them a kubeconfig that works. Certificates also expire and must be renewed. So the lesson touches two exam topics: Create and manage Kubernetes clusters using kubeadm and Manage the lifecycle of Kubernetes clusters.

The certificates kubeadm makes

kubeadm init creates a set of certificates and keys in /etc/kubernetes/pki. This set is called the cluster's PKI (public key infrastructure). These are the ones you will meet:

Next to pki/, kubeadm writes one kubeconfig per component in /etc/kubernetes: controller-manager.conf, scheduler.conf and kubelet.conf. There is also admin.conf for you, the administrator, and super-admin.conf for emergencies. What each identity may do is the topic of RBAC.

• Most certificates are valid for one year. The CAs are valid for ten years.
• The kubelet's client certificate is the exception: the kubelet renews it on its own, under /var/lib/kubelet/pki/.

Check and renew certificates

Two tools read certificates. kubeadm certs check-expiration shows when every kubeadm certificate expires. openssl x509 reads one certificate file in detail.

# exam: ssh <control-plane>; sudo -i      lab: docker exec -it cka-control-plane bash
kubeadm certs check-expiration

# Subject, issuer, expiry and SANs of any certificate
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -enddate
# subject=CN=kube-apiserver
# issuer=CN=kubernetes
# notAfter=Oct 17 09:51:49 2027 GMT
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text        # everything

# Renew one certificate, or all of them (on a real cluster; see below before you do)
kubeadm certs renew apiserver
kubeadm certs renew all

Renewing rewrites the certificate files, including the ones inside the kubeconfig files. But running control plane Pods still hold the old certificate in memory. So after renewing:

1. 1.Restart each control plane Pod. kubectl cannot restart a static Pod. Instead, move its file out of /etc/kubernetes/manifests, wait about 20 seconds for the kubelet to stop it, then move it back.
2. 2.If your own kubeconfig is a copy of admin.conf, copy the renewed file again.

kubeadm upgrade apply also renews every certificate. That is why a cluster upgraded at least once a year never hits expiry.

Inside a kubeconfig

A kubeconfig is a file on your machine. It tells kubectl where the cluster is and who you are. It holds three lists and a pointer:

apiVersion: v1
kind: Config
clusters:                 # where: API server URL and the CA that signed its certificate
  - name: kind-cka
    cluster:
      server: https://127.0.0.1:52634
      certificate-authority-data: LS0tLS1CRUdJTi...     # base64 PEM, or certificate-authority: /path/ca.crt
users:                    # who: a client certificate, a token or an exec plugin
  - name: jane
    user:
      client-certificate-data: LS0tLS1CRUdJTi...
      client-key-data: LS0tLS1CRUdJTi...
contexts:                 # a named pairing of cluster and user, with an optional default namespace
  - name: jane@kind-cka
    context:
      cluster: kind-cka
      user: jane
      namespace: dev
current-context: jane@kind-cka

To list, switch and change contexts with kubectl config, see kubectl fluency. Below you build a kubeconfig from scratch, for a new user.

A new user through the CSR API

Kubernetes has no User object. Instead, any client certificate signed by the cluster CA lets you log in. Its Common Name (CN) becomes the username, and each Organization (O) becomes a group. Example: a certificate for /CN=jane/O=developers logs in as user jane in group developers.

The CertificateSigningRequest (CSR) API gets such a certificate signed, without anyone touching ca.key:

1. 1.You submit a CSR object for the signer kubernetes.io/kube-apiserver-client.
2. 2.An administrator approves it. This signer never approves anything automatically.
3. 3.The controller manager signs it with the cluster CA. The certificate appears in the CSR's status.

expirationSeconds asks for how long the certificate stays valid, at least 600 seconds. The signer may make it shorter.

mkdir -p /tmp/cka && cd /tmp/cka
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane/O=developers" -out jane.csr

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane
spec:
  request: $(base64 < jane.csr | tr -d '\n')
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
    - client auth
EOF

kubectl get csr jane                  # CONDITION: Pending
kubectl certificate approve jane
kubectl get csr jane                  # CONDITION: Approved,Issued
kubectl get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
openssl x509 -in jane.crt -noout -subject -issuer -enddate

Now jane has a certificate. Put it in a kubeconfig file of her own, together with the cluster's address and CA. Run this in the same folder, /tmp/cka:

# A kubeconfig for jane in its own file. On a control plane node the CA is /etc/kubernetes/pki/ca.crt.
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
kubectl config set-cluster kind-cka --server="$SERVER" --certificate-authority=ca.crt --embed-certs=true --kubeconfig=jane.kubeconfig
kubectl config set-credentials jane --client-certificate=jane.crt --client-key=jane.key --embed-certs=true --kubeconfig=jane.kubeconfig
kubectl config set-context jane@kind-cka --cluster=kind-cka --user=jane --kubeconfig=jane.kubeconfig
kubectl config use-context jane@kind-cka --kubeconfig=jane.kubeconfig

kubectl --kubeconfig=jane.kubeconfig auth whoami
kubectl --kubeconfig=jane.kubeconfig get pods       # Forbidden: authenticated, not yet authorized