Logs, events and container output streams

Where container output goes, how to pull exactly the lines you need with kubectl logs, how to surface an app that logs to a file, and how events complete the picture.

The competency Manage and evaluate container output streams shows up as small, precise tasks: write the error lines of one container to a file, find which Pod crashed with a given message, make an app that logs to a file readable with kubectl logs. A container's log is simply what its processes write to stdout and stderr. The runtime captures both streams into one file per container on the node, and kubectl logs asks the kubelet to read that file back. Anything the app writes to a file inside its own filesystem never reaches that stream.

kubectl -n shop logs web-7d9c                      # the default container (see the trap below)
kubectl -n shop logs web-7d9c -c sidecar           # one named container (init containers too)
kubectl -n shop logs web-7d9c --all-containers     # every container, init containers included
kubectl -n shop logs web-7d9c --previous           # the instance before the current one (-p)
kubectl -n shop logs web-7d9c -f                   # follow; Ctrl-C to stop
kubectl -n shop logs web-7d9c --since=10m          # only recent lines (also --since-time=2026-10-17T09:00:00Z)
kubectl -n shop logs web-7d9c --tail=50            # last 50 lines
kubectl -n shop logs web-7d9c --timestamps         # prefix each line with the runtime's RFC3339 timestamp
kubectl -n shop logs -l app=web --prefix           # every Pod with the label; lines start [pod/<name>/<container>]
kubectl -n shop logs deploy/web                    # one Pod picked from the Deployment

# The typical exam shape: one container, one pattern, into a file
kubectl -n shop logs web-7d9c -c app | grep 'ERROR' > /tmp/cka/web-errors.txt

• —Default container. In a multi-container Pod, kubectl logs without -c uses the container named in the annotation kubectl.kubernetes.io/default-container, or else the first one, and prints Defaulted container "app" out of: … on stderr. Always pass -c when the question names a container.
• —Selectors cut output. With -l, --tail defaults to 10 lines per Pod instead of everything, and at most 5 Pods are streamed at once (--max-log-requests). Add --tail=-1 when you need the full log.
• —Where a crash is. The kubelet keeps the current container and normally one exited instance. While a crashed container is down waiting to restart, plain kubectl logs returns the run that just died; once it is running again, that run is only in --previous. Right after a crash --previous can answer unable to retrieve container logs, and on a Pod that never restarted it answers previous terminated container not found, so check both. All of it disappears with the Pod.
• —Output is the answer. Do not add --timestamps or --prefix when writing a file the grader compares, and remember that 2>&1 is not needed: stderr lines are already part of the container log.

Where the logs live on the node

The kubelet tells the runtime to write each container's output to /var/log/pods/<namespace>_<pod>_<pod-uid>/<container>/<restart-count>.log. /var/log/containers/ holds one symlink per container, named <pod>_<namespace>_<container>-<container-id>.log, which is what node log agents tail. Each line carries a timestamp, the stream name and the message. The kubelet rotates the file when it reaches containerLogMaxSize (default 10Mi) and keeps containerLogMaxFiles (default 5); kubectl logs only reads the newest file. When the API server is down, crictl logs on the node still works.

kubectl -n shop get pod web-7d9c -o wide          # which node?
docker exec -it cka-worker bash

ls /var/log/pods/ | grep web-7d9c
tail -n 3 /var/log/pods/shop_web-7d9c_*/app/0.log
# 2026-10-17T10:35:46.709286291Z stderr F connection reset by peer
ls -l /var/log/containers/ | grep web-7d9c

crictl ps --name app                                # container IDs on this node
crictl logs --tail 20 <container-id>
crictl ps -a --name app                             # include exited containers
crictl logs <exited-container-id>                   # works while the runtime still has it

grep -i containerLog /var/lib/kubelet/config.yaml   # rotation settings, if set

When the app writes to a file

apiVersion: v1
kind: Pod
metadata:
  name: report
  namespace: shop
spec:
  initContainers:
    - name: log-tail
      image: busybox:1.36
      restartPolicy: Always            # this line makes it a sidecar
      command: ["sh", "-c", "tail -n +1 -F /data/report.log"]
      volumeMounts:
        - name: data
          mountPath: /data
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "while true; do date >> /data/report.log; sleep 5; done"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      emptyDir: {}

An init container with restartPolicy: Always is a native sidecar, stable since v1.33. It starts before the app containers, keeps running for the Pod's whole life, is restarted if it dies and does not hold up a Job's completion. Both containers mount the same emptyDir; tail -n +1 -F prints the whole file and keeps following it across rotation. Then kubectl logs report -c log-tail shows the app's log. You cannot add containers to a running Pod, so export it with kubectl get pod report -o yaml > report.yaml, add the block, and run kubectl replace --force -f report.yaml. For a Deployment, edit the template and let it roll.

Events are the cluster's own output stream: the scheduler, kubelet and controllers record what they did to an object and why it failed. kubectl describe shows the events of that one object at the bottom. kubectl get events lists every event in the namespace, including events about objects that no longer exist and about owners you did not describe, such as a ReplicaSet's FailedCreate when a quota or admission policy rejects its Pods. Events expire after one hour by default (the API server's --event-ttl), and get events is not sorted by time unless you ask.

kubectl -n shop get events --sort-by=.metadata.creationTimestamp
kubectl -n shop get events --field-selector type=Warning
kubectl -n shop get events --field-selector involvedObject.kind=Pod,involvedObject.name=web-7d9c
kubectl -n shop get events -w                       # stream new events while you fix things
kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -20
kubectl -n shop events --for pod/web-7d9c --types=Warning   # kubectl events sorts by time itself
kubectl -n shop describe pod web-7d9c | sed -n '/^Events:/,$p'
kubectl -n shop describe pod web-7d9c | grep -A5 'Last State'   # reason and exit code of the last crash