Troubleshooting: How to Investigate Pods, Nodes and the Control Plane

The biggest slice of the CKA is troubleshooting (30%), but it’s really not new knowledge — it’s the ability that ties the whole series together. The worst thing in troubleshooting is guessing and trying things at random. The real method is one sentence: walk the Pod’s lifecycle and ask, gate by gate, “where is it stuck” — because K8s is considerate: it writes “which gate it’s stuck at” straight into the status.

Step one is always these three commands: get → describe → logs

Whatever the symptom, the opening sequence is fixed, peeling the onion from the outside in:

The most underrated one is kubectl describe: its Events section at the bottom writes out almost everything K8s “just tried to do, and where it got stuck” — why scheduling failed, the error pulling the image, a readiness probe failing again and again, being OOM-killed, it’s all there. Many people rush to the logs the moment something breaks, but the answer is often in Events, and more direct. A few common commands:

kubectl get pods -o wide              # status, restart count, which node it landed on
kubectl describe pod <p>              # read the Events section at the bottom (most important)
kubectl logs <p> --previous           # essential for CrashLoop: logs of the "previous instance that died"
kubectl get events --sort-by=.lastTimestamp   # namespace-wide event stream ordered by time

logs --previous is the key to CrashLoopBackOff: the container has already crashed and restarted, so the current logs belong to the new instance and are often empty; what you want is the last few lines left by the previous crash.

Reading Pod status: every status points the way

Spreading the four gates of the first diagram into a lookup table — see the status, know where to look and roughly what the root cause is:

Status	Stuck at	Most common root cause	Check first
Pending	① can’t schedule onto a node	not enough resources, [[k8s-scheduling-advanced	taint without toleration]], PVC won’t bind to a PV
ContainerCreating stuck	between ① and ②	CNI not configured, Volume won’t mount, Secret/ConfigMap doesn’t exist	describe Events
ImagePullBackOff	② pulling the image	image name / tag typo, private registry missing imagePullSecret	describe Events (Failed to pull)
CrashLoopBackOff	③ dies on start	program crashes on start-up, bad config, [[k8s-config-secret	missing environment variable]], probe too strict
OOMKilled	③ memory blown	actual usage exceeded the memory limit and got killed	describe (Last State: OOMKilled), adjust the limit
Running but 0/1 READY	④ readiness not passing	readiness probe keeps failing → not in [[k8s-service	Endpoints]], receives no traffic
Running but unreachable	④ network layer	selector typo so Endpoints are empty, [[k8s-ingress-dns	DNS]] won’t resolve, [[k8s-networkpolicy-cni

This table is the whole series used in reverse: every failure is some mechanism from an earlier post “not operating”. Troubleshooting doesn’t investigate anything new; it tests whether you understand those mechanisms.

Change layers: Node and Control Plane

If every Pod on a whole node is in trouble, stop staring at Pods — the problem is at the Node layer. When kubectl get nodes shows NotReady, it’s usually that machine’s kubelet died, disk/memory pressure (DiskPressure / MemoryPressure), or the network dropped. Then you have to SSH in and read the kubelet’s logs with journalctl -u kubelet, and ask the container runtime directly with crictl, rather than guessing through the API.

Higher still, if kubectl itself starts timing out and the whole cluster seems unreachable — that’s the Control Plane layer. When the api-server or etcd is in trouble, the whole cluster’s ability to “take orders” is paralysed. Because they’re static pods, you go to the control plane machine and look at /etc/kubernetes/manifests, at the container status and logs of those few pods. You climb layer by layer because the higher the layer, the bigger the blast: a dead Pod affects one service; a dead control plane affects the whole cluster.

kubectl debug: when there isn’t even a shell

A wall you often hit in practice: many images nowadays are distroless / without a shell for size and security, so kubectl exec -it -- sh fails outright and there’s no way in. kubectl debug is the answer — it uses an ephemeral container to insert a temporary container with tools into the same Pod, sharing its network and process namespace, so you can curl, read files, capture packets alongside it, without touching the original container at all:

kubectl debug -it <p> --image=busybox --target=<container>   # insert a temporary container to investigate
kubectl debug node/<node> -it --image=busybox                # even a node: open a privileged container to investigate

No need to modify the image to cram in tools for debugging, no need to restart the Pod and destroy the scene — this is the move to remember when investigating Production containers “trimmed down to nothing usable”.

Reflections

Troubleshooting skill isn’t memorising commands; it’s having a “lifecycle map”

I’ve seen too many people troubleshoot by voodoo: without reading the status carefully, they start restarting Pods, deleting and recreating, changing a pile of settings on the off chance. Effective troubleshooting means having that lifecycle map from the first diagram in your head — see Pending and you know it’s the scheduling gate; see ImagePullBackOff and it’s the image gate; see not Ready and you look at readiness and Endpoints. A status isn’t an error message; it’s K8s telling you which step it couldn’t get past. Once that map is internalised, troubleshooting turns from “try things until it works” into “one glance and you know where to dig”, and the efficiency gap is tenfold at least. It echoes the core claim of my SRE troubleshooting post: a systematic mental model always beats a flash of inspiration in the moment.

The Events section is the most underrated goldmine

If I could leave only one troubleshooting tip, it’d be: describe first, read the Events. It’s the “what just happened” that K8s writes for you proactively — why the scheduler couldn’t place it, why the image wouldn’t pull, why the probe failed, all in those few lines. My old bad habit was diving into the logs the moment something broke, but logs are the application’s output and often have nothing to do with platform-level problems like “the Pod won’t start”. Ask the platform first (Events), then the application (logs) — that order has saved me countless wasted detours. The tool laid the answer out long ago; the only difference is whether you looked in the right place first.

Only with this post does the whole series truly close the loop

Writing this, I feel it strongly: troubleshooting is the biggest part of the CKA precisely because it isn’t a standalone chapter; it’s the acceptance test for everything. You have to understand the reconcile loop to know why a Pod restarts itself, understand scheduling to read Pending, understand Service and DNS to chase “unreachable”, understand etcd and the control plane to dare touch the top layer. Troubleshooting ability is the sum of these understandings, and it can’t be faked. So I never treat “can you find problems” as a separate skill to practise — it’s the thermometer of how deeply you understand the system. The whole series, from “declarative” to “troubleshooting”, circles back to where it started: the better you understand how it normally works, the better you know, when something breaks, where it isn’t working.