Cluster Administration: kubeadm, etcd Backups, Upgrades

The previous eleven posts all stood in the position of “using the cluster”. This one moves to “building and keeping the cluster” — the hardest ops work in the Cluster Architecture domain that’s 25% of the CKA. Three things run through an administrator’s whole career: how to turn a pile of machines into a cluster (kubeadm), how to bring it back when disaster strikes (etcd backups), and how to upgrade without downtime (upgrade). The second is the one question in the whole CKA you should practise until it’s reflex, so first let me be clear about why it matters so much.

kubeadm: a pile of machines into a cluster in one step

Bringing up a control plane by hand (signing a heap of certificates, configuring the api-server, wiring in etcd) is a nightmare. kubeadm turns it into two commands:

The control plane components all run as static pods — the kubelet watches the /etc/kubernetes/manifests directory, brings up the manifests inside and keeps them running. That’s also why, when debugging the control plane, you go to that machine and look at those files and those pods, rather than reasoning with ordinary Deployment logic.

etcd: the cluster’s single source of truth, and its single weak spot

Look at etcd in that diagram — the state of every object in your cluster (Deployment, Service, Secret, RBAC…) lives in exactly one place: etcd. The first post said the reconcile loop keeps pulling reality towards the “desired state”, and that desired state lives in etcd. It keeps multiple replicas consistent with Raft consensus, so an HA deployment needs an odd number of members (3 tolerates 1 down, 5 tolerates 2) to form a majority and avoid split brain.

But no number of replicas protects against “accidental delete”, “broken certificates”, or “the whole etcd datastore corrupted”. So the administrator’s first commandment is: regularly snapshot etcd to somewhere outside the cluster. It’s your only restore point — without it, a dead cluster means rebuilding from zero.

The skeleton of the commands (in reality you pass the endpoints and the three certificate flags --cacert/--cert/--key):

# normal times: back up regularly, move snapshot.db outside the cluster
ETCDCTL_API=3 etcdctl snapshot save snapshot.db
# after disaster: restore into a new directory, then point the etcd static pod manifest at it and restart
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir /var/lib/etcd-restore

“Is there a usable etcd backup” practically defines a cluster’s disaster-recovery capability. Don’t wait for an incident to discover the snapshot never actually succeeded.

Upgrades: a relay race, one node at a time

Upgrading the cluster’s K8s version has hard rules: only one minor version at a time (1.29 → 1.30, never 1.29 → 1.31), and the control plane before the workers. The whole process is a relay race of “touch one machine at a time, the rest keep serving”:

The first control plane node uses kubeadm upgrade apply to set the version the whole cluster is moving to; the remaining control plane and worker nodes follow with kubeadm upgrade node. drain respects a Deployment‘s multiple replicas: it evicts Pods from this machine, the ReplicaSet immediately replaces them elsewhere, so as long as your service has several copies and a PodDisruptionBudget, the rolling upgrade keeps serving throughout.

Reflections

etcd backup is the kind of thing nobody remembers until the day nothing works without it

My respect for backups was fed by real fear. All of the cluster’s state condensed into one place, etcd, is elegant design, but it also means it’s the single point of death for the whole cluster — replicas withstand a machine dying, but not one bad operation deleting critical objects, or certificates expiring so etcd won’t start. In that moment, whether you have a verified, restorable snapshot in hand is the difference between “ten minutes to recover” and “an all-nighter rebuilding the whole cluster”. So I treat etcd backups as the reliability basics the SRE posts describe: not just scheduling backups, but actually rehearsing restores regularly — a backup you’ve never restored from is only a sense of safety you think you have.

The “one node at a time” upgrade philosophy is really the same thing as a rolling update

Upgrading a cluster looks scary, but taken apart it’s the scaled-up version of the same idea as a Deployment’s rolling update: only ever let a small part be in flux, keep the rest serving, and be able to back out if it breaks. At the application layer it’s “swap a few Pods at a time”; at the cluster layer it’s “upgrade one node at a time”; drain is to a node what a readiness probe is to a Pod — move the traffic off cleanly, then act. Once I saw that symmetry, my fear of “touching Production” shrank a lot: the method is the same, only the unit changed from Pod to node. Cutting a big move into a chain of reversible small steps is the most consistent, most worth-internalising principle I’ve seen across the whole K8s world.

An administrator’s value lies in the invisible everyday preparation

Writing this post made me surer: being able to kubectl apply is just the entry ticket; the real ability to carry a cluster on your shoulders hides in these silent preparations — did the backup succeed? Has the restore been rehearsed? Has the upgrade path been tested? When do the certificates expire? These have no presence at all when things are smooth, and are everything when things break. It matches my understanding of SRE exactly: reliability isn’t heroics improvised on the day of the incident; it’s the discipline before the incident, day after day, with nobody clapping. The next post in the series covers troubleshooting — when all this preparation still didn’t stop the problem, how to dig it out layer by layer.