Deployments and Self-Healing: The Reconcile Loop in Practice

The first post gave the soul (the reconcile loop), the second gave the atom (the Pod). But in practice you almost never create a Pod by hand — what you declare is a Deployment, and it’s the reconcile loop’s most practical, most common incarnation. This post looks at how it self-heals and how it changes versions with zero downtime.

Why not just create Pods

Because a bare Pod that dies has nobody to save it. Create a Pod by hand, and the moment it crashes or its node breaks, it’s simply gone — nothing remembers “there was supposed to be one of these”. What you want isn’t “start a Pod”; it’s “always keep N healthy Pods”. That’s exactly the kind of thing that needs a controller watching it, and Deployment is that controller:

The division of labour is clear: the Deployment manages versions and update strategy; the ReplicaSet beneath it does exactly one thing — watch the actual Pod count; create when short, kill when over. You only declare “I want 3 copies of v1”; everything else is that loop running.

What this “desired state” looks like in YAML

K8s is declarative: you don’t issue step-by-step commands; you write a YAML describing “what I want” and hand it to the reconcile loop — which is why people say K8s has an Infrastructure as Code flavour. That whole self-healing setup above is declared in just these lines:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3                       # desired: always keep 3 copies
  selector:
    matchLabels: { app: web }       # which Pods this Deployment manages
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1             # at most this many unavailable during a rollout
      maxSurge: 1                   # at most this many extra new ones during a rollout
  template:                         # ↓ below: the template for "what each Pod looks like"
    metadata:
      labels: { app: web }          # the Pod's labels, must be matched by the selector above
    spec:
      containers:
        - name: web
          image: myrepo/web:1.0
          resources:                # basis for scheduling and QoS
            requests: { cpu: "100m", memory: "128Mi" }
            limits:   { cpu: "500m", memory: "256Mi" }
          readinessProbe:           # not passing → not added to the Service's traffic list
            httpGet: { path: /healthz, port: 8080 }

Three places are key to understanding a Deployment:

• replicas: 3 is your “desired state” — the ReplicaSet watches it all day; short, create; over, kill. Change it to 5, kubectl apply, and the loop fills up to 5; you never issued a “start two containers” command.
• selector and template.labels must match — that’s how the Deployment recognises “which Pods are mine”. If the labels don’t line up, apply is rejected outright.
• template is the Pod template, and also the trigger for a rolling update — change any field inside it (image, env, resources…) and a rollout starts; changing only replicas isn’t a rollout, just a count adjustment. (Which is also why editing a referenced ConfigMap/Secret doesn’t roll out automatically — it never touched this template.)

Self-healing: the reconcile loop is always running

So-called “K8s self-heals” isn’t mysterious at all once you take it apart; it’s the reconcile loop doing its job:

• You declare desired = 3 Pods.
• The ReplicaSet controller keeps comparing actual: how many healthy ones right now?
• A Pod crashes, or a whole node dies → actual drops to 2 → gap → create a new Pod, back to 3.

You don’t get woken up at night to restart the service, because the loop did it for you. It’s also why the previous post said short-lived Pods are a feature: precisely because they’re disposable, a broken one can be swapped for a new one painlessly.

Rolling updates: turning “deployment” into routine

The other half of what makes a Deployment valuable is changing versions without interrupting service. Change the image from v1 to v2, and it doesn’t kill everything and restart at once; instead new ones come up one by one, old ones retire one by one:

The mechanism underneath is the same one again: during an update the Deployment starts a new ReplicaSet (v2), scaling it up while scaling the old ReplicaSet (v1) down. What if it breaks? Because the old ReplicaSet is still there, one kubectl rollout undo takes you back to v1 in seconds.

Day-to-day operation is really just these few lines, all of them “change the desired state”:

kubectl apply -f web.yaml                    # declare desired state (3 copies of v1)
kubectl set image deploy/web web=web:2.0     # change version → triggers a rolling update
kubectl rollout undo deploy/web              # broke → one-step rollback
kubectl scale deploy/web --replicas=5        # change the count → loop fills up to 5

Notice: from start to finish you never issued a single “start container” or “stop container” command — you just kept updating that “desired state”, and the reconcile loop converged reality onto it.

A common pit: if I change a ConfigMap / Secret, do the Pods swap automatically? No. The reconcile loop watches the Deployment’s Pod template; when you kubectl edit a referenced ConfigMap/Secret, the template itself hasn’t changed, so the Deployment won’t trigger a rolling update. Pods injected via env keep the old values until you run kubectl rollout restart deploy/web by hand (or add a content-hash annotation to the template so the template changes whenever the value does, rolling automatically). Files mounted via volume do get updated by the kubelet, but the application has to re-read them for the change to take effect. In one sentence: a rolling update only asks “did the template change”, never “did the thing the template points at change”.

Reflections

”Self-healing” isn’t magic; it’s that loop running all the time

The first time you see a Pod get killed and reappear on its own a few seconds later, it really does feel magical. But take it apart and there’s nothing mystical: the ReplicaSet controller just keeps asking “how many actually? how far from desired?” and acts on any difference. Once that clicked, K8s’s “resilience” stopped being black magic for me and became a very plain loop — which also gives my debugging direction: if a service isn’t being pulled back, odds are this loop is stuck on something (not enough resources to schedule, a health check failing forever…), not “voodoo”. Reducing the magical to its mechanism is the first step in how I learn any system.

Rolling updates turn “deployment” from a tense event into routine

I’m all in on this. Deployment used to be a big event: pick the middle of the night, everyone on standby, terrified of interrupting service. With a Deployment’s rolling update plus one-step rollback, deploying becomes a low-risk routine action — the new version slowly takes over, and if it breaks you’re back in seconds. It’s the same sense of safety as the “Production jobs must be idempotent and re-runnable” I described in the Airflow post: make “change” reversible and controllable, and people dare to move forward in frequent small steps, instead of hoarding a giant bundle and betting it all once.

You declare “what you want”, not “how to do it” — Deployment is the best demonstration

The spine of the whole series is most concrete in this post: what you give a Deployment is always a target state (3 copies, v2), never steps (start this first, then stop that). That “desired state” can also go into Git, be versioned, be reviewed — the declarative + GitOps dividend from the first post. Service, StatefulSet, HPA — everything you meet later is the same pattern applied differently. Hold on to “declare the desired state, let the loop converge”, and the rest of K8s is variations on a theme.