K8s Storage: Volumes, PV/PVC and StatefulSets

Pods are short-lived and disposable — self-healing swaps them out at any moment. But some things must not vanish when a pod dies: data. And a container’s filesystem is inherently ephemeral: reschedule the pod or restart the container and whatever was written inside is gone. So K8s has to decouple “storage” from “the pod’s lifecycle”. This post covers its three storage layers: Volume, PV/PVC, StatefulSet.

Volume: mount a disk into the pod first

The most basic layer is the Volume — a piece of storage mounted at some path inside the pod. There are several types: emptyDir (scratch space for the pod’s lifetime, gone when the pod goes; good for sharing temp files between containers), hostPath (a path on the host; rarely used, ties you to the node), and the one that really matters — persistent storage that “outlives the pod”, mounted through a PVC. The first two aren’t persistent; for data to survive the pod’s death, read on to PV/PVC.

PV / PVC: separate “supply” from “demand”

The central design of K8s storage is splitting “who needs storage” and “where the storage actually is” into two separate things:

Three details the CKA loves and practice needs are in the diagram: access modes decide “how many nodes can mount at once, and can they write” — most common are RWO (ReadWriteOnce, read-write from a single node, the nature of ordinary block storage) and RWX (ReadWriteMany, read-write from many nodes at once, which needs file storage like NFS); reclaim policy decides the fate of the PV after the PVC is deleted — Retain (keep the data for you to handle manually) or Delete (delete the underlying storage too). Get these two wrong and at best you leak resources, at worst your data is deleted automatically.

StatefulSet: so each stateful pod recognises its own disk

With persistent storage in place, one problem remains: for a group of stateful pods, how do you make sure each one mounts back onto “its own” disk? A Deployment can’t — its pods are disposable replicas with random names. The answer is the StatefulSet:

In YAML: one PVC, one StatefulSet

First the “demand” half — a PVC is everything the app developer has to write, and it says nothing about what kind of disk sits underneath, only how big, which access mode, and which StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data }
spec:
  accessModes: [ "ReadWriteOnce" ]     # RWO: read-write from a single node
  storageClassName: fast-ssd           # let this class dynamically provision a PV
  resources:
    requests: { storage: 10Gi }        # I want 10Gi

A stateful service doesn’t hand-write PVCs, though; it uses the StatefulSet’s volumeClaimTemplates — a “PVC mould” that stamps out a dedicated disk for each Pod (pod-0, pod-1…), remounted after a reschedule:

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: db }
spec:
  serviceName: db                      # pair with a headless Service so each Pod gets a stable DNS name
  replicas: 3
  selector: { matchLabels: { app: db } }
  template:
    metadata: { labels: { app: db } }
    spec:
      containers:
        - name: db
          image: postgres:16
          volumeMounts:
            - { name: data, mountPath: /var/lib/postgresql/data }
  volumeClaimTemplates:                # ← the key: each Pod automatically gets its own PVC
    - metadata: { name: data }
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: fast-ssd
        resources: { requests: { storage: 10Gi } }

Two details worth remembering: the PVCs produced by volumeClaimTemplates are named data-db-0, data-db-1… — the name is tied to the Pod’s ordinal, which is how “pod-0 always remounts its own disk” is implemented; and these PVCs are not deleted by default when you scale down — K8s would rather keep the data and wait for your manual confirmation than presume to delete a stateful disk. That’s the exact opposite default from a Deployment’s “the Pod leaves, nothing remains”, and exactly the caution “stateful” deserves.

Reflections

The “supply/demand separation” of PV/PVC is an abstraction I admire

The first time I really understood PV/PVC, I thought the design was beautiful. It takes something tangled together and cuts it cleanly in two: whoever writes the app only needs to say “I want a 10Gi read-write disk” (the PVC), and never needs to know whether underneath it’s AWS EBS, GCP PD, or an NFS box in the machine room. Those details go to the administrator’s PV / StorageClass. This “declare the need, hide the implementation” separation is really the essence of a good interface — like calling an Uber and saying only “from A to B”, without caring what car the driver has or which route they take. Whenever I design a system boundary now, I think of the PVC: let the consumer speak in “the language of needs” rather than forcing them to understand the supply side’s details — the most effective move I know for reducing coupling.

StatefulSet lets “short-lived pods” safely own “long-lived data”

K8s first gives the impression that “everything is disposable, everything is stateless” — a pod dies, replace it and move on. But the real world has data, and data can’t be disposable. The StatefulSet cleverly reconciles the contradiction: the pod itself can still die and be replaced, but its “identity” and “its disk” are stable — the replacement pod-0 remounts the original pod-0’s PVC. That made something click for me: “disposable” and “stateful” aren’t a black-and-white opposition. You can make the executing shell disposable (the pod) while making the data it guards persistent (the PV) — separating “what can die” from “what can’t” with a layer of abstraction, and treating each the way it deserves. It echoes the spine of the whole infra series: what’s hard about stateful things is that disk, and the StatefulSet is k8s’s standard answer to it.

Storage is the key step in K8s growing from “running containers” to “running real systems”

Early on, many people said “don’t put stateful things on K8s”, because storage wasn’t mature back then. The arrival of the whole PV/PVC/StorageClass/StatefulSet set is exactly the watershed where K8s crossed from “only good for stateless web services” to “able to run databases, message queues, real systems”. My takeaway: how mature a platform is often shows in how it handles “state”, the hardest bone of all. Anyone can schedule stateless things; the real difficulty is always “how does the data follow along safely”. K8s took several releases and several abstractions to make storage solid — a reminder that when evaluating any platform that “claims to run anything”, the first place to poke is whether its storage and state management are actually hard enough.