Reliable, Scalable, Maintainable: The Three Goals of a Data System

Starting a new series: reading Martin Kleppmann’s Designing Data-Intensive Applications (DDIA). It complements FoDE — FoDE is the practical map of data engineering, DDIA is the theory of why distributed data systems look the way they do. And the skeleton of the whole book is a fundamental question the first chapter asks: what should a good data system actually pursue? The answer is three goals: reliable, scalable, maintainable. This post also happens to be the theoretical source of the reliability line I’m writing in the SRE series.

Modern systems compete on data, not compute

Start with the data-intensive in the title. For most applications today, the bottleneck isn’t a CPU that can’t compute fast enough — it’s data: the volume of it, its complexity, the speed at which it changes. Your system has to store data, query it, remember results, move it between services. So the hard part is no longer “how fast is the algorithm” but “with this much data, how do I store it reliably, query it fast enough, and still be able to change it”. That’s exactly what the three goals answer.

The three goals that decide success

The core idea of Reliability is that the goal isn’t “no errors” but “keeps working despite errors” — fault-tolerant, not fault-eliminating. There’s an important distinction here: a fault is one component deviating from its spec; a failure is the system as a whole stopping to serve users. You can’t stop faults from happening (hardware dies, people slip, software has bugs), but you can design so that a fault doesn’t turn into a failure. The most counter-intuitive and most powerful move is deliberately inducing faults — like Netflix’s Chaos Monkey randomly killing production machines, forcing you to get fault tolerance right on ordinary days. That whole way of thinking is the theoretical foundation of the SRE line.

Scalability is a system’s ability to keep coping as load grows. Its most counter-intuitive point: before you talk about scaling, you have to be able to describe concretely what the load looks like — read/write ratio, where the hot spots are, how big the fan-out is; without measuring those first, “add machines” has nothing to start from. This is the lead-in to the book’s entire Part II, so I’ll expand on it in its own section below.

Maintainability is often neglected, but it decides a system’s long-term cost. Three design principles: Operability (make operations easy — good monitoring, enough automation, echoing eliminating toil), Simplicity (manage complexity, cut the “accidental complexity”), and Evolvability (make the system easy to change, because requirements will).

Of the three, scalability drags in the most questions, so it deserves its own section.

Scalability: first ask what the load looks like

Most people hear “scaling” and think “add machines”. But DDIA’s insight is: the first step in scaling isn’t adding machines, it’s describing your load parameters — read/write ratio, requests per second, the data’s fan-out, the distribution of hot spots. Without understanding the load first, you can’t pick the right architecture. The book’s classic example is the two ways of building Twitter’s home timeline:

There’s one more key to describing “performance”, and it echoes the diagram I drew in the SRE monitoring post directly: look at response time as a percentile (p99), not an average — an average hides the tail, the group of users with the worst experience. That isn’t a coincidence; DDIA and SRE are describing the same thing, one from system design and one from operations. Only then does how to scale come up — scale up (a stronger machine) vs scale out (more machines) — and all the hard problems scaling out brings (where does the data live, how does it sync, what happens when it breaks) are the entire content of the book’s Part II.

Reflections

The “three -ilities” are my default checklist for any system

Reliable, scalable, maintainable — these three non-functional requirements decide a system’s long-term life or death more than “can the feature be built” does. A buggy feature can be fixed, but a system that isn’t reliable, won’t scale and nobody dares change will slowly drag the team down. When I review architecture now I deliberately run these three as a checklist: what happens when it hits a fault? What happens when load grows tenfold? Can someone else take it over and change it in three months? Plenty of designs that look clever in the moment fall apart under those three questions — the most practical pair of glasses I took from this chapter.

The first step in scaling isn’t “add machines”, it’s “describe the load”

An engineer’s reflex when scaling comes up is machines, sharding, Kubernetes. DDIA’s reminder is the step before that: you have to be able to describe your load quantitatively — what’s the read/write ratio? Where are the hot spots? How big is the fan-out? The Twitter example says it best: without first establishing “reads vastly outnumber writes, but celebrity fan-out explodes”, you can’t even choose between fan-out on read and fan-out on write, and no number of machines will help. That’s the same sentence as what I keep saying in confirm the pain before you bring in the heavy weapons — measure the problem first, or the solution means nothing.

Accidental complexity is maintainability’s enemy number one

DDIA splits complexity into two kinds: essential complexity (the problem itself is hard) and accidental complexity (what we made ourselves). The first can’t be avoided; the second can be cut — and most of it comes from over-design, abstractions added to show off, and “flexibility” prepared for things that haven’t happened yet. That’s the same belief as SRE’s Simplicity and as eliminating toil: keep it simple when you can, because the cost of complexity is paid by everyone who maintains it later. I increasingly believe the mark of seniority isn’t “how complex a system I can build” but “how simply I can build what needs building” — leaving a system that the me of three months from now, or the colleague who inherits it, can read and change.