Encoding and Evolution: Letting Old and New Code Read Each Other's Data

· tech

#distributed-systems#book-notes#data-engineering

📑 Contents

The previous post was about how data gets onto disk. This one is about a problem that’s easier to underrate: when data is written out, what format is it encoded in? You might think “JSON’s fine” — until the day the schema has to change. The weight of this chapter comes from two facts you can’t escape: data outlives code (you can replace all your code today, but the row written to the database five years ago is still lying there), and old and new versions of the code run at the same time (with rolling upgrades that’s the norm, not an accident). Put those together and “encoding format” stops being a detail and becomes compatibility engineering across versions and across time.

Why you need two kinds of compatibility: rolling upgrades force old and new into the same moment

Mid rolling upgrade: old and new both running, data flows both ways new code v2the nodes already upgraded old code v1the nodes not yet reached same database / topic both sides write, both sides read backward compatibility new code can read old data everyone remembers this (migration mindset) forward compatibility old code can read new data most forgotten — but daily in rollout and rollback Every schema change needs both — miss one and errors start mid-rollout
During a rolling upgrade, new and old code run at the same time and share one database or topic — so data flows both ways. Backward compatibility (new reads old) everyone remembers, because that's the daily life of migrations; forward compatibility (old reads new) is the half most often forgotten — yet it happens every day, in every rollout window and on every rollback (you've gone back to the old version, but the data the new one already wrote is still there!). Every schema change has to cover both

First, let’s finish with JSON: it’s human-readable and works everywhere, but it has no schema enforcement (rename a field or change a type and nothing stops you at compile time — it blows up at runtime), its numbers are vague (big-integer precision, no int/float distinction), and it’s fat. For a small system, who cares; once data has to cross teams and services and live for years, you need a “binary format with a schema” — and that’s where Thrift, Protocol Buffers and Avro come in.

Field tags: the little mechanism that makes evolution safe

The core cleverness in Protobuf/Thrift is that the encoding doesn’t write field names, only a “field tag” (a number). The schema is a manual each side holds; the tag is a coordinate inside the data. And that little mechanism is exactly what lets the schema evolve safely:

No field names in the encoding, only tags — so evolution is safe schema v2 1: name 2: email 3: phone (new in v2, optional) data stores only tag number + value, never the name old code (knows tags 1, 2) reads new data hits unknown tag 3 → skip it name / email read fine → forward compatible ✓ new code reads old data (no tag 3) tag 3 missing → use the default doesn't blow up → backward compatible ✓ Rules: new field → new tag · never change/reuse a tag · new fields optional or with a default
Because the encoding holds only tag numbers, not names: when old code reads new data and hits a tag it doesn't know, it simply skips it and reads the rest (forward compatibility); when new code reads old data and can't find the new tag, it uses the default (backward compatibility). The whole safety of evolution rests on three iron rules: a new field always gets a new tag, a tag can never be changed or reused (it's the coordinate of old data!), and a new field must be optional or have a default. As a side effect, renaming a field is safe — there was never a name in the encoding

Avro takes the more extreme road: the encoding has no tags at all — values just follow one another — so it’s the most compact, but reading requires holding the writer’s schema (from when the data was written) against the reader’s schema (what you expect now) and resolving them; Avro reconciles the differences between the two versions (fields matched by name, missing ones filled with defaults). This “reconcile two schemas at read time” design makes it especially suited to schemas that change often or are generated dynamically (dumping a whole database, say) — which is why it became the mainstream in big data and the Kafka Schema Registry ecosystem: the registry manages schema versions centrally and checks every change for compatibility, turning the discipline in this chapter into an automatic gate.

Reflections

Data outlives code — compatibility is an API across time

The line that hit me hardest in this chapter is data outlives code. You can replace every line of code today, but the row written to the database five years ago, the event that landed in the log three years ago, are still lying there untouched, waiting to be read some day in the future. So the essence of schema compatibility isn’t “a small matter of format” — it’s an API contract you sign with your past and future selves: backward compatibility is taking responsibility for the past, forward compatibility is humility towards the future. Once that clicked, I treat a schema change with the same seriousness as a breaking change to a public API: changing one field is changing an interface that every piece of historical data references.

Forward compatibility is the half most easily forgotten — and it happens every day

Everyone has a sense for backward compatibility (the migration mindset); forward compatibility — old code reading new data — is the half that actually blows up in practice. It happens in two windows you can’t avoid: mid-rollout (an old instance reads data a new instance just wrote), and after a rollback (you’re back on the old version, but the data the new one already wrote is still there!). My lesson was to make it a deployment discipline: ship schema changes and code changes separately, schema first — first a version that “can read the new format but doesn’t write it yet”, confirm it’s fully rolled out, then start writing the new format. It’s the other face of the “small steps, always reversible” safety of a rolling upgrade: code can be rolled back, data can’t — so every step in a data format has to be one that both the version before and the version after can catch.

There’s no such thing as schemaless — only a schema nobody wrote down

The longer I work in data, the less I believe “schemaless is freedom”. The moment data is read, someone holds an expectation about its structure — the schema always exists; the only difference is whether it’s written down explicitly and guarded by someone, or scattered through every reader’s code and held together by tacit understanding. Schema-on-read from two posts ago defers the check, it doesn’t make the structure disappear; and that “freedom” usually means “the writer got free, the reader cleans up the mess at 3am”. So my position is clear: if data crosses teams or crosses time, make the contract explicit — Protobuf/Avro schema files in version control, Schema Registry automatically blocking incompatible changes. It’s the same discipline I’ve been arguing for from declarative config in version control to dashboards as code: an agreement that matters cannot live in someone’s head.