How to Actually Choose Technology: Reading Fundamentals of Data Engineering, Ch. 4
· tech
📑 Contents
- An all-too-common mistake: falling in love with the tool first
- Core one: the immutable foundation vs the transitory surface
- Core two: build vs buy
- Cost has two ledgers: TCO and TOCO
- Reflections
- ”Architecture before tools” is the sentence I most needed to shout at myself this half-year
- Where I’ve placed my bets is almost all on the “immutable foundation”
- Build vs buy: I’ve fallen flat “building a wheel to prove I could”
The previous post was about architecture (the why); this chapter asks next: under that architecture, how do you actually choose the technology (the how)? The line to nail into your head first — architecture first, then technology, never the reverse. Tools are means, led by the architecture’s trade-offs; if the first question is “which tool should we use”, the order is already wrong.
An all-too-common mistake: falling in love with the tool first
A lot of teams decide like this: see a hot tool → decide to use it → then go back and fit the architecture around it. The book flips it straight over: architectural decisions (why the system is cut this way, what you’re willing to trade) come first; technology decisions (which product implements it) come second. Tools are options in service of a trade-off, not the starting point.
Then, once you’ve got the order right and are ready to choose, the book gives a whole row of criteria. Condensed into one table:
| Criterion | What it asks |
|---|---|
| Team size and capabilities | Can you afford and tame this thing? |
| Speed to market | How fast can you deliver value? (often underrated) |
| Interoperability | Does it connect to what you already have? |
| Cost | TCO, TOCO, FinOps (see below) |
| Today vs the future | Immutable foundation vs transitory surface (see below) |
| Location | Cloud / on-prem / hybrid |
| Build vs buy | Is this worth building yourself? (see below) |
| Monolith vs modular | One bundled block, or swappable components? |
| Serverless vs servers | Who carries the operations? |
Three rows I think deserve the deepest dig, and that most change how you decide: today vs the future, build vs buy, cost.
Core one: the immutable foundation vs the transitory surface
The book distinguishes two kinds of technology: immutable and transitory. Immutable ones are the underlying layers that have held for decades and won’t disappear soon — object storage, SQL, networking, Unix / bash; transitory ones are the frameworks, libraries and hot tools that come and go, possibly nobody mentioning them three years later.
The advice is clear: anchor the architecture to the immutable foundation, and design the transitory surface to be swappable. Bet thirty years on SQL and object storage and you won’t be far wrong; this year’s hottest framework, don’t let it seep into every corner of the system and bind you. It’s the same nerve as the loose coupling and reversibility of Ch. 3 — bet on the stable things, and let the changeable things be replaced locally.
Core two: build vs buy
The second key criterion: should you build this thing yourself, or buy / use something off the shelf? The book’s rule of thumb is one question — is this your core differentiator?
The book’s position (and mine): default to buy / off-the-shelf; building is the exception. Amazon’s phrase “no undifferentiated heavy lifting” means exactly this — carrying undifferentiated work yourself just piles operational debt onto your future self. It’s the same sentence as Ch. 1‘s “Type A engineer first” in different words: the moment to reinvent the wheel is forced by scale, not used to prove technical chops.
As an aside, “buy” doesn’t only mean “buy a fully commercial product”. It’s a spectrum: community open source → commercialised open source (a managed version where someone carries operations for you) → fully managed proprietary product. The further right, the more you pay to shed operational burden — another trade-off.
Cost has two ledgers: TCO and TOCO
You can’t choose technology without cost, but the book reminds you there’s more than one ledger.
| TCO (total cost of ownership) | TOCO (total opportunity cost of ownership) | |
|---|---|---|
| Asks | How much does using this cost? | Being tied to this and not choosing others — what did you give up? |
| Countable | Licences, machines, people, operations | — |
| Easily missed | Hidden integration and operational cost | Lock-in, no way back, missing better options |
Most people count only TCO (the visible bill), but the book points out the more invisible and often more expensive one is TOCO — choosing A means giving up B, C and D, and if A locks you in, the cost of switching later climbs until there’s no way back. It’s the cost face of the reversibility in Ch. 3: an irreversible choice’s real price isn’t on the invoice, it’s in “can’t change your mind”.
Add the book’s constant emphasis on FinOps — cloud cost isn’t a number fixed once the contract is signed; it’s a variable to keep designing for and watching. Pay-as-you-go is flexible, but it also means the bill grows with every bad query you write.
Reflections
”Architecture before tools” is the sentence I most needed to shout at myself this half-year
I’ve written a whole row of tool notes (Airflow, Spark, Kafka, dbt) and I know the urge well: “learn a flashy tool and badly want somewhere to use it”. This chapter nails the order down: articulate the architectural trade-off first; only then does the tool come on stage. It’s fundamentally the same nerve as my confirm the pain first, then bring the heavy weapons post — without first confirming the pain and the constraints, the trendiest tool is used for the sake of using it. Now, when I review a technology choice, my first line is “which trade-off are we choosing this to resolve”, not “how powerful is it”.
Where I’ve placed my bets is almost all on the “immutable foundation”
Looking back, the choices I feel most at ease with all sit in the immutable layer: insisting on SQL as the transformation language, spreading raw data on object storage, guarding an immutable, replayable Bronze in Medallion. Those will probably still be here in ten years. Conversely, plenty of the frameworks I once thought “we have to use” are no longer maintained — the mercy is that I never let them seep into the core of the system. Bet on stable things and make changeable things disposable is a principle I trust more every year.
Build vs buy: I’ve fallen flat “building a wheel to prove I could”
The “default to off-the-shelf” rule I only truly believed after paying tuition. Early on I hand-built a whole thing for a feature that was actually purchasable — hugely satisfying at the time, and six months later an operational burden nobody wanted to touch. The book’s “undifferentiated heavy lifting” cuts it precisely: the cost of building is never the moment you write it, it’s every year of maintenance after. Now my default is buy / managed, and I only build when “this really is where we pull ahead” — and such places are far fewer than engineers imagine.