What Data Engineering Is: Reading Fundamentals of Data Engineering, Ch. 1
· tech
📑 Contents
- The definition: the systems engineering that turns raw data into “usable”
- The Data Science Hierarchy of Needs: DE is the foundation
- Two ends of a spectrum: Type A vs Type B data engineers
- Data maturity: one job title, three different jobs
- Reflections
- ”Knowing the tools” is not “knowing data engineering”
- Type A first — the same thing I’ve been saying all along
- The hierarchy of needs is the diagram I’d most like to see pinned on a wall
I’m starting a new series, reading Joe Reis and Matt Housley’s Fundamentals of Data Engineering. The book’s greatest value is that it takes “data engineering”, a function usually described in vague terms, and condenses it into a clear framework and vocabulary. The first chapter answers the most basic questions: what data engineering actually is, what a data engineer actually does, and where they stand in the wider world of data.
The definition: the systems engineering that turns raw data into “usable”
The book’s definition is worth writing down: data engineering is “the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases” (analytics, ML). It sits at the intersection of six areas — security, data management, DataOps, data architecture, orchestration, software engineering.
The key realisation: the output of data engineering isn’t “data”, it’s “trustworthy, usable data systems”. The weight is on the words “systems and processes” — it isn’t a one-off move of data from A to B, it’s building a pipeline that keeps producing clean data reliably. That matches exactly the conclusion I reached writing about dbt: what it sells isn’t “transforming data with SQL”, it’s engineering discipline.
The Data Science Hierarchy of Needs: DE is the foundation
The most famous diagram in the book is borrowed from Monica Rogati: the “Data Science Hierarchy of Needs” — modelled on Maslow’s hierarchy, it shows that before you do ML / AI, the data engineering underneath has to be solid.
The message is brutal: everyone wants the glamorous AI layer at the top, but nine out of ten failures happen because the foundation wasn’t laid. The value of a data engineer is exactly holding up those bottom layers of the pyramid.
Two ends of a spectrum: Type A vs Type B data engineers
The book has a memorable way of describing a data engineer’s orientation:
| Type A (Abstraction) | Type B (Build) | |
|---|---|---|
| Core | Abstract: use off-the-shelf, managed solutions wherever possible | Build: make your own tools and frameworks |
| Mindset | Avoid “undifferentiated heavy lifting” | Reinvent the wheel for scale and internal needs |
| Suits | Most teams, early to mid stage | Large scale, when off-the-shelf really isn’t enough |
The book’s position (and mine): most people should start as Type A — don’t reinvent wheels from day one; save your effort for problems that truly differentiate you. Type B isn’t more advanced; it’s what you need only when “the scale is there and off-the-shelf really doesn’t cut it”.
Data maturity: one job title, three different jobs
The book also has a very practical three-stage “data maturity” model — a reminder that a “data engineer” at companies of different maturity does wildly different work:
- Starting with data: almost no data infrastructure. The DE is a generalist who touches everything — first make data flow and make it storable.
- Scaling with data: data volume and the team are both growing. The DE starts building scalable, repeatable systems and introducing formal tools and practices.
- Leading with data: data is a core competitive advantage. The DE works on automation and self-service so the organisation can use data at scale.
That explains a common confusion: why two people both called “data engineer” sound like they’re doing completely different jobs — their companies are at different maturity stages.
Reflections
”Knowing the tools” is not “knowing data engineering”
The point I resonated with most is that the book positions data engineering as systems engineering, not “someone who knows Airflow / Spark / Kafka”. Tools are just means — I’ve written a whole row of tool notes over the past six months (Airflow, Spark, Kafka, dbt), but this book reminds me: stringing them into a trustworthy, maintainable system that keeps producing clean data is the real skill. Plenty of people can type the commands; very few can design a data system that doesn’t blow up at 3am.
Type A first — the same thing I’ve been saying all along
The book’s claim that “most people should start as Type A” lines up remarkably with the conclusion of every tool note I’ve written — Airflow, Spark, Kafka all say the same sentence: confirm the pain has reached that scale before bringing out the heavy weapons; if managed or off-the-shelf will do, don’t self-host. The urge to build your own is seductive, but carrying “undifferentiated heavy lifting” yourself usually just piles operational debt onto your future self. The moment for Type B is forced by scale, not used to prove your technical chops.
The hierarchy of needs is the diagram I’d most like to see pinned on a wall
Too many teams (including ones I’ve seen) rush to the ML/AI layer at the top without stabilising collection, movement and aggregation underneath — so the model eats dirty data, garbage in, garbage out. This pyramid gave me a tool for external communication: when someone asks “why can’t we do AI yet”, I can point at it and say “because the foundation isn’t ready”. Data engineering isn’t sexy, but it’s the precondition for everything above it — and that’s why I decided to read this book properly and keep writing this series.