The Data Engineering Lifecycle: Reading Fundamentals of Data Engineering, Ch. 2

The previous post gave the definition; this chapter gives the skeleton of the whole book — the data engineering lifecycle. The book’s most valuable contribution is using this framework to make “what data engineering does” clear: five stages + six undercurrents. Understand this one diagram and the remaining nine chapters are just details growing on top of it.

Five stages: from data being produced to data being used

The lifecycle cuts “raw data → useful data” into five stages:

Stage	What happens	Maps to what I’ve written
Generation	Data is produced in source systems (DBs, apps, sensors, events)	Kafka (event streams)
Storage	Where data lands — spans the middle three stages	Medallion layers
Ingestion	Moving data from sources into the system	Kafka, batch loads
Transformation	Cleaning, modelling, aggregating into a usable shape	dbt, Spark
Serving	Delivering data to analytics, ML, products	BI / reports / features

Why storage “spans” rather than being a separate step

This is the detail from the chapter most worth remembering: storage isn’t a stop inserted “after ingestion, before transformation”; it’s the underlying layer running through ingestion, transformation and serving. Data is read out and written back repeatedly across those three stages — ingestion writes into storage, transformation reads from storage and writes back, serving pulls from storage. That’s why the book draws it as a spanning bar rather than a box in the middle.

That matches exactly what I found writing about Medallion architecture: Bronze / Silver / Gold aren’t three “steps”, they’re three layers of storage the data flows through in its lifecycle.

Six undercurrents: invisible, yet they decide whether the system collapses

The five stages are “the visible pipeline”; the undercurrents are the underlying practices running through every stage — the book’s word is apt: they’re not on the surface, yet they decide the river’s course and safety.

Undercurrent	What it governs
Security	Least privilege, encryption, no hard-coded credentials
Data Management	Governance, quality, lineage, master data, privacy
DataOps	Automation, observability, incident response (DevOps for data)
Data Architecture	How systems are designed, trade-offs and choices
Orchestration	Scheduling tasks by dependency, monitoring, retrying
Software Engineering	Bringing engineering discipline into data work

Note that Orchestration is where Airflow lives — it’s not a stage, it’s an undercurrent running across all stages. That explains why Airflow is so central: it governs the “rhythm” of the whole lifecycle.

Reflections

The framework’s biggest value is a coordinate system for placing things

The first thing I did after this chapter was throw every tool I’d written about in the past six months onto this diagram: Kafka lands in generation/ingestion, Spark and dbt in transformation, Medallion is the layering of storage, Airflow is the orchestration undercurrent. Suddenly, notes that had each been written on their own had positions on the same map. A framework’s value is often not that it teaches you something new, but that it gives you a coordinate system to see how what you already know connects — that’s what this chapter did for me.

The undercurrents are the line between junior and senior

I increasingly think the five stages are what “anyone sees on day one” — everyone knows data has to be ingested, transformed, served. What really separates people is the undercurrents: whether security (no hard-coded credentials, echoing my Connections post), observability, data quality and lineage — the “invisible” things — got built in. The difference between a demo and a Production system that doesn’t fail at 3am is almost entirely in the undercurrents. Beginners show off a pipeline that runs; veterans guard the undercurrents.

If I could shore up only one undercurrent first, it’d be DataOps

The book lists the six side by side, but if I had to rank them I’d put DataOps first — automation, observability, incident response. The reason is practical: the biggest pain of data systems isn’t “can’t build it”, it’s “it broke and nobody knew, once known it’s hard to diagnose, once diagnosed it’s hard to recover”. It’s the same thing I keep stressing in the Airflow series: idempotent, rerunnable, don’t let resources drain silently. Getting DataOps in place is like fitting the whole lifecycle with dashboards and fuses — only then do the problems in the other undercurrents become visible and fixable.