Where Data Comes From: Source Systems and Data Generation, Reading Fundamentals of Data Engineering, Ch. 5

The first four chapters were the big picture — the lifecycle, architecture, choosing technology. From this chapter on, the book walks step by step into each stage of the lifecycle. The first stop is the very front, and the one engineers most easily underrate: how, and where, is data actually born? The line to remember from this chapter — data is born in systems you don’t own; you are always downstream.

Setting the tone: you’re downstream, and someone else controls the source

Data engineers rarely generate data; we receive it — from app developers, SaaS, sensors, other teams. That leads to a fact that runs through the whole chapter: those source systems aren’t yours to manage; you can’t stop them changing their schema, their logic, their format. What you can do is understand them thoroughly and build resilience to their changes.

What kinds of sources there are

The book takes stock of the common source systems. The point of remembering the list is that each has its own temperament, and the ingestion strategy follows:

Source	In one line	Watch out for in ingestion
Application DB (OLTP)	The backend store of an online system, mostly CRUD	Don’t run analytics on it directly (see below)
API / SaaS	A third-party service’s public interface	Rate limits, pagination, the schema is their call
Files / logs	CSV, JSON, log files	Messy formats, no schema guarantee
IoT / sensors	Signals devices emit continuously	Huge volume, out of order, disconnect and resend
Message queues / streams	Events flowing through in real time	Delivery semantics and ordering (see Kafka delivery guarantees)

The book also mentions a source that’s easy to miss: analog to digital. The true origin of much data is the physical world (a sentence, an action), born only at the moment some system “digitises” it — and how that step is done decides the quality of the data you get downstream.

Source databases: they’re OLTP, don’t analyse on them directly

The most common source is the database behind some app, and it’s almost always OLTP — optimised for “lots of small, fast transactions” (place an order, change an address, hit like), with ACID guaranteeing each transaction either succeeds or rolls back. It is not designed for “scan hundreds of millions of rows and aggregate”. Running analytical queries on a Production OLTP database means fighting live users for resources — the classic beginner’s disaster.

Another distinction to get straight is how data leaves a trail:

• CRUD: modify in place, delete in place. After an UPDATE, the old value simply disappears — you see only now, never history.
• Insert-only: every change adds a row, and old versions are all kept. The price is a table that keeps growing, but you get full history and replayability.

CRUD saves space but loses history; insert-only keeps history but grows fast. That trade-off affects how you build snapshots downstream and whether you can go back in time — the same nerve as Medallion guarding an “immutable, replayable” Bronze.

Two roads out of the source database: batch queries vs CDC

If you shouldn’t keep hammering the OLTP with queries, how does data get out? The book covers two main approaches, and the difference is worth drawing clearly:

CDC (change data capture) is the chapter’s key concept: instead of querying the whole table over and over, read the change log the database already writes for itself (the log that exists for recovery anyway) and stream out every insert / update / delete continuously. It’s near real time and adds almost no load to the primary, which is why it’s so often the upstream of an event stream like Kafka — turning a CRUD database’s changes into a replayable stream of events.

Schema drift: data engineering’s eternal pain

The source isn’t yours, and the most concrete pain is that the schema will change. One day an app engineer renames user_name, changes a field from string to object, quietly adds a nesting level — and your downstream pipeline silently breaks or gets polluted. The book distinguishes two kinds of source: fixed schema (relational databases, structure enforced at write time) and schemaless (much NoSQL / JSON, structure hidden in the data and liable to drift at any time). The latter is especially dangerous.

The pragmatic response, where the book and my experience agree: don’t assume the source is stable. Negotiate a data contract for critical sources, monitor schema changes at the ingestion entry point, and make what breaks an alert rather than a report quietly wrong for a week. That’s exactly the “data quality” and “monitoring” undercurrents of the lifecycle landing concretely at the very top of the stream.

Messages and streams, and “time”

If the source is real-time events, you’ll meet two kinds of infrastructure — the book’s distinction matches my Kafka series exactly: message queues (a message is deleted once consumed) vs event streaming platforms (events are retained and replayable). I took the details apart in Topics/Partitions and delivery guarantees, so I won’t repeat them.

The last concept, easy to overlook yet one that bites repeatedly, is time. The same event has three times: event time (when it actually happened), ingestion time (when it entered your system), processing time (when you computed it). The three are almost never equal — networks delay, devices come back online and resend. Confuse them and you’ll compute wrong (using processing time as event time for a daily rollup, say), which is why I stressed watching event time and watermarks in the Structured Streaming post.

Reflections

Realising “you don’t own the source” changed how I write pipelines

What hit me hardest in this chapter was that it nailed down something I vaguely knew but had never said outright: the source belongs to someone else, and it will change without me being told. Early on I wrote ingestion assuming “the format upstream gives is the format”, and one quiet column rename left a report wrong for days before anyone noticed. Now my stance is completely reversed — receive data defensively: validate the schema at the entry point, negotiate data contracts for critical sources, and prefer to argue at the boundary (an alert) over letting errors flow silently downstream. It’s the same wariness as Ch. 4‘s “keep the changeable swappable”, moved to the data layer.

CDC is the best value-for-effort move I’ve seen

The “don’t keep hammering the OLTP” rule I learned the sweet way, through CDC. Replacing a batch that scanned a full table every ten minutes — heavy, and missing changes anyway — with a CDC reading the database log, the load on the primary dropped and data freshness went up, and along the way it became a replayable event stream feeding downstream. Its beauty is “borrowing what the database already writes” — the log that exists for recovery became our source of truth for changes. Once you see this move, many “real time vs don’t load the primary” dilemmas simply dissolve.

The problems at the very top are often people problems, not technical ones

After writing this chapter I’m more certain: at the source, the real difficulty is collaboration, not code. Because the source belongs to another team, when the schema changes and why is a communication problem, not an engineering one. Now I go to the upstream app engineers proactively to align on “which columns are the contract and can’t be changed casually”, and put myself into their change process, rather than waiting passively for things to break. That echoes the softest yet most critical undercurrent in the lifecycle — data engineering is, in the end, a cross-team discipline, and it starts at the very top of the stream.