Moving Data In: Batch or Streaming? Reading Fundamentals of Data Engineering, Ch. 7

The source produces data, storage is ready to receive it, and the action in between that moves data in is this chapter’s subject: ingestion. It’s the second stop in the lifecycle, and the place where most people immediately agonise over “do we need real time”. The line to think through first — batch or streaming isn’t a matter of technical taste, it’s a matter of business value.

Ask the right questions first: the key axes of ingestion

The book’s reminder: before choosing any tool, think through these dimensions. Together they decide what your ingestion looks like:

Dimension	What it asks
Frequency	Batch? Streaming? Or micro-batch in between? (next section’s subject)
Bounded / unbounded	A fixed file, or an event stream that never ends?
Push / pull	Does the source send, or do you go and fetch? (see below)
Sync / async	Wait for it to finish, or fire and forget?
Payload	How big, what format, will the schema change? (the pain of the source chapter)
Reliability	What happens if one record is lost? Can you resend, dedupe?

The answers to these questions nearly all point to the same core decision: how real time?

The core spectrum: batch ↔ micro-batch ↔ streaming

“Batch or streaming” isn’t really a binary choice; it’s a spectrum. Every step towards real time costs one more unit of complexity and money:

Micro-batch (like Spark Structured Streaming) is a very sweet middle point in practice — “one small batch every few seconds” approaches streaming’s immediacy while largely keeping the familiar batch mindset and tooling. Many “we need real time” requirements are in fact satisfied by micro-batch.

Two ways to extract in batch: snapshot vs incremental

Going batch, there’s one more common trap — do you move the whole thing each time, or only what’s new?

• Full snapshot: grab the current state of the whole table every time. Simple and easy to reason about, but heavy, slow and expensive once the data grows.
• Incremental (differential): grab only what changed since last time. Saves a lot, but you need a way to know “what changed” — timestamps, incrementing ids, or more elegantly CDC (reading the database change log).

Small data: snapshots for peace of mind; once it grows, switch to incremental. That’s the line between batch ingestion that runs and batch ingestion you can afford to run.

Push, pull or poll: who initiates

Another dimension that’s easy to confuse: when data moves, who acts first? The book gives three patterns:

These three aren’t just vocabulary: push usually goes with streaming and low latency, pull / poll with batch. Poll too often and you crush the source; too rarely and latency climbs — so when you can push (or use CDC), it’s usually more elegant than polling furiously.

Should you go streaming? The book’s answer is restrained

By now you may be thinking: just go streaming then, real time is great. But the book’s position (and mine) is conservative: batch is still the default today; streaming is the exception you adopt for business value. Two questions to ask yourself first:

1. Is real time actually valuable? If a report seen tomorrow and a report seen in five seconds make no difference, streaming only adds cost.
2. Can downstream use real time? You work hard to stream data in, and downstream still runs a batch analysis once a day — the real-time-ness was wasted halfway.

Unless both answers are “yes”, don’t go real time for the sake of real time. That’s exactly in tune with my confirm the pain first, then bring the heavy weapons — streaming is a heavy weapon; confirm you’re really in pain first.

How to actually move data in: don’t reinvent the wheel

Finally, the practical means of transport. The book lists a whole row; I’ve ordered them by “how much of the low level you touch”:

Method	Scenario
Direct DB connection (JDBC/ODBC)	The rawest: query and move it yourself
CDC	Read the change log, near real time without loading the primary
API	The standard entry point for third-party SaaS
Messages / event streams	Real-time events, see Kafka
Managed connectors (Fivetran, Airbyte)	Common sources pre-wired, no maintenance of your own

The last row is the point: ingestion is the kind of “everyone does it, nothing differentiating” heavy lifting where managed connectors come first; don’t hand-carve a pile of API integrations and then maintain them yourself. That’s Ch. 4‘s “default to buy / use off-the-shelf” landing at the ingestion stop.

Reflections

”Do we need real time” is the question I’ve seen asked wrong most often

Almost every data request opens with “I need it real time”. But this chapter reframes the question: first ask whether real time has value and whether downstream can consume it, then decide the frequency. I’ve met too many “we need real time” requirements where digging down revealed the user looks at the report once a day — so streaming was purely making operations work for myself. My default is now reversed: first ask whether batch solves it; only if not, and real time has genuine business value, climb to micro-batch and then to streaming. Batch isn’t backward; until it’s been proven insufficient, it’s the most rational choice.

Micro-batch is the “just right” I recommend most often

When something genuinely needs to be faster, I rarely jump straight to record-by-record streaming; I try micro-batch first. Structured Streaming‘s “one small batch every few seconds” largely keeps batch’s mindset and debugging approach while pushing latency down to seconds — more than enough for the vast majority of “we’d like it a bit more real time” requirements, and it dodges the whole complexity of pure streaming: state, out-of-order events, exactly-once. Buying enough immediacy with the smallest increment of complexity is a principle I use again and again in ingestion.

Push and CDC saved me from a lot of polling disasters

Early on I loved polling for ingestion — ask the source once a minute, simple and intuitive. But as volume grew it became a dilemma: ask too often and you crush the source, too rarely and latency climbs. Then I learned push where you can, and CDC where you can read the log, and many of those dilemmas simply vanished: data flows over as soon as it changes, and I don’t have to keep knocking on the door. It’s the same cleverness as the previous chapter‘s “borrow the log the database already writes” — rather than polling madly yourself, let the source tell you at the right moment.