Structured Streaming: Treating a Stream as One Unbounded Table

The first four posts were all about batch Spark — what it is, DataFrames in practice, shuffle and tuning, getting it running. But data doesn’t wait until you’ve assembled a batch: orders, clicks and logs flow in continuously. This post is about how Spark handles streams — Structured Streaming — and its cleverest move is telling you that you don’t have to learn a new streaming API at all.

The core idea: a stream is a table that keeps growing taller

Traditional streaming frameworks make you think in a completely different way from batch — event by event, callbacks, state you manage by hand. Structured Streaming inverts that, and its core abstraction is a single sentence:

Think of a stream as an unbounded input table: every new batch of data that arrives is a few more rows appended at the tail.

So the query you run against it is identical to the one you’d run against a static DataFrame — select, filter, groupBy, join. Behind the scenes Spark turns that query into a continuous job that recomputes incrementally and updates the result whenever new data lands. You write a batch query, and Spark makes it a stream.

A minimal streaming program looks almost exactly like the batch one

Side by side it’s obvious. Batch, reading a file:

df = spark.read.format("json").load("/data/events")
counts = df.groupBy("user_id").count()
counts.write.format("console").save()

As a stream, only read → readStream and write → writeStream change; the query in between is untouched:

df = spark.readStream.format("kafka").option(...).load()
counts = df.groupBy("user_id").count() # exactly the same transformation
query = counts.writeStream.outputMode("complete").format("console").start()
query.awaitTermination()

A readStream source is commonly Kafka (picking up the Kafka ecosystem post: Spark is one of Kafka’s most typical downstream consumers), a file directory, or a socket. The groupBy().count() in the middle is the same DataFrame API as batch — every shuffle and wide-versus-narrow instinct you built applies unchanged in a stream.

Three problems only streams have

Treating a stream as a table is lovely, but “data keeps coming, and some of it arrives late” brings three problems batch doesn’t have:

1. Output mode: what to emit each time

The result table keeps changing, so which rows do you write downstream?

• Append: only rows newly settled this round that won’t change again. Suits transformations without aggregation, or aggregations paired with a watermark.
• Complete: emit the whole result table each time. Suits cases that need the full aggregate (as long as the result table can’t grow without bound).
• Update: only the rows that changed this round. The cheapest, and common for stateful aggregations written to an updatable sink.

2. Event time and watermarks: what to do about late data

The key distinction: event time (when the thing actually happened) vs processing time (when Spark received it). Network delay and offline devices mean an event that “happened at 10:00” can arrive at 10:05 — and if you want to window by event time (orders per minute, say), you have to handle that lateness.

A watermark is a promise you make to Spark: “anything more than N minutes behind the largest event time I’ve seen, I consider too late and stop waiting for”. It draws a moving line:

• Late data still inside the line → correctly folded into its time window.
• Too late, past the line → dropped.

It resolves a very practical dilemma: without a watermark, Spark has to keep the state of every time window forever in case a late row that may never come turns up → state explodes and OOMs. A watermark lets expired windows close safely and release their state.

3. Checkpoints: how it picks up after a crash

A stream is a long-lived job, and it will be restarted at some point. Structured Streaming uses a checkpoint (written to reliable storage such as HDFS or S3) to record how far it has read and what its aggregation state is. Paired with a replayable source (Kafka can re-read by offset), that gives exactly-once end to end — after a crash it resumes from the checkpoint with nothing duplicated and nothing lost. Same thinking as the Kafka delivery guarantees post: a replayable source plus a consumer that remembers its progress = nothing lost, nothing duplicated.

Micro-batch: it’s really “batch, but fast”

One misconception to clear up: by default Structured Streaming does not process event by event — it uses micro-batches, cutting the incoming flow into very small batches (triggered as soon as data is there, by default) and running each through the Spark engine. That’s how it reuses the whole batch optimization stack (Catalyst, Tungsten, AQE), at the price of latency in the hundreds of milliseconds to seconds, rather than true per-record milliseconds.

Which is the biggest dividing line between it and Kafka Streams from the previous series: Kafka Streams is embedded in your app, per record, milliseconds; Spark Structured Streaming is cluster-scale micro-batch, and what you get in exchange is one set of DataFrame code covering batch and streaming, plus enormous throughput. (Spark also has a low-latency Continuous Processing mode, but it’s limited in what it supports and rarely used in practice.)

Reflection

”A stream is an unbounded table” is one of the finest abstractions I’ve met

When I first came to streaming I assumed I’d have to relearn how to think — and Structured Streaming told me: the batch you already know is the stream. That abstraction — hiding the time dimension inside a table that grows — let me carry my existing DataFrame, SQL and shuffle instincts over wholesale. What I took from it later is that a good abstraction doesn’t give you more new concepts, it saves you from learning new concepts: it buries the hard parts (incremental computation, state management, fault tolerance) underneath, and what surfaces is still the table you know. That’s worth more than any feature the framework has.

The hard part isn’t the API, it’s event time and lateness

Writing readStream.groupBy isn’t hard. Working out which time you’re counting by, and how late you’ll tolerate is. The streaming bugs I’ve seen most aren’t syntax errors — they’re using processing time while pretending it’s event time, so the moment data arrives late the numbers are quietly wrong and almost impossible to spot. A watermark looks like an OOM-prevention knob and is really a way of forcing you to answer a business question head on: how late can data be and still count? There’s no default answer to that; you have to settle it with the business. I now ask it before writing any streaming code.

Not everything called “real time” needs a stream

Streaming is seductive, but it’s a long-lived job — checkpoints, state, monitoring, restarts — and its operational cost sits a notch above batch. My test is to ask first whether the latency requirement is seconds or minutes. Plenty of requirements that call themselves “real time” are entirely satisfied by a batch job every 5 minutes (scheduled with Airflow), which is easier to maintain and far easier to debug. Only when you genuinely need to get down to seconds, on a sustained High-throughput feed, is it worth paying streaming’s operations tax. Confirm you really need a stream before you build one — the same note this whole series keeps hitting: a tool is there to multiply output, not to add debt.