Reading a Spark Execution Plan: What .explain() Is Actually Telling You

· tech

#spark#data-engineering#performance

📑 Contents

Two lines have come up again and again through this series: “trust the Catalyst optimizer” and “open the Spark UI to find the bottleneck”. But there’s a question I never answered: how exactly do you see what Spark did? Trusting the optimizer shouldn’t mean trusting it blindly — you need a way to check “the join I assumed would broadcast, did it? Was my filter actually pushed down?” This post fills in that skill: reading the execution plan.

From one line of code to execution: what happens in between

The DataFrame or SQL you write doesn’t run directly. It first becomes a logical plan of “what you want”, gets rewritten by Catalyst, then lands as a physical plan of “how to do it”, and only then is cut into tasks and run:

your code DataFrame API or Spark SQL — equivalent Logical Plan · “what you want” resolves the tables and columns you named; not optimized yet Catalyst optimizer filter pushdown · prune unused columns · pick a join strategy Physical Plan · “how to do it” Exchange (= shuffle) and the join strategy are settled execution cut into stages / tasks and run on the executors
Every piece of Spark code you write takes this road. DataFrame and SQL converge at the first step, which is why they perform identically — this is the second half of the “compiled into one plan” figure from the DataFrame post. .explain() is what prints this road out for you

.explain(): printing the plan

Put .explain() on the end of any DataFrame and you see its physical plan — without running anything; it’s static analysis:

df = (big.filter(F.col("date") >= "2026-01-01")
         .join(F.broadcast(dim), on="id")
         .groupBy("cat").count())

df.explain()             # physical plan only (what you'll use most)
df.explain(True)         # prints logical → optimized → physical as well
df.explain("formatted")  # sectioned, with column detail — the easiest to read

There’s one rule for reading it: read from the bottom up. The data source (the scan) is at the bottom, then filters, joins, shuffles and aggregations layer upward, and the final result is at the top. Data flows upward.

The three words to watch in a physical plan

A plan has a lot of words in it, but nine tenths of your performance judgment comes from three of them:

HashAggregate ← the groupBy result Exchange hashpartitioning(cat) BroadcastHashJoin [id] Filter date ≥ 2026-01-01 FileScan parquet · big PushedFilters: [date ≥ …] FileScan dim → broadcast Exchange = shuffle! the priciest step in the query small side broadcast, big joined in place ✓ big table never moves — one shuffle saved filter pushed down to the scan ✓ Parquet reads only matching rows, skips the rest read bottom-up ↑
The physical plan for that query. Bottom up: scan → filter → join → shuffle → aggregate. Three words are enough — Exchange is a shuffle, BroadcastHashJoin is the good join, PushedFilters tells you whether the filter went down

Printed out it looks roughly like this (the trimmed explain("formatted") version):

*(3) HashAggregate(keys=[cat], functions=[count(1)])
+- Exchange hashpartitioning(cat, 200)                    ← the shuffle is here
   +- *(2) HashAggregate(keys=[cat], functions=[partial_count(1)])
      +- *(2) BroadcastHashJoin [id], [id], Inner, BuildRight   ← broadcast join, big table stayed ✓
         :- *(2) Filter (date >= 2026-01-01)
         :  +- *(1) FileScan parquet big[id,cat,date]
         :        PushedFilters: [GreaterThanOrEqual(date, 2026-01-01)]  ← filter pushed down ✓
         +- BroadcastExchange
            +- *(1) FileScan parquet dim[id]

What those three words mean maps straight onto what you already learned:

  • Exchange = one shuffle. This is the word to hunt for in a plan — count the Exchanges and you roughly know where this query’s money goes. groupBy, a non-broadcast join, distinct and repartition all produce one.
  • BroadcastHashJoin vs SortMergeJoin: the first is a broadcast join — the small table is copied to every node and the big one is joined where it sits, with no shuffle; the second shuffles both sides by key. If a join you expected to broadcast shows up as SortMergeJoin, the small table went over the threshold and wasn’t broadcast — the most commonly missed performance problem there is.
  • PushedFilters: whether your filter made it down to the scan. When it does, a columnar format like Parquet skips non-matching data at read time and never loads it; when it doesn’t, everything is read in and then filtered, moving a pile of rows for nothing.

One trap: AQE rewrites the plan at run time

.explain() prints the plan before execution. But AQE (adaptive query execution), on by default in Spark 3, rewrites it again while the job runs — coalescing partitions that came out too small, swapping a SortMergeJoin for a broadcast, splitting a skewed partition. So what .explain() showed isn’t necessarily what finally ran. To see the real final plan, go to the SQL / Query page in the Spark UI, where the plan diagram shows the post-AQE shape along with how many rows each node actually handled and how much each shuffle read and wrote. Static .explain() for the structure, the live Spark UI for the facts — use both.

Reflection

”Trust the optimizer” is not “don’t look at what it did”

In the DataFrame post I said to hand optimization to Catalyst, which is smarter than you. That’s true, and it’s easily misread as “it’ll handle it, so I don’t have to care”. Since learning to read .explain() my position has got much more precise: I trust Catalyst, and I verify. I assumed the filter would push down — open the plan and look for PushedFilters. I assumed that join would broadcast — check whether it says BroadcastHashJoin. Four times out of five it did the right thing, and what’s actually slowing you down is usually in the other fifth, where you assumed it would and it didn’t. Trust without blind trust comes down to being able to open the box and look.

Seeing an Exchange should make you tense up

The whole shuffle post was about how expensive shuffle is, and in a plan, shuffle is called Exchange. Once that mapping clicks, reading a plan gets a sense of direction: I stop reading line by line and instead scan for how many Exchangees there are and which operation caused each one, then ask “can this movement be avoided?” — filter it smaller first, broadcast it away, drop a groupBy. Turning the abstract “performance problem” into the concrete “count the Exchanges” is the biggest thing I got out of learning to read plans.

explain is the hypothesis, the Spark UI is the fact

This one I learned by getting burned: reading .explain() alone lets a static plan fool you. It doesn’t know whether your data is actually skewed, or whether one key holds nine tenths of the volume; you only see that by running it and looking at the spread of task durations in the Spark UI. So my order now is — use .explain() first to confirm the structure is right (join strategy, filter pushdown, how many shuffles), then run it and use the Spark UI to check reality (which stage stalls, whether one task runs far longer than its siblings). Same attitude I take to every tool: see where the pain is before you act — and reading the plan is what turns “the pain” from a vague feeling into a node on the screen you can see and point at.