What Spark Performance Actually Is: Shuffle and Tuning

Part one and part two both ended on the same line: Spark performance is shuffle. This post takes it apart — why shuffle is expensive, how it cuts a job into stages, and four moves that genuinely shuffle less and run faster.

Why shuffle is so expensive

Wide transformations like groupBy, join and distinct need the rows for one key gathered together. But those rows start out scattered across different partitions on different executors, so Spark has to write the data to disk → move it over the network to other nodes → read it back and regroup it. Disk plus network plus serialization: every one of those is orders of magnitude slower than computing in memory.

Shuffle also cuts stage boundaries: Spark splits a job into stages at each shuffle, chaining narrow transformations together inside one stage (a pipeline) and starting a new stage whenever it hits a wide one.

So the first principle of tuning is simple: don’t shuffle if you can avoid it, and when you must, move less. All four moves below circle that.

1. Filter and select early, so less data reaches the shuffle

The cheapest and most effective thing there is: cut the data down before the wide transformation. Half as much data into the shuffle is half the network and half the disk. Catalyst does some of this for you (filter pushdown), but writing the filter and selecting only the columns you need up front is both the clearest statement of intent and the safest bet.

2. Broadcast join: don’t shuffle the big table to join a small one

The default join shuffles both tables by key. But when one side is a small dimension table there’s no need for that — copy the small table whole (broadcast it) to every executor, and each partition of the big table joins where it already is, with the big table never moving.

from pyspark.sql.functions import broadcast

big = spark.read.parquet("orders.parquet")      # big
dim = spark.read.parquet("customers.parquet")   # small dimension table

joined = big.join(broadcast(dim), on="customer_id", how="left")

Just don’t let the small side get too big: Spark has a spark.sql.autoBroadcastJoinThreshold (about 10MB by default) that broadcasts small enough tables on its own, and manually broadcast()-ing an oversized one will blow out the driver’s or executors’ memory.

3. Partition count: spark.sql.shuffle.partitions

How many partitions should the data be cut into after a shuffle? The default is 200. Too many → each partition is so small that scheduling overhead swamps the actual work; too few → each partition is too big for memory and spills to disk. The rule of thumb is to land each partition in the one-to-two-hundred MB range.

spark.conf.set("spark.sql.shuffle.partitions", 64)   # scale it to your data; don't cling to 200

4. AQE and cache

AQE (adaptive query execution) is on by default in Spark 3, and it adjusts the shuffle partition count at run time from the actual data volume, coalesces partitions that came out too small, and even handles a skewed join. Trust it first, and only reach for manual tuning when it genuinely isn’t enough:

spark.conf.set("spark.sql.adaptive.enabled", True)   # already on by default in Spark 3

cache / persist: as part one noted, under lazy evaluation calling several actions on the same pipeline recomputes it from scratch. If an intermediate result is used many times, cache it:

paid = df.filter(F.col("status") == "paid").cache()
paid.count()                 # first time: actually computes, and stores it in memory
paid.groupBy("month").count()  # reused after that: served from cache, not recomputed
paid.unpersist()             # release it when you're done

But cache cuts both ways: don’t cache something you use once — it occupies memory for nothing and can push out data that deserved to stay.

Reflection

Think “shuffle less” before you think “tune a parameter”

I’ve watched plenty of people meet a slow Spark job and immediately start spinning shuffle.partitions and adding memory. But nine performance problems in ten come from too much shuffle or too big a shuffle, not from a badly set parameter. My order is fixed: can I filter/select the data smaller, can I broadcast one of these joins away, can I drop a groupBy — push the volume of shuffle down, and the payoff dwarfs haggling over settings. Parameters are the last fine adjustment, not the first move.

Broadcast join is the highest-value single trick

“Big fact table joined to a small dimension table” is a pattern you meet constantly in practice, and a broadcast solves nearly all of them. One broadcast(dim) and the entire shuffle of the big table goes away — the single highest-return move I learned while tuning Spark. The one thing to remember is to keep an eye on the small side’s size, and not broadcast a table that isn’t actually small and blow up memory.

Don’t optimize early — let the Spark UI find the bottleneck

Same attitude I take to Airflow and to any piece of infrastructure: confirm the pain first, then act. Get the logic right, run it once, open the Spark UI, see which stage is slowest and which shuffle reads and writes the most, and go after that one — rather than adjusting things everywhere on a hunch. Since AQE arrived, a lot of what used to need manual tuning is handled automatically; rather than wrestling with it, spend the effort on the more upstream question of whether this shuffle needs to exist at all.