Airflow + Spark on K8s: How Different Nodes Run Different Pods

Airflow handles “when, and in what order” jobs run; Spark handles “getting the big data computed”. When both move onto Kubernetes, the most common confusion is: what exactly is running, is each thing a pod, and which machine did it get thrown onto? This post draws it out in two diagrams — how different nodes on K8s carry the various Airflow and Spark pods.

Nail down three words first: Node, Pod, Scheduler

In the K8s world, remember three roles and everything else connects:

• Node: a real machine (on the cloud, usually a VM). It’s a relatively long-lived hardware resource with fixed CPU and memory ceilings.
• Pod: K8s’s smallest deployable unit, running one (or a few) containers. It’s disposable — finishing, dying, being rescheduled are all normal.
• Scheduler (in the control plane): looks at how much each pod asks for (requests) and any stated preferences (affinity / nodeSelector), then decides which node this pod goes into.

In one line: the Node is the house, the Pod is the tenant, the Scheduler is the agent. Everything about Airflow and Spark on K8s is “who spawns which pods, and which node the Scheduler assigns them to”.

One diagram: who is a pod, and which node it lands on

This diagram is the mental model for the whole post: the cluster is a pool of nodes, Airflow and Spark are just “applications that spawn pods”, and what actually decides who runs on which machine is the Scheduler. The next two sections look at which pods Airflow and Spark each spawn.

Which pods Airflow has on K8s

Airflow’s pods come in two lifespans: long-lived core components, and disposable tasks.

Long-lived pods (Node 1 in the diagram; they run and never stop):

• Scheduler: parses DAGs, decides which task should run now — Airflow’s brain.
• Webserver: the UI.
• Triggerer: runs deferrable operators (waiting on external events without occupying a worker).
• Metadata DB: stores the state of every DAG / task, usually Postgres. It’s the only stateful role in all of Airflow — on K8s it runs as a StatefulSet with a PersistentVolume (the cylinder on Node 1), so the data survives a reschedule or restart. In practice it’s more common to hook up a managed database (RDS / Cloud SQL) and hand the “keep this DB healthy” responsibility to the cloud provider. It’s the exact opposite extreme from the disposable Spark pods: one must remember everything; a crowd can vanish at any moment.

Task pods take the shape of your executor choice, the first decision to understand for Airflow on K8s:

Approach	What one task is	Good for
KubernetesExecutor	The Scheduler starts a pod per task, deleted when done	Bursty task volume, wanting full isolation
CeleryExecutor	Tasks go to long-lived worker pods (a fixed group of workers)	Steady task volume, saving pod start-up latency
KubernetesPodOperator	You write explicitly in the DAG “this task starts a pod running some image”	Tasks that are themselves containerised programs

The key difference: KubernetesExecutor is “Airflow automatically wraps every task in a pod”; KubernetesPodOperator is “you actively tell Airflow to start a pod”. The former governs where a task runs, the latter what a task runs.

Which pods Spark has on K8s

The earlier Spark post covered that Spark is always the Driver + Executor + Cluster Manager triangle. Moved onto K8s, only one thing changes: the Cluster Manager is K8s itself, and Driver and Executors all become pods.

On submit, --master points at the cluster’s API server:

spark-submit \
  --master k8s://https://<api-server>:6443 \
  --deploy-mode cluster \
  --conf spark.executor.instances=4 \
  --conf spark.kubernetes.container.image=myrepo/spark:3.5 \
  jobs/daily_etl.py

What happens next:

1. K8s first creates a Driver pod (Node 2 in the diagram).
2. Once the Driver starts, it calls the K8s API directly to ask for executors, as many as spark.executor.instances says.
3. K8s creates Executor pods, spread by the Scheduler onto nodes with room (both Node 2 and Node 3 in the diagram).
4. When the job finishes, all Spark pods are deleted; the nodes stay for the next job.

So one Spark job’s pods naturally span several nodes — which is exactly why it scales horizontally. When there are more executors than one machine can hold, they spill onto the next.

One more thing that’s easy to overlook: the “data” Spark computes on isn’t in the cluster. Once executors are up, they go to the external Business DB / data source (the operational Postgres, MySQL, warehouse or object storage) to pull data in, compute, and write results back (the cylinder at the bottom of diagram one). It’s a completely different DB from the Airflow Metadata DB above; don’t confuse them:

	Metadata DB	Business DB
What it stores	Airflow’s DAG / task scheduling state	The actual business data to be processed
Who uses it	Airflow Scheduler	Spark Executors reading / writing back
Where	In-cluster (StatefulSet) or managed externally	Almost always outside the cluster, run by the data team / cloud provider
In one line	Remembers “how far the schedule got”	Remembers “what happened in the business”

Putting it together: the pod lifecycle of one DAG run

Chaining Airflow triggering Spark, the common tool is SparkKubernetesOperator: an Airflow task submits a Spark job to K8s, the Driver comes up and spawns Executors, and the Executors finally write the computed results into the Business DB. The whole chain:

Every box on the chain (except the Business DB outside the cluster on the far right) is one pod or a group of pods. Laying the same chain out on a timeline shows most clearly “who is long-lived, who is disposable”:

Understand this diagram and you’ve grasped K8s’s biggest value for data engineering: resources are occupied only while something is actually computing. With no job running, the cluster holds only a few lightweight long-lived Airflow pods; at peak, Executors sprout all at once and are deleted when done. The cost structure is completely different from the traditional approach of keeping a crowd of idle workers around.

Controlling who runs on which node

By default the Scheduler just picks a node “with room”, but in data-engineering scenarios you usually want to arrange things deliberately. The common knobs:

• requests / limits: how much CPU/memory each pod declares. The Scheduler bin-packs by requests; get this number wrong and either nothing schedules or a node gets crammed.
• nodeSelector / affinity: pin “Spark executors” to a “Spark-only node pool” so they don’t fight the Airflow core for resources.
• taints + tolerations: “lock” certain nodes so only pods with a matching toleration can enter — for instance, reserve a pool of high-memory machines for Spark alone.
• Cluster Autoscaler: when executor pods are stuck pending because no node can take them, add nodes automatically; scale back down when idle.

The most practical pattern is the node pooling drawn in diagram one:

Node pool	Runs	Why
on-demand (stable)	Airflow Scheduler / Webserver, Metadata DB, Spark Driver	If these die the whole batch of jobs is toast (especially the stateful DB); they can’t run on machines that get reclaimed
spot / preemptible (cheap)	Spark Executor	A single executor dying means Spark recomputes that piece — tolerable, in exchange for big savings

Whether it can die, and whether it can recover if it does, decides which kind of node it should run on. That’s the design principle to think through most carefully once Airflow + Spark move onto K8s.

Reflections

”Everything is a pod” is this architecture’s greatest liberation

The change I felt most: Spark on YARN and Airflow’s workers used to be two completely different resource worlds — one tuned via YARN queues, the other via Celery worker counts, each with its own temperament. On K8s, they become the same kind of thing: pods that declare requests and get placed on some node by the Scheduler. Monitoring, scheduling, scaling, isolation all converge into one K8s vocabulary. Learn it once, and Airflow, Spark, even dbt containers are managed the same way. That satisfaction of “one abstraction over heterogeneous workloads” is the main reason I’d recommend a team go this way.

Separating long-lived from ephemeral is where you save money and where things blow up

The dividing line in diagram two — “long blue bars vs short bars” — I only truly learned after getting burned: to save money I once threw the Airflow Scheduler onto spot nodes too, and the moment the cloud reclaimed the node the whole schedule stopped and half-run DAGs ended up in a confused state. The lesson is simple — Executors can run on machines that disappear, because Spark recomputes; but roles like the Scheduler and Driver, where “if it dies nobody takes over”, must be pinned to stable on-demand nodes. The criterion is one sentence: if this pod dies, can the system recover by itself? If yes, put it on spot and save money; if no, pay for stability.

Disposable pods force you to build up observability

K8s’s most counter-intuitive side effect: the executor pod that failed has usually already been deleted by the time you want its logs. In the YARN days I’d SSH in and dig through logs; that move is useless here. So the moment you go onto K8s, logs and metrics have to be shipped out in real time (centralised logging, a Spark History Server and the like); you can’t rely on “fetch them from the machine afterwards”. It’s the same thinking as the idempotency and re-runnability in the Airflow post — a Production system can’t assume any machine or any pod will still be there afterwards.

Same as always: confirm the pain before going onto K8s

Cold water to finish. K8s makes this architecture sound beautiful, but it is itself a mountain that needs operating. When the data isn’t yet big enough to span machines and the team has no K8s background, forcing it in just swaps the trouble of “tuning Spark” for the double trouble of “tuning Spark + fixing K8s”. My priority is always confirm the pain first, then bring in the heavy weapons: use a managed platform (Databricks, EMR, Glue) as long as it holds up; only when you truly need to pack many kinds of workload into one elastic pool, and have people who can look after it, is it K8s’s moment to shine. The unified abstraction is its reward; the operating cost is its entry fee — count both.