Reading a Spark Execution Plan: What .explain() Is Actually Telling You
· tech · 12 min read · 📚 Spark — Learning Notes #6
Two lines have come up again and again through this series: "trust the Catalyst optimizer" and "open…
Tech
Service: A Fixed Address in Front of Short-Lived Pods
· tech · 13 min read · 📚 Kubernetes — Learning Notes #4
The second post left a question open: if Pods are short-lived and get a new IP when replaced, how do…
Your SQL Doesn't Run in the Order You Wrote It
· tech · 11 min read · 📚 SQL: I Thought I Knew It #1
When you write SQL, you almost always start with SELECT. Write it long enough and it feels natural t…
The Future of Data Engineering: Tools Change, the Foundation Doesn't, Reading Fundamentals of Data Engineering, Ch. 11 (Finale)
· tech · 10 min read · 📚 Fundamentals of Data Engineering — Reading Notes #11
Eleven chapters in, one last question: what will the future of data engineering look like? The book'…
Deployments and Self-Healing: The Reconcile Loop in Practice
· tech · 15 min read · 📚 Kubernetes — Learning Notes #3
The first post gave the soul (the reconcile loop), the second gave the atom (the Pod). But in practi…
Pod, Node, Scheduler: The Three Atoms of a Kubernetes Cluster
· tech · 12 min read · 📚 Kubernetes — Learning Notes #2
The previous post covered the soul of K8s: you declare a desired state, and the reconcile loop keeps…
What Is Kubernetes? From 'Running Containers' to 'Declaring the State You Want'
· tech · 13 min read · 📚 Kubernetes — Learning Notes #1
Many people find Kubernetes (K8s) hard because they're buried from day one under kubectl, a pile of …
The Chapter That Matters Most and Gets Ignored Most: Security and Privacy, Reading Fundamentals of Data Engineering, Ch. 10
· tech · 12 min read · 📚 Fundamentals of Data Engineering — Reading Notes #10
Past serving, the lifecycle still has one undercurrent running through the whole thing that hasn't b…
The Last Mile of Data: Serving Analytics and ML, Reading Fundamentals of Data Engineering, Ch. 9
· tech · 13 min read · 📚 Fundamentals of Data Engineering — Reading Notes #9
The previous chapters walked through the source, storage, ingestion, modeling — but all that work on…
Making Data Useful: Queries, Modeling and Transformation, Reading Fundamentals of Data Engineering, Ch. 8
· tech · 14 min read · 📚 Fundamentals of Data Engineering — Reading Notes #8
Data has been ingested and stored; the next question is: how do you turn it into something genuinely…
Moving Data In: Batch or Streaming? Reading Fundamentals of Data Engineering, Ch. 7
· tech · 15 min read · 📚 Fundamentals of Data Engineering — Reading Notes #7
The source produces data, storage is ready to receive it, and the action in between that moves data …
Where to Store Data: The Storage Hierarchy and Its Abstractions, Reading Fundamentals of Data Engineering, Ch. 6
· tech · 21 min read · 📚 Fundamentals of Data Engineering — Reading Notes #6
The previous post was about data being born at the source. The first thing after it's born: where do…
Airflow + Spark on K8s: How Different Nodes Run Different Pods
· tech · 24 min read · 📚 Kubernetes — Learning Notes #8
Airflow handles "when, and in what order" jobs run; Spark handles "getting the big data computed". W…
Where Data Comes From: Source Systems and Data Generation, Reading Fundamentals of Data Engineering, Ch. 5
· tech · 18 min read · 📚 Fundamentals of Data Engineering — Reading Notes #5
The first four chapters were the big picture — the lifecycle, architecture, choosing technology. Fro…
How to Actually Choose Technology: Reading Fundamentals of Data Engineering, Ch. 4
· tech · 15 min read · 📚 Fundamentals of Data Engineering — Reading Notes #4
The previous post was about architecture (the why); this chapter asks next: under that architecture,…
Structured Streaming: Treating a Stream as One Unbounded Table
· tech · 16 min read · 📚 Spark — Learning Notes #5
The first four posts were all about batch Spark — what it is, DataFrames in practice, shuffle and tu…
Designing Good Data Architecture: Reading Fundamentals of Data Engineering, Ch. 3
· tech · 13 min read · 📚 Fundamentals of Data Engineering — Reading Notes #3
The previous post gave the whole lifecycle (what data is doing); this chapter asks — how do you desi…
#data-engineering#book-notes#architecture
The Data Engineering Lifecycle: Reading Fundamentals of Data Engineering, Ch. 2
· tech · 10 min read · 📚 Fundamentals of Data Engineering — Reading Notes #2
The previous post gave the definition; this chapter gives the skeleton of the whole book — the data …
What Data Engineering Is: Reading Fundamentals of Data Engineering, Ch. 1
· tech · 12 min read · 📚 Fundamentals of Data Engineering — Reading Notes #1
I'm starting a new series, reading Joe Reis and Matt Housley's Fundamentals of Data Engineering. The…
Confirm the Pain Before You Bring In the Heavy Weapons
· tech · 7 min read
This is a concept note — a judgment call I keep reusing, split out into its own post so other articl…