Operating and Deploying Kafka: KRaft, Retention/Compaction and Monitoring
· tech · 18 min read · 📚 Kafka — Learning Notes #5
The first four posts were about how to use Kafka — the replayable log, the core model, delivery guar…
Tech
The Kafka Ecosystem: Connect, Schema Registry and Streams
· tech · 20 min read · 📚 Kafka — Learning Notes #4
The first three posts were all about the Kafka broker itself — the replayable log, the core model, d…
Kafka's Delivery Guarantees: acks, ISR and At-Least-Once / Exactly-Once
· tech · 19 min read · 📚 Kafka — Learning Notes #3
Part two finished with how events are laid out and who reads them, and left a sharper question open:…
Kafka's Core Model: Topic, Partition, Offset and Consumer Group
· tech · 14 min read · 📚 Kafka — Learning Notes #2
Part one pinned Kafka's mental model to a single line: a replayable event log. But how is that log a…
What Is Apache Kafka? From Message Queue to Event Streaming
· tech · 12 min read · 📚 Kafka — Learning Notes #1
In one sentence: Apache Kafka is a distributed event streaming platform — it collects the events flo…
Getting Spark Running: From Your Laptop to a Cluster and a Managed Platform
· tech · 16 min read · 📚 Spark — Learning Notes #4
The first three posts were about how to write Spark and how to make it fast (what it is, DataFrames …
#spark#data-engineering#deployment
The Medallion Architecture: Managing Data Quality in Bronze / Silver / Gold Layers
· tech · 14 min read
In one sentence: the Medallion architecture is a design convention that splits data into three layer…
What Spark Performance Actually Is: Shuffle and Tuning
· tech · 11 min read · 📚 Spark — Learning Notes #3
Part one and part two both ended on the same line: Spark performance is shuffle. This post takes it …
Spark DataFrames in Practice: Read, Transform, Write and Spark SQL
· tech · 9 min read · 📚 Spark — Learning Notes #2
Part one covered Spark's concepts: driver and executors, partitions, lazy evaluation, shuffle. This …
What Is Apache Spark? Distributed Data Processing in One Post
· tech · 13 min read · 📚 Spark — Learning Notes #1
In one sentence: Apache Spark is an engine that takes data too big for one machine to hold or finish…
Leadership — Seeing Yourself Through a Journal
· tech · 5 min read · 📚 Becoming a Tech Leader — Reading Notes #6
Last time we covered self-blindness, the first obstacle to innovation — you can't change a thing you…
Leadership — The Three Great Obstacles to Innovation
· tech · 8 min read · 📚 Becoming a Tech Leader — Reading Notes #5
Innovation isn't the flash of a few geniuses; it's the ability of an environment to let you put a di…
Leadership — The Myths and the Pain of Becoming a Leader
· tech · 8 min read · 📚 Becoming a Tech Leader — Reading Notes #4
Before you actually become a leader, certain beliefs will keep you outside the door. Plenty of peopl…
Leadership — How Leaders Develop
· tech · 8 min read · 📚 Becoming a Tech Leader — Reading Notes #3
This model describes how skill changes over time. The same growth curve takes a completely different…
Leadership — MOI
· tech · 12 min read · 📚 Becoming a Tech Leader — Reading Notes #2
A model of leading is a handful of dimensions used to describe an environment. The MOI model is one …
Leadership
· tech · 7 min read · 📚 Becoming a Tech Leader — Reading Notes #1
The word "leadership" gets read as "senior title", "gives the orders", "has people who do what he sa…
When Residents Suspect the Lottery Code Is Rigged
· tech · 28 min read
A while back I built a small parking-space lottery page for the apartment complex I live in. The oth…