Tech

103 posts translated into English so far. For the complete list, including posts still only in Chinese, see Start here.

The cost of interrupts: not time, but fragmentation work ramp-up interrupt ① a fragmented day complete deep work ≈ none (all fragments) ② interrupts batched into one block uninterrupted deep work (one whole block) interrupts same total interruption → deep work = one whole block An interrupt's cost isn't the time it takes; it's fragmenting what's left until deep work is impossible

Operational Interrupts: What Kills Productivity Isn't the Workload, It's Fragmented Time

· tech · 10 min read · 📚 Google SRE — Reading Notes #17

The on-call post covered "how to design alerts and who carries the pager"; the toil post covered "th…

#sre#reliability

PRR: a gate before launch Dev builds the service features OK ≠ launch-ready PRR · Production Readiness Review ① SLO defined? ② monitoring + alerts (golden signals) ③ load test / capacity / load shedding ④ can a release roll back fast? ⑤ a dependency dies → degrade? ⑥ runbook: usable at 3am pass fail SRE takes over shares the pager sent back to fix pager stays with Dev SRE won't take an "unoperable" service — PRR makes reliability a hard launch gate

Production Readiness Review (PRR): What Makes a Service Worth SRE Taking Over

· tech · 13 min read · 📚 Google SRE — Reading Notes #16

The previous fifteen posts almost all talked about "the service is already live; how do we make it m…

#sre#reliability

Wouldn't change (the core) the FSM · lookup-as-validation append facts + derive on read batches that silt but don't fall 3NF · two stock columns per-provider payment fact tables five boring components · one VM order the queue · merge = release transcoding as validation · generic FKs Would add (protection) four golden signals + a batch lag gauge a language for incidents: severity · runbooks a dead-letter queue for poison pills Cloudflare on day one scheduled invariant queries (self-reconciliation) deleted_at · a reverse lookup before sweeping load testing · the other half of the checklist — all protection, not one of them a feature The core was right; everything owed was protection Which also explains why I later moved towards SRE

Re: If I Really Started Over

· tech · 21 min read · 📚 Re:Building a Live-Commerce Platform from Zero #21

This series started with the journey of one comment and ran for twenty chapters. The finale adds no …

#war-story#live-commerce#retrospective

Station 1: one instance + tenant_id every table carries tenant_id · leading composite index the terminus for most SaaS trigger to move on: connections, data volume, noisy neighbours — not "it feels time to distribute" Station 2: read/write separation replicas absorb reporting and storefront reads writes still single-point; this station buys time trigger: reads overwhelm writes, reports disturb trading Station 3: pool + silo hybrid small merchants share a pooled shard · big hosts get their own DB the DB-layer answer to noisy neighbours = enterprise tiering per-tenant backup/restore = the silo's killer advantage upgrading tiers means a move: one merchant's downtime, affecting only them, scheduled late at night Station 4: real sharding Citus-style shard by tenant_id / NewSQL needed only when the pool itself must scale out honestly: most live-commerce SaaS never reach this station Every station has a trigger — untriggered, you stay put; evolution is a response, not an interest

A Parallel World: What If It Had Become a SaaS

· tech · 32 min read · 📚 Re:Building a Live-Commerce Platform from Zero #20

First, something nineteen chapters never mentioned: every one of us took a pay cut to join. What the…

#war-story#live-commerce#system-design

the peak seven people · shipping daily that day CTO: development is paused contract renegotiated · possible layoffs a month later three people left me · one front end · the CTO new direction purchasing from external orders the split bidding out · monolith demoted Background: the storefront is still up, but there are no customers

Microservices for Three People: After Development Was Paused

· tech · 25 min read · 📚 Re:Building a Live-Commerce Platform from Zero #19

The last chapter ended by saying this team's end was nearer than anyone thought. This chapter starts…

#war-story#live-commerce#microservices

the commerce line me (backend lead) backend ×1 front end ×2 CTO: full-time PM wrote no code during commerce engineering output = 4 engineers the bidding line (separate) backend ×1 · front end ×1 dedicated PM ×1 honestly: I never knew this line (early) 1–2 contractors — a tourniquet for the single-backend period fully remote · the split line = the org line (foreshadowing the next chapter)

Six Engineers Running at the Speed of Twenty

· tech · 22 min read · 📚 Re:Building a Live-Commerce Platform from Zero #18

The system's story closed with the last chapter; the remaining chapters are about the people who bui…

#war-story#live-commerce#engineering-management

the live floor assistant screenshots the stream pastes into a dialog, confirms front end: optimisation crop square · convert to webp for experience and bandwidth back end: guarantee always re-encode + thumbnail original bytes never land GCS image + thumbnail two files transcoding is validation: a file that won't decode fails by itself image_metadata: path · content type · object id The DB stores the path (fact); the API resolves a URL (derivation) — images are served from GCS behind a CDN What the front end does is optimisation; what the back end does is guarantee

Easy to Upload, Hard to Delete: The Life Cycle of Images and Resources

· tech · 26 min read · 📚 Re:Building a Live-Commerce Platform from Zero #17

A detour into the least glamorous topic here: product images. "Isn't that just file upload?" — this …

#war-story#live-commerce#system-design

Inside a database Our system WAL: write the log before anything else comments cleaned, then appended and landed log consumer: digest the log, build indexes FSM batch digests comments, builds carts materialised view: computed and stored sold-quantity count (the only materialisation) view: computed at read time payment / order status derived on read redo log: a replayable history of change allocation log: rebuildable end to end built-in scheduler: vacuum, checkpoint heartbeat scans: chasing payment, settlement repair: fix it back from the facts hourly recomputation of sold quantities The price of unbundling: restoring, one by one, the guarantees a database gives you free And this chapter answers how we restored the one called "consistency"

Reconciliation: We Never Built It, So Why Did the Books Balance?

· tech · 21 min read · 📚 Re:Building a Live-Commerce Platform from Zero #16

In the plan this chapter was called "the three ledgers: stock, orders, payments", and it was going t…

#war-story#live-commerce#data-consistency

13,000 viewers online one VM traefik API+WS API+WS API+WS API+WS Celery: heartbeat the scheduling heart Celery: fetch comments exactly 1 worker Celery: async tasks 10 workers · acks_late Redis RabbitMQ Deliberately doing it on as little as possible — a philosophy, not a budget Cloud SQL 8 cores · no self-run DB

The Years Without an SRE: A Backend Lead's Production Diary

· tech · 26 min read · 📚 Re:Building a Live-Commerce Platform from Zero #15

The last chapter was how the peak hits; this one is about the person fighting it. There was no such …

#war-story#live-commerce#sre

200/s 10–20/s showing the product bidding opens: 10–20× in one second promos dropped during the close → a run of waves one close: 10 seconds at the short end, 3 minutes on average 13,000 viewers online; with a second channel opening at once, ~250 comments/s at worst

The Moment of Opening: The Three Seconds After the Host Calls a Key

· tech · 25 min read · 📚 Re:Building a Live-Commerce Platform from Zero #14

The transaction and operations stories are done; the next few chapters are about things that cut acr…

#war-story#live-commerce#scalability

1st offence a warning dialog on arrival press "confirm" to unlock signing for it · false positives ≈ 0 2nd offence locked for one month expires automatically 3rd offence locked for three months expires automatically 4th offence permanent ban support can lift it Reoffend and you climb a tier — grading makes false positives cheap and lets repeat offenders climb into the heavy penalties

Risk Control and Blocklists: Not an Eviction Notice, a Credit System

· tech · 21 min read · 📚 Re:Building a Live-Commerce Platform from Zero #13

Let's state the threat model first. In this system, calling an order in the comments holds stock imm…

#war-story#live-commerce#risk-control

message severity → private reply automatic · free · high volume win notice + binding token policy wall: one reply per comment, that's all email routine · internal notices async completions go here Google Workspace, off the shelf downside: it sinks phone (support) manual · priciest · always lands final notice: before the money is cleared and they're blocklisted channel bought with free shipping SMS: OTP + one final chase per round whoever the SMS doesn't move gets a call Channel cost tracks message severity; and reach isn't granted by an API — the product bought it

Notifications: Two Columns, One Scheduled Job, and a Phone Call

· tech · 22 min read · 📚 Re:Building a Live-Commerce Platform from Zero #12

What "a notification system" looks like in a textbook: a notification service, a message queue, a te…

#war-story#live-commerce#notification

With a coupon: one coupon = a combination of three parameters Effect amount off over a threshold free shipping over a threshold Threshold how much spend triggers it + product allowlist (what counts) Scope one round / several rounds all rounds combined e.g. "300 off over 3000 · round A only · selected products" No coupon: multi-buy offers buy N, get M free one product can carry several the ordering story is the next section Experiment layer: Django admin every flavour of buy-A-get-B applied urgently, experimentally promoted to a real rule once it holds

Promotions and Amounts: A Miscalculated Discount Is Harder to Find Than an Oversell

· tech · 38 min read · 📚 Re:Building a Live-Commerce Platform from Zero #11

An oversell explodes and a payment failure shouts; a miscalculated discount makes no sound at all. A…

#war-story#live-commerce#pricing

Act I Django group + permission model-level CRUD a machine's granularity Act II (+1 week) permissions stuffed into the JWT wrong granularity, in concrete a change = a re-login Act III move to role-based semantics finally right but an orthogonal axis appears and roles start exploding Landing stackable capability roles cost monitor multiply → add Every step was locally reasonable: use the built-in (saves time) → stuff it in the JWT (matches statelessness) → switch to roles (needs semantics) The error is in none of the steps — it's never having asked what this system's natural unit of permission is

Permissions: Who Can Press Which Button — We Rebuilt It Three Times

· tech · 30 min read · 📚 Re:Building a Live-Commerce Platform from Zero #10

Permissions are something I'll certify as one of the parts we didn't get right: rebuilt three times,…

#war-story#live-commerce#authorization

Host view only, no actions comment stream (with blocklist tags) units sold who ordered viewer count (FB API) reads the rhythm Assistant the live console enter product info (ahead or on the fly) open · close bidding re-open add stock acts for the host Operations its own pages close a round shipping fees goods-in allocation owns round rhythm Support handles exceptions adjust carts clear carts multi-binding catches the residue Engineers Django admin safe console Celery tasks odd requests (rough-draft lab) uncertainty lives here One set of facts and APIs underneath — the same data through five different windows

The Host and Operations Consoles: There Is No Such Thing as "the Back Office"

· tech · 18 min read · 📚 Re:Building a Live-Commerce Platform from Zero #9

The first eight chapters were the skeleton of a transaction: comments in, stock held, money collecte…

#war-story#live-commerce#internal-tools

sales stock (the promise) cap · sold count — guards overselling at order time lives in millisecond transactions physical stock (reality) goods-in appended — ops logs what arrived lives on logistics' daily rhythm The two books drift by nature: 100 agreed, 80 delivered allocation system real stock → assigned to orders; ops decides how allocation log appended — movements rebuildable from events order fully allocated → a shipping note (items go together, never split) 7-Eleven pickup API · customer picks a store online home delivery (own courier) CSV exported by hand to the courier

Pre-Shipment: You Sell a Promise, You Ship a Reality

· tech · 24 min read · 📚 Re:Building a Live-Commerce Platform from Zero #8

The money is in; time to ship. This chapter is where the system meets the physical world — comments,…

#war-story#live-commerce#fulfillment

orders payment status not stored, aggregated on read: paid = SUM(facts) ≥ due Bank A · smart transfer fact table Bank A · card fact table Bank B · payment fact table cash receipt ops-marked 150,000 cap 30k each = 5 facts webhook (live) the bank pushes it polling (backstop) scheduled queries Both channels write the same fact tables — facts are idempotent, so overlap is harmless and each backs the other

Third-Party Payments: Three Textbook Pitfalls, and Terrain That Has None

· tech · 21 min read · 📚 Re:Building a Live-Commerce Platform from Zero #7

The last chapter aggregated orders into three layers; now we have to collect the money. Payments are…

#war-story#live-commerce#payment

reserving cart (live) can span several rounds / hosts non-reserving cart (storefront) same cart item table, polymorphic source merged checkout orders payment —— the unit of paying pay once across rounds · aggregates payment facts order (round A) fulfilment unit · round coupons here order (round B) split by round order (round C) its own shipping rhythm order item —— the accounting unit: amount frozen at sale

From Cart to Order: The State Machine the Hosts Ripped Out

· tech · 27 min read · 📚 Re:Building a Live-Commerce Platform from Zero #6

The last stop on the transaction spine: how an order travels from cart to order. Comments made it in…

#war-story#live-commerce#system-design

product price, product info changes all through a stream 1:1 inventory table (hot data, its own table) stock cap —— a restock adjusts only this cart qty (reserved) order qty (paid) payment: cart → order, moved inside one transaction invariant: cart + order ≤ cap (sellable is derived, never stored) FSM batch orders · LWW edits customer adjusts quantity support clears · adjusts carts assistant / ops restock · close round Four writers — but both counts are recomputed hourly from cart/order items (a derived value healing itself)

Stock: Never Overselling Is This System's One Iron Rule

· tech · 27 min read · 📚 Re:Building a Live-Commerce Platform from Zero #5

With comments parsed and orders hung off an identity, the main line reaches the heart: stock. This s…

#war-story#live-commerce#inventory

identity: a fact fb user (PSID) created on the spot, at comment time ig user own-studio user account: an aggregate account appears only after login claims identities' orders, 1:N binding (can be multiple) cart item (reserves stock) attached to the fb user, not the account fbmsgtocartitem msg id · bidding key id aggregate view: claim it, don't move it

Identity and Accounts: Who Exactly Is the Person Commenting?

· tech · 25 min read · 📚 Re:Building a Live-Commerce Platform from Zero #4

The FSM in the last chapter worked out "who bought what" — except that "who" is only a string of dig…

#war-story#live-commerce#identity

FB (100) · poll IG (10) · webhook Own (1) · push Fetch: per-source adaptive: busy→fast, idle→slow all into one table Clean → land append in a unified format dedupe: source+message id batch 200 · FSM parse → look up bidding key decrement sold + identity raw is stored · FB re-fetchable — never used failure → skipped, no recovery path All for the freshest stock the host can see; the price: minutes of lag at peak, orders lost in silence

Comment as Order: Turning a Chat Room into an Order Channel

· tech · 42 min read · 📚 Re:Building a Live-Commerce Platform from Zero #3

With the big picture and the opening move in place, here's the first battle on the main line: how on…

#war-story#live-commerce#fsm

Synchronous · ms Django API request in, response out cart · checkout · accounts Real-time · push WebSocket viewer comments → live to the host dashboard Async · sec to min RabbitMQ (the pipe) Celery workers fetch comments · FSM orders · invoices email · order exports PostgreSQL the single truth: orders · stock · members Redis speed: fast banned-user checks One specialist per timescale, with one truth and one speed underneath

The Opening Move: Five Components and One CI/CD Pipeline

· tech · 15 min read · 📚 Re:Building a Live-Commerce Platform from Zero #2

With the big picture laid out, and before we get into the battles over comments and stock, let's ope…

#war-story#system-design#django

Comments arrive FB / IG / own studio Unified event one adapter per source Parse key+n same person → last wins Identity order without an account Reserving cart stock −n · no overselling Checkout two carts merge Payment gateway webhook · idempotent Pre-shipment consolidate · to courier unpaid past deadline → release stock Every stop is a chapter: comment intake, identity, stock, cart, payments, pre-shipment

The Big Picture: Every System One Comment-Placed Order Touches

· tech · 22 min read · 📚 Re:Building a Live-Commerce Platform from Zero #1

This is a new series, and the first war story on this blog. At a previous company I built a live-str…

#war-story#live-commerce#system-design

Traditional: one box, everything bundled storage engine indexes cache mat. view log ↓ unbundle: each feature handed to a specialised system ↓ log as hub (Kafka) — the glue that sets order OLTP DBstorage + transactions Elasticsearch= the index Redis= the cache warehouse / Gold layer= materialized view every system is a follower of the log, consuming in the same order → each consistent Your data platform = a database "turned inside out"; what holds it together is the log

The Future of Data Systems: The Unbundled Database, Kappa, and End-to-End Correctness (Finale)

· tech · 13 min read · 📚 Designing Data-Intensive Applications — Reading Notes #12

The final chapter. The first eleven took storage, replication, partitioning, transactions, consensus…

#distributed-systems#book-notes#data-engineering

✗ Dual write: the app writes three copies application DB cache search index disease 1: crash halfway → some written, some not no transaction can roll back across three systems disease 2: concurrent writes arrive in different orders DB gets A then B, cache gets B then A → different values → three systems diverge permanently, silently ✓ Log first: write to one place only application only writes here one ordered log (source of truth) DB cache search index all consume in the "same order" → consistent, replayable every downstream is a follower, converging to one state One dataset into N systems? Pick one source of truth, make all the rest followers

Stream Processing: The Dual-Write Trap, CDC, and Stream–Table Duality

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #11

Batch processes data that has "all arrived"; streaming processes data that "keeps coming". Much of t…

#distributed-systems#book-notes#streaming

input (immutable) log shard 1 log shard 2 log shard 3 ① map (in place, parallel) (/a,1)(/b,1)… (/a,1)(/c,1)… (/b,1)(/a,1)… extract (key, value) per record, no data moved ② shuffle (redistribute by key) same key → same node, sorted the only big move = most expensive ③ reduce (aggregate groups) /a:(1,1,1)→ /a=3 /b:(1,1)→ /b=2 … output: written as "new" files map runs in place (bring compute to the data) · shuffle is the only big move, and where all the cost lives groupBy, join, dedupe… every "same key must meet" operation is a shuffle underneath

Batch Processing: The Spirit of MapReduce, Two Roads to a Join, and the Virtue of Immutable Inputs

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #10

Part II held consistency together inside a single system; Part III's theme switches to data flowing …

#distributed-systems#book-notes#data-engineering

Final whistle: result written, but two replicas at different progress leader: 2 : 1 final replica 1 (caught up): 2:1 final replica 2 (lagging): 1:1 playing ① Alice sees "final 2:1" → tells Bob ② Bob refreshes → "still playing"?! A "later" read sees an "older" state → the single-copy illusion shatters = not linearizable

Consistency and Consensus: Linearizability, the Honest Version of CAP, and Total Order Broadcast

· tech · 16 min read · 📚 Designing Data-Intensive Applications — Reading Notes #9

The previous post concluded that no single node's judgment can be trusted, so truth can only be deci…

#distributed-systems#book-notes#consistency

your node sent request, waiting… ① request lost on the networkthe other side never received it ② the other node crashedactually dead ③ the other side is just slow (overloaded, in a GC pause)it'll handle it in a moment — may be handling it right now ④ it finished, but the "response" was lost on the way backthe action already happened, and you think it didn't from your end: all four identical = no response A timeout firing means "you decided to stop waiting", not "you know what happened"

The Trouble with Distributed Systems: Unreliable Networks, Untrustworthy Clocks, and Half-Dead Nodes

· tech · 13 min read · 📚 Designing Data-Intensive Applications — Reading Notes #8

The previous post ended on a hook: once a system spans several machines, even "locks" and "validatio…

#distributed-systems#book-notes#reliability

Invariant: doctors on call ≥ 2 (now: Alice and Bob both on) Alice's txn ① count on call → reads 2 ≥ 2 ✓ ② set "self" to on leave Bob's txn ① count on call → also reads 2 ✓ ② set "self" to on leave (snapshot isolation: both see the old snapshot's 2) (different rows → no write conflict, both commit) Result: doctors on call = 0, invariant broken 💥 Each txn is right "on its own"; the error: the condition you checked was silently changed by the other's write Same script: double-booked meeting room, two accounts claiming a username, a balance debited twice at once

Transactions: The Write Skew Snapshot Isolation Can't Stop, and Three Roads to Serializability

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #7

I laid the foundations of transactions in the SQL series: ACID's emphasis is on the I, the three ano…

#distributed-systems#book-notes#transactions

By key range A–F G–R S–Z like encyclopedia volumes, keys ordered ✓ range scans efficient (one contiguous read) ✗ key is a timestamp → today's writes all hit the last partition (hot spot) HBase / early Bigtable By hash of key partition 0 partition 1 partition 2 hash sprays adjacent keys evenly ✓ load even, hot spots scattered ✗ order lost → a range scan must ask every partition Cassandra / Redis Cluster (CRC16) / Kafka Compromise: compound key (hash first column → partition, sort the rest within it) — Cassandra's signature Hash can't save a single extremely hot key (the celebrity problem) — salt it in the app, split one key into many

Partitioning: Range or Hash, Where Secondary Indexes Live, and How to Rebalance

· tech · 13 min read · 📚 Designing Data-Intensive Applications — Reading Notes #6

Replication keeps the same data on several machines; partitioning (also called sharding) splits the …

#distributed-systems#book-notes#partitioning

Single-leader Leader follower follower all writes conflicts: killed at source (one writer) price: failover is hard (who takes over?) MySQL / Postgres / Redis / Kafka Multi-leader Leader A Leader B datacenter 1datacenter 2 both edit the same row → conflict! gain: write locally, survive a netsplit price: resolve write conflicts later multi-DC / offline editing / collaborative docs Leaderless replica replica replica client writes n copies at once quorum: w + r > n gain: no leader, no failover price: complex paths, read repair Dynamo / Cassandra Three answers to one question: block conflicts at the source, resolve later, or reconcile on read?

Replication: Single-Leader, Multi-Leader, Leaderless, and Three Replication-Lag Anomalies

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #5

Into Part II, and data starts crossing machines. The first move is replication: the same data on sev…

#distributed-systems#book-notes#replication

Mid rolling upgrade: old and new both running, data flows both ways new code v2the nodes already upgraded old code v1the nodes not yet reached same database / topic both sides write, both sides read backward compatibility new code can read old data everyone remembers this (migration mindset) forward compatibility old code can read new data most forgotten — but daily in rollout and rollback Every schema change needs both — miss one and errors start mid-rollout

Encoding and Evolution: Letting Old and New Code Read Each Other's Data

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #4

The previous post was about how data gets onto disk. This one is about a problem that's easier to un…

#distributed-systems#book-notes#data-engineering

LSM-tree: append-only, tidy later memtablein memory, catches writes full → flush batch sequentially SSTable (new, immutable) SSTable (older) SSTable (oldest) compactionbackground merge + dedupe write: always sequential append → fast read: may search several SSTables (bloom filter helps) B-tree: page tree, update in place root page page page ✎ overwrite in place leaf page WAL: sequential log entry before touching a page read: walk 3–4 levels down → steady write: random I/O in place + WAL first LSM: write-optimised (RocksDB / Cassandra / HBase) · B-tree: steady reads (nearly every relational DB)

Storage Engines: LSM-Trees, B-Trees, and Column-Oriented Storage

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #3

The previous post chose the data model. This one drills to the very bottom: how does a database actu…

#distributed-systems#book-notes#storage

Pub/Sub: broadcast, gone is gone PUBLISH news "…" channel: news (not stored) Sub online✓ got it Sub online✓ got it Sub offline✗ missed nobody listening → gone · no persistence · no replay · no ack Stream: a kept log, replayable XADD stream * … (append) m1m2m3m4m5 old consumeranother reads from here messages kept (MAXLEN cap optional) re-read anywhere · catch up after being offline · groups + ack

Pub/Sub vs Stream: Redis's Version of a Messaging System

· tech · 12 min read · 📚 Redis — Learning Notes #12

Redis can serve as a messaging system too, but it has two completely different things for it, and us…

#redis#distributed-systems

One by one N × RTT vs pipeline 1 × RTT ① one by one: each waits a round trip clientserver 3 cmds = 3 round trips = 3 × RTT ② pipeline: packed into one round trip clientserver cmd1 ; cmd2 ; cmd3 sent together three replies back together 3 cmds = 1 round trip = 1 × RTT ✓

Pipelining, Transactions and Lua: Saving RTTs vs Atomicity

· tech · 12 min read · 📚 Redis — Learning Notes #11

Three things are often lumped together that actually solve completely different problems: pipelining…

#redis#distributed-systems

Where a key lands: CRC16 → 16384 slots → node key: "user:1000" CRC16(key) % 16384= slot 5798 Node Aslot 0 – 5460(+ one replica) Node B ✓slot 5461 – 109225798 is here → B holds it Node Cslot 10923 – 16383(+ one replica) the 16384 fixed slots are a "middle layer"; a node just claims a range of slots so moving data = moving slots; scaling stays clean, no recomputing every key's place

Redis Cluster: How 16384 Slots Shard and Rescale

· tech · 13 min read · 📚 Redis — Learning Notes #10

The single-thread post said one Redis's bottleneck is memory and the network. When one machine can't…

#redis#distributed-systems

The five steps of Sentinel automatic failover ① monitorkeeps pinging masterand replicas ② subjectively downone Sentinel seesno response (SDOWN) ③ objectively downmajority agree "really dead"(ODOWN) ④ elect a Sentinel leader → promote a replicapick the most complete replica as the new masterpoint the other replicas at the new master ⑤ tell clients the new master's address clients askSentinel"who is themaster now?"

High Availability: How Sentinel Fails Over Automatically

· tech · 12 min read · 📚 Redis — Learning Notes #9

The previous post's replication gave you copies, but left a big hole: when the master dies, nobody t…

#redis#high-availability

One master writes, many replicas read applicationwrite → masterread → replicas Masterread/write · sole write pointonly one write Replica 1read-only Replica 2read-only async replication readreads spread over the replicas → scale reads horizontally

Master-Replica Replication: Read/Write Splitting and the Oddities of Replication Lag

· tech · 11 min read · 📚 Redis — Learning Notes #8

However fast one Redis is, it has ceilings on memory and traffic, and when it dies the data hangs in…

#redis#distributed-systems

Acquiring the lock: one atomic command SET lock:res <token> NX PX 30000 NXset only if the key doesn't exist→ mutual exclusion PX 30000built-in 30 s TTL→ holder dies, auto-release; no deadlock <token>a random value→ marks "this lock is mine" Releasing the lock: verify the owner (Lua, atomic) if GET(key)==token then DEL(key)GET and DEL must be atomic → only delete your own lock DEL without checking token→ deletes someone's renewed lock ✗

Distributed Locks: From SETNX to Redlock, and That Famous Argument

· tech · 14 min read · 📚 Redis — Learning Notes #7

When several processes or machines compete for the same resource (only one may decrement stock or ru…

#redis#distributed-systems

① scheduled to a nodethe Scheduler picks one ② image pulledkubelet pulls from the registry ③ container up · alivestarts and doesn't crash ④ Ready · in Endpointstraffic only after readiness passes ✓ serving traffic Pendingnot enough resources / taint without toleration / PVC won't bind ImagePullBackOff · ErrImagePullimage name typo / private registry missing imagePullSecret CrashLoopBackOff · OOMKilledcrashes on start, restarts forever / killed for exceeding memory limit Running but not Ready · unreachablereadiness failing / Endpoints empty (bad selector)/ DNS won't resolve / NetworkPolicy blocking

Troubleshooting: How to Investigate Pods, Nodes and the Control Plane

· tech · 16 min read · 📚 Kubernetes — Learning Notes #13

The biggest slice of the CKA is troubleshooting (30%), but it's really not new knowledge — it's the …

#kubernetes#troubleshooting

Control Plane node · kubeadm init api-server scheduler controller-mgr etcd all static pods: kubelet reads /etc/kubernetes/manifests brings them up and keeps them running prints a join token + command kubeadm join <token> Worker noderuns only the kubelet + your Pods Worker nodeadd as many as you need add more CP nodes (join) → HA control plane

Cluster Administration: kubeadm, etcd Backups, Upgrades

· tech · 13 min read · 📚 Kubernetes — Learning Notes #12

The previous eleven posts all stood in the position of "using the cluster". This one moves to "build…

#kubernetes#operations

kubectl / Pod sends a request ① authn who are you? certs · token · OIDC ② authz may you do this? ← this is RBAC execute API Server identity unverified → 401 no permission → 403 two gates, one job each: first confirm "who you are", then decide "whether you may"

RBAC: Who Can Do What to the Cluster

· tech · 15 min read · 📚 Kubernetes — Learning Notes #11

The previous ten posts were about making things "run and be reachable". This one switches dimension:…

#kubernetes#security

user one domain one public IP GET shop.com/api Ingress L7: reads the HTTP Host + path shop.com/ shop.com/api img.shop.com terminates TLS here too (https → http) web-svc ClusterIP → web Pods api-svc ClusterIP → api Pods img-svc ClusterIP → img Pods one IP, one entrance, routed by URL to many internal Services — something an L4 Service can't do

Ingress and Cluster DNS: One Entrance In, One Name to Recognise Each Other

· tech · 13 min read · 📚 Kubernetes — Learning Notes #9

The fourth post gave short-lived Pods a fixed address, the Service, but left two loose ends: first, …

#kubernetes#networking

Pod nodeSelector / affinity "I want a disk=ssd node" toleration: immune to gpu taint Node label: disk=ssd taint: gpu=true:NoSchedule non-immune Pods are driven off pull: Pod picks a node by label push: node drives off non-immune Pods a toleration is only immunity; it doesn't "attract" the Pod

Advanced Scheduling: Getting Pods onto the Right Node

· tech · 20 min read · 📚 Kubernetes — Learning Notes #7

The second post covered how the Scheduler picks a node for a Pending Pod in two steps: filter first …

#kubernetes#scheduling

PV / PVC: separating "demand" from "supply" StorageClass: dynamic provisioning — a PVC arrives, a PV is created Podwants a disk PVC (demand)"I want 10Gi · RWO"all the app developer asks for bound PV (supply)the actual 10Gi of storagebackend: EBS / NFS / Ceph… Supply / demand separation: the app says "how big, which access mode", never what hardware access modes: RWO (one node) · ROX (many nodes, read-only) · RWX (many nodes read-write; needs NFS or similar) reclaim policy: after the PVC is deleted, Retain the PV or Delete it (backend included)

K8s Storage: Volumes, PV/PVC and StatefulSets

· tech · 13 min read · 📚 Kubernetes — Learning Notes #6

Pods are short-lived and disposable — self-healing swaps them out at any moment. But some things mus…

#kubernetes#storage

Three breaches, three different shapes Penetration query a "nonexistent" key not in cache, not in DB → cache can never block it every one hits the DB Breakdown one hot key ⏰ just expired a burst of concurrent misses → all rush to rebuild from DB concentrated on one point Avalanche many keys ⏰ expire together (or Redis goes down entirely) → misses across the board DB collapses → cascade in common: requests bypass the cache and flatten the DB; the difference is the breach — a nonexistent key (penetration) · one hot spot (breakdown) · a whole swath (avalanche)

The Three Cache Disasters: Penetration, Breakdown, Avalanche, and the Right Fixes

· tech · 11 min read · 📚 Redis — Learning Notes #6

Using Redis as a cache, the classic pattern is cache-aside: a read checks the cache first, returns o…

#redis#cache

One image, every environment imagemy-app:1.0(immutable) DevelopmentConfigMap + Secret StagingConfigMap + Secret ProductionConfigMap + Secret Pod (dev) Pod (staging) Pod (production) 12-factor: config in the environment, never baked in — only then can one image be reused

ConfigMap and Secret: Pulling Configuration and Secrets Out of the Image

· tech · 12 min read · 📚 Kubernetes — Learning Notes #5

Over the previous posts you've learned to deploy an app and expose it as a service. But a real app i…

#kubernetes#concept

How expired keys get cleared: lazy + periodic, two prongs key whose TTL has expiredstill occupies memory (not cleared yet) ① lazy deletion (passive)someone GETs it →found expired → deleted on the spot, returns nil ② periodic deletion (active)background samples ~10×/s →random batch from keys with TTL, delete expired so "expired" ≠ "freed" — untouched and not yet sampled, it just lies there

Redis Expiration and Eviction: TTL, Lazy Deletion and maxmemory Policies

· tech · 12 min read · 📚 Redis — Learning Notes #5

Use Redis as a cache and sooner or later you hit two questions: how does a key with a TTL get cleare…

#redis#cache

Snapshot (RDB) vs running log (AOF) RDB snapshotperiodic full picture loss window ⚡ small file, fast load, good for backups ✓ | lose what happened between snapshots (maybe minutes) ✗ AOF logevery write logged ⚡ safer (lose at most one fsync interval) ✓ | big file, slow load (replay) ✗ Hybrid (Redis 4+): RDB snapshot at the head of the AOF + incremental commands → fast load, little loss (recommended)

Redis Persistence: RDB Snapshots vs AOF Logs, and Whether Data Is Actually Lost

· tech · 14 min read · 📚 Redis — Learning Notes #4

"Redis is an in-memory database; lose power and all the data's gone" — that line is half right, half…

#redis#persistence

Single-machine cron is easy; "reliable" cron is hard cron (one machine)time's up, run it — trivial ✗ machine dies → all schedules stop (SPOF) must runeven if it dies replica (leader) replica / standby replica / standby consensus log (Paxos)records: which jobs have run one dies → re-elect leader, restore state from consensus → no re-run, no skip Making the simplest cron reliable puts distributed consensus underneath it

Reliable cron: The Simplest Scheduled Job Gets Hard the Moment It's Distributed

· tech · 12 min read · 📚 Google SRE — Reading Notes #15

cron may be the simplest piece of infrastructure there is: time's up, run a job. Anyone who has writ…

#sre#reliability

Single thread + event loop: one queue, one at a time client client client client 10k+ conns epollI/O multiplexingone thread watches all command queue (one line)cmd1cmd2cmd3 single-thread executionone at a time, each atomicno locks, no races Redis 6's "multithreading" is only for socket reads/writes, the network chores — command execution stays single-threaded, so atomicity and lock-freedom are unchanged

Why Is Single-Threaded Redis So Fast? And the Landmine of O(N) Commands

· tech · 11 min read · 📚 Redis — Learning Notes #3

The first post said one reason Redis is fast is "single thread + no locks". That sounds backwards — …

#redis#performance

Five core structures: pick the right one, the problem is half solved Stringbytes / number 42 cache · counter (INCR) · distributed lock (SET NX) Listordered, both ends queue (LPUSH / RPOP) · latest N items (LPUSH+LTRIM) Hashfield → value name: Aidanage: 30 store objects · change one field without moving the whole thing Setunordered, deduplicated dedup · tags · intersection (mutual friends, SINTER) Sorted Setscored members, kept sorted a:1b:2c:3 leaderboard · range queries · delayed queue (score = time)

The Soul of Redis: Five Core Data Structures + Advanced Weapons

· tech · 12 min read · 📚 Redis — Learning Notes #2

The previous post said the soul of Redis is "data structures" — so this one opens the toolbox. Ninet…

#redis#data-structures

Traditional KV cache (memcached) key → "a blob of string" (opaque) changing one field = GET whole → edit in app → SET whole Redis (data structure server) key → List / Hash / Set / ZSet operate server-side (atomic) = HSET · LPUSH · ZADD · ZRANGE e.g. leaderboard = one Sorted Set, ZADD to score + ZRANGE for top N no fetching the whole board to sort in the app — compute next to the data, fast and atomic

What Is Redis? Not Just a Cache, but an In-Memory Data Structure Server

· tech · 14 min read · 📚 Redis — Learning Notes #1

Most people first meet Redis as a "cache" — throw database query results in, grab them next time. Th…

#redis#concept

Grab one real request and follow it all the way userone real request entry LB(in-house) API service(in-house framework) queue(in-house) databasefinal stop Stop at every hop and ask these four ① what is this? who maintains it? ② how do I see its health? monitoring / logs? ③ if it breaks, what happens downstream? ④ normally, what do traffic / data look like? One lap and you hold a "living" architecture map — more accurate than any wiki

An SRE Parachuted into a 'Build Everything In-House' Company: Standing Firm in the First 90 Days

· tech · 22 min read · 📚 Google SRE — Reading Notes #14.5

The most thrilling kind of job change is joining a company that builds almost everything itself: no …

#sre#career

Homebrew leader election: the network cuts, each side rules itself network cut ✂ Left half: Node A "no word from B → I'm leader" write arrives: X = 1 Right half: Node B "no word from A → I'm leader" write arrives: X = 2 Network heals: is X 1 or 2? both think they're right → data diverges, unrecoverable (split brain)

Distributed Consensus: How Machines That Crash Agree on One Thing

· tech · 19 min read · 📚 Google SRE — Reading Notes #14

A group of machines agreeing on "one thing" — who is the leader? who holds this lock? what's the lat…

#sre#distributed-systems

One request, two layers of load balancing useranywhere ① Frontend LBwhich datacenter?DNS · Anycast · VIP by geography / health / capacity datacenter(chosen) ② In-datacenter LBwhich machine?by real load / health task 1 task 2 task 3 handles "across datacenters":proximity, avoid dead sites, spread capacity handles "inside the datacenter":don't overload one, don't feed the broken

Load Balancing: Pick the Right Datacenter, Then the Right Machine

· tech · 11 min read · 📚 Google SRE — Reading Notes #13

Between a user sending a request and the request being handled, it passes through two layers of load…

#sre#networking

Automation's evolution: climb up, take the human out of the loop automation ↑ · fewer hands ④ Autonomous / self-healingthe system runs itself → humans leave the loop ③ General automation platformreused across systems, consistent, scalable ② Task-specific scriptssave time, but need care; new scenario, they break ① Manual operation (toil)slow, error-prone, inconsistent, can't scale ⚠ But automation amplifies the blast radius do it right consistently, do it wrong consistently — one bad click can switch off a whole datacenter

Automation, Release Engineering and Simplicity: Making Change Fast and Safe

· tech · 15 min read · 📚 Google SRE — Reading Notes #12

Automation, release engineering, simplicity — at first glance three unrelated topics. Put them side …

#sre#automation

what you think is safe backups every day ✓✓✓ safe "on paper" when you actually restore… it falls apart for real backup file itself corrupt (never verified) restore procedure never run → scramble restore too slow → SLA blown, data gone never-restored backup = Schrödinger's backup (alive or dead? open it to find out) what matters isn't the "backup", it's the "recovery" — the backup is only the means

Data Pipelines and Data Integrity: Having a Backup Doesn't Mean You Can Restore

· tech · 10 min read · 📚 Google SRE — Reading Notes #11

Beyond the "service stays up" reliability covered so far, a data system has a more fundamental layer…

#sre#data-engineering

one falls → shifted traffic crushes the next → domino wipeout Server Aoverloads first → down ✗ Server Btakes A's traffic → down ✗ Server Cgets everything → down ✗ and the retry storm pours fuel on the fire: requests fail / slow clients retry total load rises positive feedback: more failure → more retries → more failure (self-amplifying)

Cascading Failures and Overload: Don't Let One Server Take Down the Rest

· tech · 9 min read · 📚 Google SRE — Reading Notes #10

The first post said the goal of reliability is "keeps working when things go wrong". But one kind of…

#sre#reliability

E2E Integration Unit ↑ few, slow, brittle (full user path) components assembled (service + DB…) ↓ many, fast, stable (one function) Inverted (lots of E2E, few unit) = anti-pattern: slow, flaky, and hard to tell which layer broke

Testing for Reliability: Tests Don't Prove the Absence of Bugs, They Let You Move Fast

· tech · 10 min read · 📚 Google SRE — Reading Notes #9

This post is about something usually filed under "the developers' business" that is in fact a corner…

#sre#reliability

interface DevOps ← philosophy / culture: defines "what to do" reduce silos · failure is normal · incremental change · measure everything · automate implements (gives the concrete how) class SRE implements DevOps ← the how error budget · blameless postmortem · canary · SLI/SLO · eliminate toil DevOps is direction and principles, SRE is one concrete implementation — not either/or, different layers

DevOps vs SRE: One Is an Interface, the Other an Implementation

· tech · 8 min read · 📚 Google SRE — Reading Notes #1.5

With the previous post covering what SRE is, let me deal with a comparison that gets asked constantl…

#sre#culture

Blame culture · vicious cycle incident "whose fault?" hunt the culprit people hide mistakes, stay quiet nothing learned → repeat Blameless · virtuous cycle incident "why did the system allow it?" everyone tells the whole story fix the system → steadily stabler Same outage; asking "who" or asking "the system" takes the team down opposite cycles

Blameless Postmortems: Turning Outages into Organisational Learning

· tech · 11 min read · 📚 Google SRE — Reading Notes #7

The previous post was about finding the root cause — but then what? The postmortem turns one expensi…

#sre#culture

① Triage keep the system alive (root cause can wait) ② Examine monitoring / logs four golden signals ③ Diagnose hypothesise → test → eliminate bisect to close in ④ Treat one variable at a time reversibly not fixed? new hypothesis Every step "narrows the range"; the anti-patterns (guessing, swapping parts, changing many things) never narrow, only gamble

Effective Troubleshooting: Debugging Is a Method, Not a Talent

· tech · 10 min read · 📚 Google SRE — Reading Notes #6

The previous post said on-call stops the bleeding first; but once the bleeding has stopped, you stil…

#sre#incident

An alert comes in: does it need a person? How urgently? Page a person must intervene "now", or users are being affected → wake someone Ticket a person needs to handle it, but it isn't urgent → working hours Log nobody needs to look; keep it for reference / later analysis → disturb nobody Stuff the non-urgent into Page → alert fatigue, crying wolf; the real incident gets ignored

Alerting and On-Call: When to Wake Someone Up

· tech · 11 min read · 📚 Google SRE — Reading Notes #5

The previous post ended on a line: when should you wake someone up? This post answers it. It's reall…

#sre#incident

Relational users orders FK tables + foreign keys, SQL many-to-many, strong joins Document name, email positions [ … ] (nested) education [ … ] (nested) nested JSON, natural one-to-many good locality (one read, whole thing) Graph vertices + edges, highly connected the relationships are the point

Data Models: Relational, Document, Graph — What Are You Actually Choosing?

· tech · 10 min read · 📚 Designing Data-Intensive Applications — Reading Notes #2

The first post covered what a data system should pursue (reliable, scalable, maintainable). This one…

#distributed-systems#book-notes#data-modeling

Coordinator receives, plans, dispatches Segment 1 Segment 2 Segment 3 DISTRIBUTED BY (key): hash(key) decides which segment each row lands on; queries run on every node in parallel ⚠ uneven key → one segment overloaded (skew), parallelism lost, the slowest node drags down all

When SQL Runs on MPP: Greenplum and Cloudberry

· tech · 13 min read · 📚 SQL: I Thought I Knew It #12

The finale of the whole SQL series. The previous 11 posts all ran on single-node PostgreSQL; this on…

#sql#data-engineering

① Dirty Read T1 T2 balance → 200 (uncommitted) reads balance 200 ❌ ROLLBACK ② Non-repeatable Read T1 T2 reads balance 100 balance → 200, commits reads balance 200 again ❌ ③ Phantom Read T1 T2 finds 3 orders inserts 1 matching order, commits now finds 4 orders ❌ time →

Transactions and Isolation Levels: Graded Ways to Keep Concurrency from Fighting

· tech · 19 min read · 📚 SQL: I Thought I Knew It #11

Everything so far has been "how one query runs fast" (indexes, EXPLAIN); this post switches dimensio…

#sql#concept

Nested Loop outer outer outer inner(faster if indexed) each outer row → one inner lookup suits: small tables / indexed inner Hash Join small hash table (memory) big small builds hash, big probes suits: big tables, equality joins Merge Join 1 3 5 2 4 6 sort both sides, merge like a zipper suits: already sorted / indexed

Reading EXPLAIN: How the Optimizer Actually Runs Your Query

· tech · 10 min read · 📚 SQL: I Thought I Knew It #10

The previous post left a question: is the index actually being used? The answer is in EXPLAIN. This …

#sql#performance

Reliability keeps working when things go wrong · fault-tolerant, not fault-free · fault ≠ failure · deliberately inject faults hardware / software / human error Scalability holds up as load grows · describe the load first · measure with percentiles · scale up vs scale out know the load before adding machines Maintainability lets people work on it well · Operability · Simplicity · Evolvability complexity is paid for by others Three non-functional requirements — features decide if it works; these decide if it survives

Reliable, Scalable, Maintainable: The Three Goals of a Data System

· tech · 14 min read · 📚 Designing Data-Intensive Applications — Reading Notes #1

Starting a new series: reading Martin Kleppmann's Designing Data-Intensive Applications (DDIA). It c…

#distributed-systems#book-notes

GROUP BY date_trunc directly datecount 7/013 7/025 no row for 7/03 ✗ 7/042 fill the hole generate_series + COALESCE 0 7/013 7/025 7/030 7/042 GROUP BY only emits buckets that "have data" — 7/03 with no orders vanishes, and the series breaks

Time Bucketing and SCD: Two Traps in Handling Time with SQL

· tech · 10 min read · 📚 SQL: I Thought I Knew It #8

Following the previous post's "ranges", this one covers two traps time sets in SQL — both rooted in …

#sql#data-engineering

your service ① Latency how long until a request returns? track success and failure latency separately ② Traffic how busy is the system right now? QPS / requests per second ③ Errors how many requests failed? failure rate (incl. "200 but the content is wrong") ④ Saturation how close to the limit? resource utilisation → the best early warning First three = user experience (usable directly as SLIs); the fourth, saturation = how long you can hold

Monitoring: The Four Golden Signals

· tech · 12 min read · 📚 Google SRE — Reading Notes #4

The previous post said the SLI is the measured reliability number — and those numbers come from moni…

#sre#monitoring

Is it toil? Check six traits ✓Manual — a person does it, step by step ✓Repetitive — done many times, and will be done again ✓Automatable — a machine could do it; nobody has written it yet ✓No enduring value — the system is no better afterwards ✓Scales linearly — the service grows, so does it ✓Reactive — done when triggered, not planned More matches → more toil. Note: meetings, planning, documentation are overhead, not toil

Eliminating Toil: Treat Repetitive Operations as Bugs to Be Killed

· tech · 12 min read · 📚 Google SRE — Reading Notes #3

The first post said SRE's core is "operations can be engineered". The toil in this post is the thing…

#sre#reliability

① Identical rows (1, A, 100) (1, A, 100) (1, A, 100) DISTINCT (1, A, 100) whole row identical → drop the extras DISTINCT / GROUP BY solves it ② Same key, several versions user1 · pending · 2/01 user1 · paid · 2/03 user1 · refunded · 2/05 ROW_NUMBER keeps latest user1 · refunded · 2/05 rows differ → DISTINCT does nothing you have to pick the "representative" row

Deduplication Done Right: DISTINCT Isn't the Only Answer

· tech · 11 min read · 📚 SQL: I Thought I Knew It #6

Duplicate data is almost daily life in data engineering: a pipeline rerun imports twice, CDC pulls i…

#sql#data-engineering

Input (orders): 4 rows A · 100 A · 250 B · 80 B · 120 GROUP BY (collapses) A · SUM 350 B · SUM 200 4 rows → 2 rows (one per group) SUM() OVER (no collapsing) A · 100group 350 A · 250group 350 B · 80group 200 B · 120group 200 4 rows → 4 rows (each gains a whole-group value)

Window Functions: Aggregation Without Collapsing

· tech · 11 min read · 📚 SQL: I Thought I Knew It #5

The previous post's GROUP BY collapses each group into one row. But you've surely met this need: "I …

#sql#concept#window-function

99.0% 100% SLA 99.5% external contract · breach pays out SLO 99.9% internal target (aim here) breach missed target (not yet a breach) healthy ← SLI 99.95% actually measured safety buffer: feel the pain inside before the customer does Strictness: SLA (loose) < SLO (strict) ≤ SLI (measured when healthy)

SLI / SLO / SLA: A Measurement, a Target, a Contract

· tech · 12 min read · 📚 Google SRE — Reading Notes #2

The previous post said error budget = 1 − SLO. But what is an SLO? And how does it differ from the o…

#sre#reliability

Set the reliability target at 99.9%, not 100% the remaining 0.1% isn't a regret, it's a "budget" you can spend Error Budget = 1 − SLO requests succeed ≥ 99.9% (SLO target) fail ≤ 0.1% (exaggerated) ≈ 43 min/month of downtime Chasing 100%: cost explodes, marginal benefit → 0 — and users can't tell 99.9% from 100% (their network, phone and Wi-Fi were never that reliable anyway)

What Is SRE? Start with the Error Budget

· tech · 13 min read · 📚 Google SRE — Reading Notes #1

"SRE" is a hot word, and it's very often misread as "slightly more advanced operations" or "a sysadm…

#sre#reliability

raw rows (orders) A · amount 100 A · amount 250 B · amount 80 B · amount 120 B · amount 50 GROUP BY customer collapsed: one row per group customer = ACOUNT=2 · SUM=350 customer = BCOUNT=3 · SUM=250 amount has several values per group (100/250) → can't select it bare; aggregate (SUM/AVG…) into one number

GROUP BY: Collapsing Many Rows into One

· tech · 12 min read · 📚 SQL: I Thought I Knew It #4

Everyone can write GROUP BY, but nearly everyone has also been stopped by column "..." must appear i…

#sql#concept

SQL logic has three values (not two) TRUE condition holds FALSE condition fails UNKNOWN don't know ↑ every comparison with NULL lands here age = NULL and age <> NULL alike WHERE / ON / HAVING let only TRUE through TRUE → ✓ kept FALSE → ✗ dropped UNKNOWN → ✗ dropped

NULL Isn't a Value, It's "Don't Know"

· tech · 13 min read · 📚 SQL: I Thought I Knew It #3

The previous post's LEFT JOIN pads NULL for rows that matched nothing. That NULL is this post's subj…

#sql#concept

Changing tools — always replaced, ever simpler hot framework managed service new platform next year's tool ↑ may be unused in three years ↓ a thirty-year bet you won't regret The unchanging foundation source ingestion storage transformation serving undercurrents security · data management · orchestration · software engineering · DataOps

The Future of Data Engineering: Tools Change, the Foundation Doesn't, Reading Fundamentals of Data Engineering, Ch. 11 (Finale)

· tech · 10 min read · 📚 Fundamentals of Data Engineering — Reading Notes #11

Eleven chapters in, one last question: what will the future of data engineering look like? The book'…

#data-engineering#book-notes

desired state you declare: replicas = 3 Controller control loop compare & correct actual state only 2 right now ① read ② act ③ observe 2 → create 1 → 3 ✓ reconcile loop: keep comparing desired with actual, fix the gap — this is the soul of K8s

What Is Kubernetes? From 'Running Containers' to 'Declaring the State You Want'

· tech · 13 min read · 📚 Kubernetes — Learning Notes #1

Many people find Kubernetes (K8s) hard because they're buried from day one under kubectl, a pile of …

#kubernetes#concept

sensitive data PII · payments… as asset · value analytical insight training ML models supporting decisions as liability · risk breaches compliance fines ransomware collapsed trust every sensitive record is both at once — so: collect only what you must, delete what you should

The Chapter That Matters Most and Gets Ignored Most: Security and Privacy, Reading Fundamentals of Data Engineering, Ch. 10

· tech · 12 min read · 📚 Fundamentals of Data Engineering — Reading Notes #10

Past serving, the lifecycle still has one undercurrent running through the whole thing that hasn't b…

#data-engineering#book-notes#security

modeled data Warehouse · Lakehouse business analytics BI · dashboards embedded analytics in-product, for customers machine learning features · training data Reverse ETL back into CRM · ad platforms

The Last Mile of Data: Serving Analytics and ML, Reading Fundamentals of Data Engineering, Ch. 9

· tech · 13 min read · 📚 Fundamentals of Data Engineering — Reading Notes #9

The previous chapters walked through the source, storage, ingestion, modeling — but all that work on…

#data-engineering#book-notes#data-serving

Normalized orders customers products little duplication · consistent writes but queries need many joins Denormalized · wide table one big table lots of duplication, but few joins suits columnar warehouse analytics OLTP transactions (write-heavy) OLAP analytics (read-heavy)

Making Data Useful: Queries, Modeling and Transformation, Reading Fundamentals of Data Engineering, Ch. 8

· tech · 14 min read · 📚 Fundamentals of Data Engineering — Reading Notes #8

Data has been ingested and stored; the next question is: how do you turn it into something genuinely…

#data-engineering#book-notes#data-modeling

Batch hours – days scheduled · mature (default) Micro-batch seconds – minutes one small batch at a time Streaming ms – seconds event by event · real time high latency · simple · cheap low latency · complex · expensive every step towards real time costs one more unit of complexity and money

Moving Data In: Batch or Streaming? Reading Fundamentals of Data Engineering, Ch. 7

· tech · 15 min read · 📚 Fundamentals of Data Engineering — Reading Notes #7

The source produces data, storage is ready to receive it, and the action in between that moves data …

#data-engineering#book-notes#ingestion

↑ higher: faster, pricier, smaller CPU cache~1 ns RAM~100 ns · volatile SSD~0.1 ms HDD (spinning disk)~10 ms object storage (S3 / GCS)~100 ms · very cheap archive / cold storagemins–hrs · cheapest ↓ lower: slower, cheaper, bigger

Where to Store Data: The Storage Hierarchy and Its Abstractions, Reading Fundamentals of Data Engineering, Ch. 6

· tech · 21 min read · 📚 Fundamentals of Data Engineering — Reading Notes #6

The previous post was about data being born at the source. The first thing after it's born: where do…

#data-engineering#book-notes

K8s Control Plane · Scheduler places pods by resource request / affinity Node 1 on-demand pool (stable) Node 2 spot pool Node 3 spot pool Airflow Scheduler Airflow Webserver Airflow Triggerer Metadata DBPersistentVolume (stateful) Spark Driver Spark Executor Spark Executor Spark Executor Spark Executor executors the Driver requests are spread across nodes by the Scheduler executors read the source / write results back Business DBoutside the cluster · source / sink Airflow pod Spark Driver Spark Executor Metadata DB Business DB

Airflow + Spark on K8s: How Different Nodes Run Different Pods

· tech · 24 min read · 📚 Kubernetes — Learning Notes #8

Airflow handles "when, and in what order" jobs run; Spark handles "getting the big data computed". W…

#kubernetes#airflow#spark#data-engineering

Source systems — owned by others, change on their own application DB (OLTP) API / SaaS files / logs IoT / sensors message queues / streams your boundary Ingestion your lifecycle starts here downstream

Where Data Comes From: Source Systems and Data Generation, Reading Fundamentals of Data Engineering, Ch. 5

· tech · 18 min read · 📚 Fundamentals of Data Engineering — Reading Notes #5

The first four chapters were the big picture — the lifecycle, architecture, choosing technology. Fro…

#data-engineering#book-notes

Transitory surface — frameworks · libraries · hot tools hot framework trendy tool library platform SDK comes and goes with fashion → design it to be swappable Immutable foundation object storage · SQL · networking · Unix / bash

How to Actually Choose Technology: Reading Fundamentals of Data Engineering, Ch. 4

· tech · 15 min read · 📚 Fundamentals of Data Engineering — Reading Notes #4

The previous post was about architecture (the why); this chapter asks next: under that architecture,…

#data-engineering#book-notes

Generation source systems Ingestion move data in Transformation clean, model, aggregate Serving BI, ML, products Storage (spans the middle three stages) Undercurrents — the six currents holding up the whole lifecycle Security Data Management DataOps Data Architecture Orchestration Software Engineering

The Data Engineering Lifecycle: Reading Fundamentals of Data Engineering, Ch. 2

· tech · 10 min read · 📚 Fundamentals of Data Engineering — Reading Notes #2

The previous post gave the definition; this chapter gives the skeleton of the whole book — the data …

#data-engineering#book-notes#lifecycle

Pain not there yet → go light Local PySpark · dbt + warehouse Direct API calls · two layers Cheap and easy to maintain Pain is real → heavy weapons Spark cluster · Kafka Airflow · multi-layer Medallion Powerful, but needs daily feeding Pain threshold Pain magnitude (data volume · number of sources · orchestration complexity) →

Confirm the Pain Before You Bring In the Heavy Weapons

· tech · 7 min read

This is a concept note — a judgment call I keep reusing, split out into its own post so other articl…

#concept#data-engineering