Tech

✗ Dual write: the app writes three copies application DB cache search index disease 1: crash halfway → some written, some not no transaction can roll back across three systems disease 2: concurrent writes arrive in different orders DB gets A then B, cache gets B then A → different values → three systems diverge permanently, silently ✓ Log first: write to one place only application only writes here one ordered log (source of truth) DB cache search index all consume in the "same order" → consistent, replayable every downstream is a follower, converging to one state One dataset into N systems? Pick one source of truth, make all the rest followers

Stream Processing: The Dual-Write Trap, CDC, and Stream–Table Duality

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #11

Batch processes data that has "all arrived"; streaming processes data that "keeps coming". Much of t…

#distributed-systems#book-notes#streaming

input (immutable) log shard 1 log shard 2 log shard 3 ① map (in place, parallel) (/a,1)(/b,1)… (/a,1)(/c,1)… (/b,1)(/a,1)… extract (key, value) per record, no data moved ② shuffle (redistribute by key) same key → same node, sorted the only big move = most expensive ③ reduce (aggregate groups) /a:(1,1,1)→ /a=3 /b:(1,1)→ /b=2 … output: written as "new" files map runs in place (bring compute to the data) · shuffle is the only big move, and where all the cost lives groupBy, join, dedupe… every "same key must meet" operation is a shuffle underneath

Batch Processing: The Spirit of MapReduce, Two Roads to a Join, and the Virtue of Immutable Inputs

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #10

Part II held consistency together inside a single system; Part III's theme switches to data flowing …

#distributed-systems#book-notes#data-engineering

Final whistle: result written, but two replicas at different progress leader: 2 : 1 final replica 1 (caught up): 2:1 final replica 2 (lagging): 1:1 playing ① Alice sees "final 2:1" → tells Bob ② Bob refreshes → "still playing"?! A "later" read sees an "older" state → the single-copy illusion shatters = not linearizable

Consistency and Consensus: Linearizability, the Honest Version of CAP, and Total Order Broadcast

· tech · 16 min read · 📚 Designing Data-Intensive Applications — Reading Notes #9

The previous post concluded that no single node's judgment can be trusted, so truth can only be deci…

#distributed-systems#book-notes

your node sent request, waiting… ① request lost on the networkthe other side never received it ② the other node crashedactually dead ③ the other side is just slow (overloaded, in a GC pause)it'll handle it in a moment — may be handling it right now ④ it finished, but the "response" was lost on the way backthe action already happened, and you think it didn't from your end: all four identical = no response A timeout firing means "you decided to stop waiting", not "you know what happened"

The Trouble with Distributed Systems: Unreliable Networks, Untrustworthy Clocks, and Half-Dead Nodes

· tech · 13 min read · 📚 Designing Data-Intensive Applications — Reading Notes #8

The previous post ended on a hook: once a system spans several machines, even "locks" and "validatio…

#distributed-systems#book-notes#reliability

Invariant: doctors on call ≥ 2 (now: Alice and Bob both on) Alice's txn ① count on call → reads 2 ≥ 2 ✓ ② set "self" to on leave Bob's txn ① count on call → also reads 2 ✓ ② set "self" to on leave (snapshot isolation: both see the old snapshot's 2) (different rows → no write conflict, both commit) Result: doctors on call = 0, invariant broken 💥 Each txn is right "on its own"; the error: the condition you checked was silently changed by the other's write Same script: double-booked meeting room, two accounts claiming a username, a balance debited twice at once

Transactions: The Write Skew Snapshot Isolation Can't Stop, and Three Roads to Serializability

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #7

I laid the foundations of transactions in the SQL series: ACID's emphasis is on the I, the three ano…

#distributed-systems#book-notes

By key range A–F G–R S–Z like encyclopedia volumes, keys ordered ✓ range scans efficient (one contiguous read) ✗ key is a timestamp → today's writes all hit the last partition (hot spot) HBase / early Bigtable By hash of key partition 0 partition 1 partition 2 hash sprays adjacent keys evenly ✓ load even, hot spots scattered ✗ order lost → a range scan must ask every partition Cassandra / Redis Cluster (CRC16) / Kafka Compromise: compound key (hash first column → partition, sort the rest within it) — Cassandra's signature Hash can't save a single extremely hot key (the celebrity problem) — salt it in the app, split one key into many

Partitioning: Range or Hash, Where Secondary Indexes Live, and How to Rebalance

· tech · 13 min read · 📚 Designing Data-Intensive Applications — Reading Notes #6

Replication keeps the same data on several machines; partitioning (also called sharding) splits the …

#distributed-systems#book-notes

Single-leader Leader follower follower all writes conflicts: killed at source (one writer) price: failover is hard (who takes over?) MySQL / Postgres / Redis / Kafka Multi-leader Leader A Leader B datacenter 1datacenter 2 both edit the same row → conflict! gain: write locally, survive a netsplit price: resolve write conflicts later multi-DC / offline editing / collaborative docs Leaderless replica replica replica client writes n copies at once quorum: w + r > n gain: no leader, no failover price: complex paths, read repair Dynamo / Cassandra Three answers to one question: block conflicts at the source, resolve later, or reconcile on read?

Replication: Single-Leader, Multi-Leader, Leaderless, and Three Replication-Lag Anomalies

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #5

Into Part II, and data starts crossing machines. The first move is replication: the same data on sev…

#distributed-systems#book-notes

Mid rolling upgrade: old and new both running, data flows both ways new code v2the nodes already upgraded old code v1the nodes not yet reached same database / topic both sides write, both sides read backward compatibility new code can read old data everyone remembers this (migration mindset) forward compatibility old code can read new data most forgotten — but daily in rollout and rollback Every schema change needs both — miss one and errors start mid-rollout

Encoding and Evolution: Letting Old and New Code Read Each Other's Data

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #4

The previous post was about how data gets onto disk. This one is about a problem that's easier to un…

#distributed-systems#book-notes#data-engineering

LSM-tree: append-only, tidy later memtablein memory, catches writes full → flush batch sequentially SSTable (new, immutable) SSTable (older) SSTable (oldest) compactionbackground merge + dedupe write: always sequential append → fast read: may search several SSTables (bloom filter helps) B-tree: page tree, update in place root page page page ✎ overwrite in place leaf page WAL: sequential log entry before touching a page read: walk 3–4 levels down → steady write: random I/O in place + WAL first LSM: write-optimised (RocksDB / Cassandra / HBase) · B-tree: steady reads (nearly every relational DB)

Storage Engines: LSM-Trees, B-Trees, and Column-Oriented Storage

· tech · 12 min read · 📚 Designing Data-Intensive Applications — Reading Notes #3

The previous post chose the data model. This one drills to the very bottom: how does a database actu…

#distributed-systems#book-notes#storage

Pub/Sub: broadcast, gone is gone PUBLISH news "…" channel: news (not stored) Sub online✓ got it Sub online✓ got it Sub offline✗ missed nobody listening → gone · no persistence · no replay · no ack Stream: a kept log, replayable XADD stream * … (append) m1m2m3m4m5 old consumeranother reads from here messages kept (MAXLEN cap optional) re-read anywhere · catch up after being offline · groups + ack

Pub/Sub vs Stream: Redis's Version of a Messaging System

· tech · 12 min read · 📚 Redis — Learning Notes #12

Redis can serve as a messaging system too, but it has two completely different things for it, and us…

#redis#distributed-systems

One by one N × RTT vs pipeline 1 × RTT ① one by one: each waits a round trip clientserver 3 cmds = 3 round trips = 3 × RTT ② pipeline: packed into one round trip clientserver cmd1 ; cmd2 ; cmd3 sent together three replies back together 3 cmds = 1 round trip = 1 × RTT ✓

Pipelining, Transactions and Lua: Saving RTTs vs Atomicity

· tech · 12 min read · 📚 Redis — Learning Notes #11

Three things are often lumped together that actually solve completely different problems: pipelining…

#redis#distributed-systems

Where a key lands: CRC16 → 16384 slots → node key: "user:1000" CRC16(key) % 16384= slot 5798 Node Aslot 0 – 5460(+ one replica) Node B ✓slot 5461 – 109225798 is here → B holds it Node Cslot 10923 – 16383(+ one replica) the 16384 fixed slots are a "middle layer"; a node just claims a range of slots so moving data = moving slots; scaling stays clean, no recomputing every key's place

Redis Cluster: How 16384 Slots Shard and Rescale

· tech · 13 min read · 📚 Redis — Learning Notes #10

The single-thread post said one Redis's bottleneck is memory and the network. When one machine can't…

#redis#distributed-systems

The five steps of Sentinel automatic failover ① monitorkeeps pinging masterand replicas ② subjectively downone Sentinel seesno response (SDOWN) ③ objectively downmajority agree "really dead"(ODOWN) ④ elect a Sentinel leader → promote a replicapick the most complete replica as the new masterpoint the other replicas at the new master ⑤ tell clients the new master's address clients askSentinel"who is themaster now?"

High Availability: How Sentinel Fails Over Automatically

· tech · 12 min read · 📚 Redis — Learning Notes #9

The previous post's replication gave you copies, but left a big hole: when the master dies, nobody t…

#redis#high-availability

One master writes, many replicas read applicationwrite → masterread → replicas Masterread/write · sole write pointonly one write Replica 1read-only Replica 2read-only async replication readreads spread over the replicas → scale reads horizontally

Master-Replica Replication: Read/Write Splitting and the Oddities of Replication Lag

· tech · 11 min read · 📚 Redis — Learning Notes #8

However fast one Redis is, it has ceilings on memory and traffic, and when it dies the data hangs in…

#redis#distributed-systems

Acquiring the lock: one atomic command SET lock:res <token> NX PX 30000 NXset only if the key doesn't exist→ mutual exclusion PX 30000built-in 30 s TTL→ holder dies, auto-release; no deadlock <token>a random value→ marks "this lock is mine" Releasing the lock: verify the owner (Lua, atomic) if GET(key)==token then DEL(key)GET and DEL must be atomic → only delete your own lock DEL without checking token→ deletes someone's renewed lock ✗

Distributed Locks: From SETNX to Redlock, and That Famous Argument

· tech · 14 min read · 📚 Redis — Learning Notes #7

When several processes or machines compete for the same resource (only one may decrement stock or ru…

#redis#distributed-systems

① scheduled to a nodethe Scheduler picks one ② image pulledkubelet pulls from the registry ③ container up · alivestarts and doesn't crash ④ Ready · in Endpointstraffic only after readiness passes ✓ serving traffic Pendingnot enough resources / taint without toleration / PVC won't bind ImagePullBackOff · ErrImagePullimage name typo / private registry missing imagePullSecret CrashLoopBackOff · OOMKilledcrashes on start, restarts forever / killed for exceeding memory limit Running but not Ready · unreachablereadiness failing / Endpoints empty (bad selector)/ DNS won't resolve / NetworkPolicy blocking

Troubleshooting: How to Investigate Pods, Nodes and the Control Plane

· tech · 16 min read · 📚 Kubernetes — Learning Notes #13

The biggest slice of the CKA is troubleshooting (30%), but it's really not new knowledge — it's the …

#kubernetes#troubleshooting

Control Plane node · kubeadm init api-server scheduler controller-mgr etcd all static pods: kubelet reads /etc/kubernetes/manifests brings them up and keeps them running prints a join token + command kubeadm join <token> Worker noderuns only the kubelet + your Pods Worker nodeadd as many as you need add more CP nodes (join) → HA control plane

Cluster Administration: kubeadm, etcd Backups, Upgrades

· tech · 13 min read · 📚 Kubernetes — Learning Notes #12

The previous eleven posts all stood in the position of "using the cluster". This one moves to "build…

#kubernetes#operations

kubectl / Pod sends a request ① authn who are you? certs · token · OIDC ② authz may you do this? ← this is RBAC execute API Server identity unverified → 401 no permission → 403 two gates, one job each: first confirm "who you are", then decide "whether you may"

RBAC: Who Can Do What to the Cluster

· tech · 15 min read · 📚 Kubernetes — Learning Notes #11

The previous ten posts were about making things "run and be reachable". This one switches dimension:…

#kubernetes#security