Load Balancing: Pick the Right Datacenter, Then the Right Machine

Between a user sending a request and the request being handled, it passes through two layers of load balancing, answering two questions at different levels: which datacenter? and, once inside, which machine? The two layers care about completely different things, and load balancing gets much clearer once you look at them separately.

Two layers: pick the datacenter, then the machine

The frontend LB uses DNS / Anycast / VIPs to steer users to the best datacenter — but DNS has built-in limits (it gets cached, doesn't update until the TTL expires, can't see backend health), so you can't rely on it alone. The in-datacenter LB is where the real per-request, live-load, health-aware fine-grained distribution happens. This layer is exactly what a K8s Service does: it stands in front of a group of Pods and sends traffic only to the healthy ones

The frontend layer deals with problems at the level of “geography and disaster”: steer the user to a datacenter that’s near them, still alive, and has room. The usual tools are DNS, Anycast, VIPs — but DNS has an inherent weakness: it’s cached layer upon layer, doesn’t update until the TTL runs out, and can’t see whether the backend is healthy right now. So DNS can only do coarse-grained splitting; the fine work is left to the in-datacenter layer.

Why Round Robin isn’t good enough

Inside the datacenter, the most intuitive method is Round Robin — requests go to each machine in turn, everyone gets an equal share. Sounds fair, but it quietly assumes three things, and in reality all three are wrong:

Round Robin evens out the "request count", but what we actually want to even out is "load" — and requests come light and heavy, machines strong and weak, so the two aren't equivalent at all. Hence weighting by the live utilisation the backends report. The most insidious trap: a broken machine that "returns errors instantly" looks the most attractive to a "pick the least busy" strategy, so all the traffic pours in and makes things worse

So the better approach is to have backends actively report their live utilisation, and let the LB weight by it (Weighted Round Robin) — busy ones get less, weak ones get less, unhealthy ones get none. And that trap of “fast failure attracts traffic” is in essence a form of cascading failure: the broken node not only isn’t isolated, it’s rewarded with more traffic.

Two practical details that are easy to overlook

Subsetting: if every client opens connections to every backend, N × M connections blow up. In practice each client connects only to a subset — saving a great deal of connection and health-check cost without giving up much balance.
Lame duck state: when taking a machine out of service, don’t just kill it — it may have requests half-processed. The right way is to enter “lame duck” first: tell the LB to stop sending new requests, but finish the ones in hand, drain, and only then shut down. Healthy → lame duck (draining) → dead, not a clean chop. It’s the same thinking as K8s readiness probes plus graceful shutdown.

Reflections

”Evenly distributed” isn’t “evenly loaded” — a trap I’ve stepped in

Round Robin’s most seductive quality is that it looks so fair — each machine takes one in turn; what could be more even? But I’ve been burned by it myself: years ago I built a service with Round Robin in front, the load-test numbers looked great, and after launch there were always one or two machines with unusually high CPU and occasional timeouts. It took a long dig to understand that the problem wasn’t the machines; it was that I’d evened out the wrong thing: I balanced “request count”, but some requests ran a very heavy query and some returned instantly, and the machine specs were a mix of new and old. Equal counts, wildly different load. Since then, for any “even split” mechanism I ask one more question: is the unit I’m splitting the same thing as what I actually want to balance? Splitting requests, connections, partitions — these are often not the same as splitting “real work”.

Broken things attracting traffic is the most counter-intuitive failure

“Pick the least busy machine” sounds unquestionably right, until you realise: a machine spraying errors at top speed is, in the eyes of “pick the least busy”, the least busy one — it responds fast (with errors, but fast) and its queue is empty. So the load balancer cheerfully steers all the traffic there, which amounts to escorting every user into the fire. That trap taught me: health can’t be judged by “responds quickly” alone; it has to be “actually did the job right”. Fast failure that isn’t correctly marked unhealthy is more dangerous than slowness — it disguises itself as high performance. It’s also why health checks (monitoring) have to look at success rate, not latency alone.

Graceful exit is the dividing line of a mature system

The lame duck concept struck a chord, because “how to take a machine down safely” looks small yet best separates mature systems from immature ones. An immature system takes machines down with a clean chop — every half-finished request becomes an error in a user’s eyes; a mature system blocks new work first, finishes the old, and leaves only once drained. I’ve felt the same thing repeatedly on K8s: when a Pod is about to be replaced, it first gets removed from the Service‘s list and stops taking new traffic, then gets a grace period to wrap up. Whether you can exit gracefully often shows more skill than whether you can launch impressively — because on exit you’re facing “real traffic in flight”, and that can’t be faked.