Load Balancing: How One Address Fans Out to a Hundred Servers

One server can hold maybe 10,000 concurrent connections before its CPU, memory, or socket table gives out. When a product grows from 100 users to 100,000, no single box survives — so you run twenty, leaving clients with a question they should never have to answer: which of the twenty do I talk to?

A load balancer removes that question. It sits behind one stable address — a virtual IP, or api.example.com — and forwards each request to a backend that can serve it. Clients see one endpoint; behind it you can add capacity, drain a broken node, or roll a new version, unnoticed.

That indirection buys scale (add backends for capacity), availability (route around dead nodes), and flexibility (deploy and rebalance without downtime). The interesting engineering is how it picks a backend, knows which ones are healthy, and survives losing itself.

L4 or L7: where the balancer makes its decision#

The first fork is which layer of the network stack the balancer inspects.

An L4 (transport-layer) balancer works on TCP/UDP: it sees IP and port, and forwards packets without opening the payload. It just moves bytes, which is why it's fast — millions of connections per second per box, negligible added latency. AWS NLB and HAProxy in TCP mode work this way.

An L7 (application-layer) balancer terminates TLS and reads the request, so it can route on path, header, or cookie — send /video/* to the media fleet, pin a session, run a WAF, canary traffic. That intelligence costs CPU and roughly 1–5 ms of latency. Envoy, NGINX, and AWS ALB live here.

Reach for L4 when the workload is non-HTTP, throughput-bound, or needs DSR. Reach for L7 when routing needs to read the request itself. Large systems run both: a stateless L4 tier at the edge feeding an L7 tier for application-aware calls.

How does a load balancer pick a server?#

This is where most of the subtlety hides, because the obvious algorithm is quietly bad.

Round-robin hands request 1 to A, request 2 to B, request 3 to C, then back to A — even counts, blind to load. A slow request leaves round-robin stacking more work onto a backend that's already drowning.

Least-connections tracks in-flight requests per backend and routes to whoever has fewest. Better under uneven costs, but it needs a global, constantly-updated count, and under bursts every instance can spot the same "least loaded" node and pile on at once.

Power-of-two-choices (P2C) is what modern proxies default to instead: sample two backends at random, send the request to whichever is less loaded. One comparison, near-optimal distribution, O(1) state, no coordination — Mitzenmacher's result shows why: random placement alone leaves max load around O(log n / log log n), but comparing two candidates collapses that to O(log log n). Envoy and Finagle pair P2C with an EWMA of recent latency.

Walkthrough: one slow request among three backends#

Backend A gets stuck on a slow request (cost 100); everything else costs 1. "Load" is total in-flight request cost.

Round-robin sends request 4 straight back to A, already buried under the 100-cost request, since it only counts turns. P2C never does that: whenever A lands in a sample, its lighter rival wins, so A holds at 100 while cheap work spreads across B and C — the gap between a smooth p99 and a tail-latency cliff.

Health checks: the feature that causes outages#

A load balancer must stop sending traffic to broken backends. Sounds simple; the naive version is a top cause of self-inflicted outages.

A shallow check — a TCP connect, or GET /health returning 200 — passes even when the process is useless: DB pool exhausted, GC thrashing, disk full. The balancer keeps routing to a node that fails every real request.

The over-correction is worse: a deep check that queries the database on every probe means that when the database slows, every backend's check times out at once. The balancer marks the whole fleet unhealthy and pulls it from rotation — the health-check death spiral.

The fix is keeping three questions separate:

• Liveness — is the process running? (restart if not)
• Readiness — can it serve traffic now? (pull from rotation if not)
• Deep/dependency — are downstreams okay? (alerting, not capacity decisions)

Then add guardrails: a panic threshold (50% in Envoy) that stops ejecting once too many look unhealthy, and outlier detection, ejecting backends by real success rate with backoff before they return.

Sticky sessions without breaking on rebalance#

Sometimes a request must reach a specific backend — a websocket, or a session cached in one server's memory. The lazy fix, a cookie pinning the client to backend #7, breaks the moment the cookie is lost and shards unevenly.

The durable fix is consistent hashing on a stable key like user_id: hash it, and the same user always lands on the same backend, no cookie required. What makes it "consistent" is that adding or removing a backend moves only about 1/N of keys, not the near-total reshuffle plain hash % N would cause.

Google's L4 balancer, Maglev, implements this as a lookup table (~65,537 slots) each backend fills via a permutation until full — under 1% load imbalance, ~1/N disruption on membership change, O(1) lookup instead of a binary search. Add a small Redis cache for session state so a backend taking over mid-churn pulls it back almost instantly.

Going global: routing users to the nearest region#

Serving users across NA, EU, and APAC turns the balancer question geographic. Three approaches, each with a real weakness:

Anycast gives sub-second failover — Cloudflare and Google use it for ingress — but a BGP route flap can move a live TCP flow to a different PoP mid-connection, which is why flow-aware balancers like Maglev pair with it. Most production setups layer all three: Anycast for the initial hop, DNS for region selection, an app-level redirect only where a session must stay pinned.

How this shows up in interviews#

Load balancing is rarely the whole question — it's the layer interviewers poke to see if you understand what happens under load. Expect follow-ups on why round-robin is a bad default, how to avoid the health-check death spiral, and how sticky sessions survive fleet scaling — plus one harder one: what breaks if the load balancer itself dies?

That last one is the real test. TCP state lives on the balancer, so killing it naively resets every in-flight connection. The answer is a stateless L4 tier: ECMP spreads flows across N balancer VMs, and Maglev-style consistent hashing lets survivors forward each packet to the same backend without shared state — losing one balancer reshuffles only ~1/N of flows instead of dropping everything. For L7, graceful drain (stop new connections, let in-flight ones finish over 30–60s) upgrades binaries with zero dropped requests.

Name what you'd reverse if the workload changed, state the source of truth, and walk one request end to end.

Further learning#

• What is LOAD BALANCING? — Gaurav Sen — a clear conceptual walkthrough of why balancers exist and the core algorithms.
• Load Balancing (Interactive Visual Guide) — samwho.dev — an animated, play-with-it explanation of round-robin, least-connections, and P2C that makes the algorithm differences click.
• Practice it on the Load Balancing roadmap topic, then connect it to CDN, Message Queues, Consistent Hashing, and API Gateway.

FAQ#

What is the difference between L4 and L7 load balancing?#

An L4 balancer works purely on the connection's addressing — source and destination IP plus port — and shovels the raw bytes onward without ever decrypting or parsing them. That blindness is the point: it clears millions of connections a second with latency you can barely measure. An L7 balancer instead unwraps TLS and parses the HTTP or gRPC message, so its routing decision can depend on the URL path, a header value, or a cookie — the prerequisite for canary rollouts, WAF rules, and session pinning. You pay for that in CPU and roughly a millisecond or two per hop. Rule of thumb: L4 for non-HTTP protocols or when throughput is the binding constraint; L7 when the decision has to see inside the request.

Why do modern proxies default to power-of-two-choices instead of round-robin?#

Round-robin spreads request counts evenly but ignores load, so a slow request can bury one backend while it keeps feeding more work. Least-connections tracks load but needs a shared, constantly updated count, causing herd effects under bursts. Power-of-two-choices instead samples two backends at random and picks whichever is less loaded — one comparison, near-optimal, no shared state. It cuts theoretical max load from O(log n / log log n) to O(log log n), why Envoy and Finagle default to it.

How do health checks cause outages instead of preventing them?#

The trap is a probe that's too thorough. If every health check runs a live database query, then the moment the database gets slow, all of those probes time out simultaneously — not because the backends are broken, but because their shared dependency is. Each node fails its check at the same instant, the balancer concludes the entire fleet is dead, and it withdraws your last remaining capacity right when demand hasn't dropped. That's the death spiral. Break the coupling by splitting checks by purpose: liveness answers "is the process alive" and drives restarts, readiness answers "can this node take traffic now" and drives rotation, and dependency status feeds dashboards and pages — never the routing table. A panic threshold (Envoy defaults to 50%) is the backstop: once too much of the fleet looks sick at once, stop ejecting and assume the probe, not the servers, is lying.

How do you keep sticky sessions working when you add or remove servers?#

Cookie pinning is the tempting shortcut and the wrong one — traffic clumps unevenly, and a dropped cookie strands the client. The robust approach hashes something durable about the caller, like their user_id, and derives the target backend from that hash, so no server-side pin or cookie is needed to route them back to where their state already lives. The reason to reach specifically for consistent hashing rather than a plain modulo is what a scale event does to each: with hash % N, changing N repositions almost every key at once and scatters everyone to a new server. Consistent hashing (Maglev, rendezvous) is engineered so that adding or dropping a backend relocates only about one N-th of the keyspace; the rest of your users never notice. Back it with a short-TTL Redis cache for session data, so the handful who do move rehydrate in milliseconds rather than being forced to reconnect.

What happens to in-flight connections if a load balancer instance dies?#

The hazard is that a TCP connection's state — sequence numbers, window, the whole handshake — is remembered only on the balancer handling it, so a box that dies takes that memory with it and its clients all get a reset. You dodge this by making sure no single balancer is irreplaceable. Two mechanisms combine. The switch fabric uses ECMP to hash flows across a pool of balancer VMs, and every one of those VMs runs the same deterministic hash (Maglev-style) over each packet, so any survivor computes the identical backend for a given flow without ever having seen it before. When a VM drops out, ECMP only rehashes the fraction of flows it was carrying — on the order of one N-th — while the rest keep landing exactly where they were. Planned changes are gentler: drain the node first (refuse new connections, give existing ones 30 to 60 seconds to finish), and for L7 you can even hand live sockets across a binary upgrade (SO_REUSEPORT, Katran-style kernel handoff) with zero dropped requests.