The load balancer is the box every candidate draws and almost nobody defends. It appears in the first thirty seconds of a system design answer, receives no further attention, and then the interviewer asks one question about health checks and the answer falls apart.
The material below is what those follow-ups actually cover.
Layer 4 or layer 7
A load balancer distributes requests across a pool and stops sending traffic to instances failing health checks. Where it operates changes what it can do.
Layer 4 forwards on IP and port without reading the payload. It is fast, works for any TCP or UDP protocol, and cannot make decisions based on content. It suits high-throughput non-HTTP traffic, and cases where you must not terminate TLS.
Layer 7 terminates the connection and reads the request. That unlocks the features most web architectures actually want: routing by path or header, per-tenant routing, header-based canary releases, request-level retries, response compression, and TLS termination in one place.
The decision rule worth stating: if you need to make a decision based on anything inside the request, you need layer 7. Most modern edges are layer 7 precisely because those capabilities are the reason the component exists.
Choosing the algorithm
Round robin is fine when requests cost roughly the same — uniform API traffic, static content. It is the default for a reason and there is no shame in choosing it, provided you say why.
Least connections wins when request cost varies widely. A server stuck handling several slow requests stops receiving new ones, whereas round robin keeps feeding it until it falls over. If your endpoints range from a cached lookup to a report generation, this is usually the right pick.
Weighted variants handle heterogeneous hardware, which matters during a migration when half the fleet is a newer instance type.
Consistent hashing is the interesting one. Hashing on a key routes the same client or the same cache key to the same instance, which preserves cache locality and sticky sessions — and unlike naive modulo hashing, adding or removing an instance reshuffles only a small fraction of keys instead of all of them. Mention it when the design has a per-instance cache; it is the difference between a deploy warming caches gradually and a deploy cold-starting every cache at once.
Health checks: where outages are made
This is the question that separates people who have run this in production.
Active checks poll an endpoint on an interval and remove an instance after a threshold of consecutive failures. Passive checks observe real traffic and eject instances that return errors. Most production setups run both.
The trap is depth. If the health endpoint verifies the database connection, then a brief database problem fails the health check on every instance simultaneously, the load balancer removes the entire fleet, and a degradation becomes a total outage. The fleet was fine; the check reported on something the fleet did not control.
Two related details worth having ready: thresholds should require several consecutive failures so a single slow response does not eject a healthy instance, and there must be a floor below which the balancer stops removing instances — serving degraded is better than serving nothing.
What happens during a deploy
Interviewers ask this because it produces a visible error spike that teams often misattribute to application bugs.
When an instance goes away, in-flight requests must finish. Connection draining stops new requests going to the instance while allowing existing ones to complete within a timeout, after which they are terminated. Without it, every deploy and every scale-in produces a burst of 502s.
The sequence that avoids it: mark the instance unhealthy or deregister it, wait for the balancer to notice, then let in-flight work finish, then terminate. Skipping the middle step is the common mistake, because the application shuts down before the balancer has stopped sending it traffic.
"The load balancer is a single point of failure"
Acknowledging it is most of the answer.
A managed load balancer is already a redundant fleet behind a single address, but the reasoning still matters and interviewers want to hear it: multiple instances across availability zones, DNS or anycast distributing across them, and health-based failover between them. For a self-managed setup, the classic pattern is an active-passive pair sharing a virtual IP.
The honest caveat worth adding: DNS failover is slow, because resolvers cache beyond your TTL. If seconds matter, anycast or a managed global balancer is the answer rather than DNS.
What separates a strong answer
Weak:
"I'd put a load balancer in front of the app servers to distribute traffic."
Strong:
"Layer 7 at the edge so we can route by path and do header-based canaries. Least connections, because our report endpoints are much heavier than our reads. The health check hits a shallow endpoint that doesn't touch the database — a deep check pulls the whole fleet during a database blip — and instances drain for thirty seconds before termination so deploys don't produce 502s."
Four sentences, four defended decisions, and every one of them is something the interviewer would otherwise have had to ask about.
Practice these
- Design an API Gateway — the layer 7 component that does routing, auth and rate limiting together
- Design a Distributed Rate Limiter — what usually sits alongside the balancer at the edge
- Design a Content Delivery Network (CDN) — global traffic distribution, one layer further out
- Design a Key-Value Store — where consistent hashing does the same job for data placement
Related concepts: load balancer, circuit breaker, rate limiting, tail latency.
Frequently asked questions
What is the difference between a load balancer and a reverse proxy? Every load balancer is a reverse proxy; not every reverse proxy balances load. The terms overlap heavily in practice, and no interviewer will penalise using them interchangeably as long as you are clear about which behaviours you mean.
Round robin or least connections by default? Round robin when request costs are uniform, least connections when they vary. If you do not know the distribution, least connections is the safer default because it degrades more gracefully when one instance slows down.
How do sticky sessions work, and should you use them? The balancer pins a client to an instance via a cookie or a hash of the client address. They are best avoided — they undermine even distribution and complicate deploys — but they are sometimes the pragmatic answer for legacy applications holding in-memory session state.
Where does TLS termination belong? Usually at the load balancer, which centralises certificate management and lets it read requests for layer 7 routing. Where internal traffic must also be encrypted, terminate at the edge and re-encrypt to the backend, accepting the extra CPU cost.
How does a load balancer handle WebSockets? Long-lived connections change the calculus: least connections becomes more important because a connection may live for hours, idle timeouts must exceed expected connection lifetime, and draining takes far longer since you cannot wait for a WebSocket to finish naturally.