System Design Key Concepts
What is a Load Balancer?
A load balancer is a device, hardware or software, that sits in front of a group of backend servers and distributes incoming traffic across them. It acts as a reverse proxy: clients connect to it, and it decides which server actually handles each request.
Think of a busy restaurant. You don't walk into the kitchen to find a free chef; you talk to the host at the door, who seats you at the least busy table. The load balancer plays the host, spreading work so no single server is overwhelmed while others sit idle.
Where Do Load Balancers Sit?
Load balancers are not only at the front door. In a scalable system they sit between every tier:
- User to web server, distributing internet traffic across web servers such as Nginx or an AWS ALB.
- Web server to app server, spreading internal requests across the application logic tier.
- App server to database, routing queries to read replicas or shards.
Putting a balancer at each boundary lets every tier scale and fail over on its own.
Layer 4 vs Layer 7 Load Balancing
This is a favorite interview question. Load balancers operate at different layers of the OSI model, and the layer one works at decides how much it can see.
Layer 4 (Transport Layer)
A Layer 4 load balancer routes purely on transport-level information: the source and destination IP address and port (TCP or UDP). When a packet arrives it rewrites the destination address to a chosen backend (NAT) and forwards it, without ever looking inside the packet.
Because it does almost no inspection, it is extremely fast and can sustain millions of concurrent connections. The trade-off is that the routing is blind. It cannot send /video to your video servers and /chat to your chat servers, because it never sees the URL.
Layer 7 (Application Layer)
A Layer 7 load balancer routes on the content of the request: HTTP headers, the URL path, cookies. To read those it terminates the client's TCP connection, decrypts the request (SSL termination), inspects it, and then decides where to send it.
That inspection enables smart routing, such as sending mobile users to a mobile fleet, and it lets the balancer add caching and compression. The cost is more CPU per request and two TCP connections to manage, one from the client to the balancer and one from the balancer to the server.
| Feature | Layer 4 LB | Layer 7 LB |
|---|---|---|
| OSI Layer | Transport (TCP/UDP) | Application (HTTP/HTTPS) |
| Routing Data | IP + Port | URL, Headers, Cookies |
| Speed | Very Fast | Slower (CPU Intensive) |
| Example | AWS Network Load Balancer | Nginx, AWS ALB |
Comparison: L4 vs L7 Load Balancing
Load Balancing Algorithms: How to Choose?
The algorithm decides which server gets each request. The wrong choice can overload some servers while others stay idle, even when your total capacity is more than enough.
1. Round Robin (The Static Approach)
Round robin hands out requests in a fixed rotation: server A, then B, then C, then back to A. It is trivial to implement and costs almost nothing to compute.
The weakness is that it is completely unaware of load. It will happily send a request to a server already pinned at 100% CPU, and it assumes every server is equally powerful, so it falls apart when one box is far stronger than another. It works best for simple, stateless applications running on identical hardware.
2. Weighted Round Robin
Weighted round robin fixes the equal-hardware assumption. You assign each server a weight based on its capacity, and a server with weight 5 receives five times as many requests as a server with weight 1. This is ideal for clusters with mixed hardware: when you add newer, more powerful machines to an older cluster, give them higher weights so they shoulder more of the traffic.
3. Least Connections (The Dynamic Approach)
Least connections tracks how many open connections each server currently has and sends the next request to whichever has the fewest. This makes it naturally load aware, steering work away from servers that are already busy.
The catch is slow start. A freshly added server has zero connections, so the balancer floods it with every new request until its count catches up, a burst sometimes called the thundering herd. It shines for long-lived connections like WebSockets, chat, or database queries, where connection count is a good proxy for load.
4. Least Response Time
Least response time watches the actual latency of each backend, the time to first byte, and routes to whichever is responding fastest right now. It suits latency-sensitive applications where the user experience lives or dies on speed.
5. IP Hash (Sticky Sessions)
IP hash derives the server from a hash of the client's IP address, roughly hash(IP) % N, so a given client always lands on the same server. That stickiness keeps a server's local cache warm and is sometimes required by apps that store session state on the server itself, though storing session state locally is a design you should usually avoid.
Its weakness is uneven load. If a large corporate office sits behind a single IP, all of its traffic hashes to one server and can crush it while the others stay quiet. Cookie-based stickiness at Layer 7 is the more common way to pin sessions without this IP-clumping problem.
Health Checks (The Life Saver)
A load balancer does more than route; it keeps the pool healthy. It periodically sends a heartbeat to each backend, for example a GET /health every few seconds. If a server misses several checks in a row, say three, the balancer marks it unhealthy and stops routing to it. When the server recovers and starts passing again, traffic resumes automatically. This is what lets a cluster lose a node without users noticing.
Summary for Interviews
When you design a system, a few load-balancing decisions come up almost every time:
- Put balancers between tiers (web, app, database) so each layer has redundancy.
- Default to Layer 7 for web apps, because it gives you content-based routing and SSL termination.
- Drop to Layer 4 when you need extreme throughput or are routing non-HTTP traffic such as raw database connections.
- Call out health checks as the core mechanism for fault tolerance.