System Design Key Concepts
What is Caching?
Caching stores copies of frequently accessed data in a temporary, high-speed layer so reads can be served without going all the way to the source.
The motivation is load. Your database is the single source of truth, but if a million users request their profile at the same time, the database becomes the bottleneck, and latency climbs or the system falls over. Putting a cache in front of it absorbs most of that traffic. When the cache holds the requested data it returns it immediately, which is a cache hit; when it does not, the request falls through to the database, which is a cache miss. As long as most requests are hits, the database sees only a fraction of the load.
Where to Cache?
Caching can happen at almost every layer between the user and the database. Each layer trades a different balance of speed, freshness, and complexity.
1. At Client End
Data is stored directly on the user's device, in the browser or the mobile app.
Mobile apps keep data in an embedded database such as SQLite or Realm. This is what makes offline mode possible: the user can read and even make changes with no connection, and the app reconciles those changes with the backend once it is back online.
Browsers cache static assets like images, CSS, and JavaScript using HTTP headers such as Cache-Control. Once a file is cached the browser stops re-downloading it on every visit, which is one of the cheapest ways to speed up page loads.
2. At Server End (In-Memory)
The cache lives in the RAM of the server running your backend code. Access is as fast as it gets because there is no network hop, but it creates a consistency problem. Each server has its own copy, so when Server A updates its cache, Server B keeps serving the old value until something invalidates it. Working around this means pinning users to one server with sticky sessions, or building cross-server invalidation, both of which add complexity.
3. At CDN (Content Delivery Network)
A content delivery network caches static data such as images, video, CSS, JavaScript, and even whole HTML landing pages.
A CDN is a network of edge servers spread around the world. A request is served from the edge server physically closest to the user, so someone in London is served from a London edge node instead of waiting on the origin server in New York. The shorter distance is a large, direct cut in latency, and the origin is only touched when the edge misses.
4. External Cache (Redis/Memcached)
An external cache is a separate cluster dedicated to caching, typically Redis or Memcached, that all your backend servers talk to.
Because every server reads and writes the same cache, the inconsistency of per-server in-memory caching disappears: there is one shared copy of the data. It also scales on its own. When you need more cache capacity you add Redis nodes without touching the application tier.
5. Database Caching
Most databases cache internally, so some caching happens even if you add none yourself. PostgreSQL and MySQL keep recently used rows and index pages in a region of memory called the buffer pool. Common queries are then served straight from RAM instead of slow disk I/O, and it all happens transparently to the application.
Cache Replacement Policies
When the cache is full, something has to be evicted to make room for new data. The policy decides what goes:
- LRU (least recently used) evicts the item untouched for the longest time. It is the sensible default for most workloads.
- LFU (least frequently used) evicts the item with the fewest accesses, which suits stable access patterns where past popularity predicts future use.
- FIFO (first in, first out) evicts the oldest inserted item regardless of how often it is read. Simple, but it ignores how hot an item is.
- LIFO (last in, first out) evicts the most recently added item, which only makes sense in narrow stack-shaped workloads where older data stays the most valuable.
Cache Write Strategies
The other half of the problem is writes. When data changes, how do you keep the cache and the database in agreement?
1. Write-Through Caching
With write-through, the application writes to the cache, and the cache synchronously writes to the database before the call returns. The write is not done until both are updated.
The payoff is consistency. The cache is never ahead of or behind the database, so reads are always fresh. The cost is latency, because every write is really two writes and the application waits for both.
2. Write-Around Caching
With write-around, the application writes straight to the database and skips the cache entirely. The cache is filled only later, when a read misses and pulls the value from the database.
This suits write-heavy data that is rarely read back soon, because it keeps the cache from filling up with entries nobody requests, a problem called cache pollution. The downside is that the first read after a write is always a miss and pays the full database latency.
3. Write-Back (Write-Behind)
With write-back, also called write-behind, the application writes only to the cache and gets an immediate success. The cache then flushes to the database asynchronously in the background.
This gives the fastest writes of any strategy, which is why it suits write-heavy data like counters. The risk is durability. If the cache server crashes before it flushes, the unsynced writes are gone for good.
Cache Invalidation Methods
Keeping stale data out of the cache is one of the hardest problems in distributed systems. The common approaches are:
- Purge, or write-based invalidation, explicitly deletes the affected keys the moment the underlying data changes, usually right after a database write. It keeps the cache tight and consistent, but doing it efficiently is hard when keys are sharded across many cache nodes.
- Refresh updates a key in place by re-fetching the latest value rather than deleting it, often as a background job for hot keys. Users always get fresh data with no miss penalty, at the cost of work that is wasted when the refreshed value is never read.
- TTL (time-to-live) gives every key a lifespan, say 300 seconds, after which it is discarded automatically. It is the easiest to implement but leaves a window of inconsistency where readers see stale data until the key expires.
- Stale-while-revalidate serves the expired value immediately while kicking off a background fetch to refresh it. The current reader gets a fast, slightly stale response and the next reader gets fresh data, which avoids the latency spike of a hard miss.
Cache Read Strategies
1. Read-Through Cache
In a read-through cache the application treats the cache as its data store and never talks to the database directly. It asks the cache for a key, and on a miss the cache loads the value from the database, stores it, and returns it. This keeps application code simple, but it requires a cache that knows how to fetch from your database.
2. Read-Aside (Cache-Aside)
With cache-aside, also called read-aside, the application drives the flow. It checks the cache first, and on a miss it reads from the database and then writes the value back into the cache for next time. It is robust and works with any cache and database pairing, which is why it is the most common read pattern in practice.
Common Caching Problems
1. Cache Avalanche
A cache avalanche happens when a large number of keys expire at the same instant. All the traffic those keys were absorbing falls through to the database at once, and the sudden spike can overwhelm or crash it. The closely related thundering herd is the same effect concentrated on a single hot key that many requests want the moment it expires.
The fix is to stop everything expiring together. Add random jitter to the TTL so that instead of a flat 60 minutes, keys expire at 60 minutes plus a random 0 to 300 seconds. Spreading the expirations out smooths the load the database sees.
2. Cache Penetration
Cache penetration happens when requests ask for data that does not exist, for example id = -999, whether from a bug or a malicious client. The cache misses, the request hits the database, and the database also returns nothing. Because empty results are usually not cached, every repeat of that request misses again and pounds the database, which can amount to a denial of service.
The fix is to cache the empty result with a short TTL, say 30 seconds, so repeats are absorbed, or to put a Bloom filter in front of the cache to reject keys that cannot exist before any lookup happens.
3. Hot Keys
A hot key is a single key, say a celebrity's profile, that takes millions of requests per second. Even with a distributed cache cluster, any given key lives on one node, so that one node is swamped while the rest of the cluster sits idle.
There are two common fixes. Put a small local cache, an L1, on the application servers for those specific keys so most reads never reach the shared cache. Or replicate the hot key across several nodes by appending a suffix, turning bieber into bieber_1, bieber_2, and so on, so the load spreads across the cluster.
Caching in System Design Interviews
In an interview, knowing when to introduce caching matters as much as knowing how.
Start without it. Sketch the basic path first, from load balancer to application to database, and add a cache only once you can point to the database as the bottleneck for a read-heavy workload. Reaching for caching before there is a problem to solve is a red flag.
Then justify the specific choice. Say why Redis rather than local memory, for instance because you need shared state across distributed servers. And name the trade-off out loud: caching buys speed at the cost of consistency, so pair it with an invalidation strategy such as TTL to show you understand the operational cost, not just the benefit.
Summary
Caching is the main lever for scaling read-heavy systems. Storing data closer to the user with a CDN or client cache, or in faster memory with Redis, cuts latency and shields the database from load. The price is consistency, which is why the write strategy, the invalidation method, and the failure modes matter as much as the cache itself. Cache-aside for reads, a deliberate write strategy, and defenses against avalanche, penetration, and hot keys are the pieces worth having at your fingertips.