Redis

Redis (REmote DIctionary Server) is an in-memory data store that keeps its entire dataset in RAM and answers most operations in well under a millisecond. Developers usually reach for it first as a caching layer, but that undersells it. Redis is a data structure server: it gives you strings, lists, sets, sorted sets, hashes, streams, and more as first-class server-side types, with atomic operations on each. That combination of speed and rich primitives is why the same system shows up as a cache, a rate limiter, a leaderboard, a session store, a job queue, and a message broker, often in the same application.

A read from main memory takes on the order of 100 nanoseconds. A read from an SSD is tens of microseconds, and a random read from a spinning disk is several milliseconds. By keeping everything in RAM, Redis skips the slowest part of a traditional database, the trip to disk, on every operation. That's the whole performance story in one sentence: it's fast because it doesn't touch disk on the hot path.

The feature that separates Redis from a plain key-value cache like Memcached is that values are typed data structures, and the operations on them run on the server. Instead of reading a list into your application, modifying it, and writing it back (three round trips and a race condition), you send one command and Redis mutates the structure in place, atomically. Picking the right structure for the job is most of the skill in using Redis well.

Take a leaderboard that ranks millions of players by score. In most stores that's awkward: you sort on read or maintain a separate index. In Redis it's a single sorted set. ZADD leaderboard 4200 player:42 records a score, ZINCRBY bumps it, ZREVRANK leaderboard player:42 returns the player's rank, and ZREVRANGE leaderboard 0 9 WITHSCORES returns the top ten. All of it's server-side, atomic, and fast, because the sorted set keeps members ordered as you write them.

Redis executes commands one at a time on a single thread, in the order they arrive. This sounds like it should be a bottleneck, but in practice it's a large part of why Redis is both fast and easy to reason about. There are no locks, no mutexes, and no lock contention, because there's nothing to contend over. Every individual command is atomic for free: while INCR runs, no other command is running, so two clients incrementing the same counter can never lose an update. The CPU stays busy serving requests instead of coordinating between threads.

Every command is normally a round trip: the client sends one command, waits for the reply, then sends the next. The waiting is the problem. If the network round-trip time is half a millisecond, a thousand commands issued one after another spend about half a second just waiting, even though Redis handles each one in microseconds. Pipelining removes that wait by sending a batch of commands at once and reading all the replies together. Redis does the same work; what disappears is the round-trip pause between commands, which is why a bulk load can drop from hundreds of milliseconds to single-digit milliseconds. Pipelining is purely a latency optimization, not a transaction: the batched commands aren't atomic, and another client's commands can still run in between yours.

A transaction is the tool when you do need a group of commands to run together with nothing interleaved. You start it with MULTI, queue your commands (each one replies QUEUED instead of running), and fire them with EXEC. Because Redis is single-threaded, once EXEC begins the whole batch runs to completion before any other client is served, so no other command can slip in. For a read-modify-write that has to be safe against concurrent changes, you add WATCH: you watch one or more keys before MULTI, and if any of them changes before EXEC, the EXEC aborts and returns nil so you can retry. That's optimistic locking, and it's how you do a safe check-then-set without holding a lock.

Because the data lives in RAM, a restart or crash would lose everything unless Redis wrote something to disk. It offers two mechanisms, and you can run either, both, or neither depending on how much data you can afford to lose.

The durability dial that matters most is the AOF fsync policy. With appendfsync always, Redis flushes every write to disk before acknowledging, which is the safest and the slowest. With appendfsync everysec (the default), it flushes once per second, so a crash can lose up to one second of writes in exchange for far better throughput. With appendfsync no, it lets the operating system decide when to flush, which is fastest and least safe. The common production choice is RDB for fast restarts plus AOF with everysec for bounded loss, and Redis can combine them so the AOF starts from an RDB preamble.

Two different mechanisms remove keys, and they're easy to confuse. Expiration is something you ask for: you set a TTL with EXPIRE key 60 or SET key val EX 60, and the key disappears when its time is up. Eviction is something Redis does to you: when memory hits the maxmemory limit, it has to free space to accept new writes, and the maxmemory-policy decides which keys to sacrifice.

A single Redis node is a single point of failure. Replication fixes the read side and the durability side: a master accepts writes and streams them to one or more replicas, which serve reads and stand ready to take over. Replication is asynchronous by default, so the master acknowledges a write to the client before the replicas have it. That keeps writes fast, but it means a master can fail with a few writes that never reached any replica, and those writes are lost on failover.

Replication alone doesn't give you automatic failover; something has to notice the master is dead and promote a replica. Redis Sentinel is that something: a set of Sentinel processes monitor the master and replicas, agree by quorum that the master is down, elect a replica, promote it, and tell clients about the new master. Sentinel handles failover for a single master with its replicas. When you also need to shard data across many masters, that's Redis Cluster, which has its own failover built in.

Replication scales reads, but every write still goes to one master and the whole dataset still has to fit on one machine. When the data or the write rate outgrows a single node, Redis Cluster shards the keyspace across several masters. It divides the key space into 16384 hash slots; every key is mapped to a slot by CRC16(key) mod 16384, and each master owns a contiguous range of slots. Add a node and you move some slots (and their keys) to it; this is how the cluster rebalances. Each master can have its own replicas, so the cluster keeps working when a master fails.

Clients can talk to any node. If a node doesn't own the slot for a key, it replies with MOVED (or ASK during a slot migration) telling the client where to go; cluster-aware clients cache the slot map and route directly after that. The catch is multi-key operations. A command that touches several keys (an MGET, a set intersection, a transaction) only works if all those keys live in the same slot, because no single node holds the whole keyspace.

Most Redis questions are really questions about one of a few patterns. The most common is the cache-aside (lazy-loading) cache: the application checks Redis first, and on a miss it reads the database, writes the value back into Redis with a TTL, and returns it. Subsequent reads hit the cache until the TTL expires or the key is evicted.

Rate limiting is the next-most common. A fixed-window limiter is just a counter per user per window: increment on each request and reject once the count crosses the limit, letting the key expire at the end of the window. The subtlety is that the counter and its expiry must be set together, or a crash between the two can leave a counter that never expires. Doing it in a Lua script makes the whole thing atomic on Redis's single thread.

Distributed locks are the pattern people get wrong most often. The basic idea is sound: SET lock:order:99 <random-token> NX PX 30000 acquires a lock only if it doesn't already exist (NX) and auto-expires after 30 seconds (PX) so a crashed holder can't hold it forever. Releasing the lock must be conditional: you only delete it if the token is still yours, or you risk deleting a lock that another client has since acquired. That check-and-delete has to be atomic, which again means a Lua script.

Two more worth naming. Leaderboards and any ranked or time-ordered index are sorted sets, as shown earlier. For messaging, Redis offers two very different tools: Pub/Sub is fire-and-forget (a message is delivered only to clients connected at that instant and is never stored), while Streams are an append-only, persistent log with consumer groups and acknowledgments, much closer to a real message queue. If you need messages to survive a disconnect or to be processed exactly once per group, use Streams, not Pub/Sub.

Redis fits when you need very low latency on a working set that fits in memory, and when the data is either rebuildable (a cache) or where a small, bounded loss on failover is acceptable. Caching, rate limiting, leaderboards, session storage, ephemeral counters, real-time queues, and presence (who is online) are all natural fits. It's the wrong tool when the dataset is far larger than affordable RAM, when you need rich queries and joins (use a relational database), when you need strong durability and strict consistency for a system of record (use a database designed for it), or when you need full-text search or analytical scans. Redis usually sits alongside a primary database rather than replacing it.

Expect a handful of recurring questions. "Why is Redis fast?" (in-memory data, single-threaded so no lock contention, simple operations). "Redis versus Memcached?" (Redis has rich data structures, persistence, replication, and clustering; Memcached is a simpler multi-threaded cache, occasionally better for plain large-value caching). "How does Redis persist data?" (RDB snapshots and the AOF log, with the everysec fsync trade-off). "What happens when it runs out of memory?" (the maxmemory-policy decides; the default noeviction rejects writes). "How do you scale Redis?" (replicas for reads, Redis Cluster with 16384 hash slots for writes). "How would you build a rate limiter / leaderboard / distributed lock?" (the patterns above, and the lock-correctness caveat is the part that impresses). "Pipelining versus transactions?" (pipelining batches commands to cut network round trips and isn't atomic; MULTI/EXEC runs a batch as one uninterrupted unit). "Do Redis transactions roll back on an error?" (no, the remaining queued commands still run and EXEC reports the failure; you get isolation, not rollback). And "What are the consistency guarantees?" (asynchronous replication, so failover can lose recent writes; not a strongly consistent store).

Redis is fast because it serves everything from memory and executes commands one at a time on a single thread, which removes lock contention and makes every command atomic. It's more than a cache because its values are typed data structures (strings, hashes, lists, sets, sorted sets, streams) with rich server-side operations, which is what makes leaderboards, rate limiters, and queues a few commands rather than an application subsystem. It survives restarts through RDB snapshots and the AOF log, trading data-loss window against throughput via the fsync policy. When memory fills, the maxmemory-policy decides what to drop, and the default noeviction is the wrong choice for a cache. It scales reads with asynchronous replicas and scales writes with Redis Cluster's 16384 hash slots, at the cost of weaker consistency on failover and constraints on multi-key operations.