Most people can name a dozen apps they use every day that depend on Redis. Almost nobody can tell you what it actually is. Daniel wants to fix that. Here's what he sent us.
He writes, "Yesterday we touched on backend optimization and caching, but one technology seems to come up over and over again whenever people discuss high-performance web applications: Redis. I'd love a deep dive from the ground up. What problem was it originally created to solve? Who created it, and what was happening in web development at the time that allowed it to become so influential? How did it go from being one of many open-source infrastructure projects to becoming almost the default answer whenever people mention caching?"
"More fundamentally, what actually is Redis? People often describe it as an in-memory key-value store, but that doesn't really explain why it's so useful. How does it differ from a traditional relational database like PostgreSQL or MySQL, and from document databases like MongoDB? Why is RAM such an advantage, and what trade-offs come with storing data primarily in memory?"
Then he wants the practical side. Where does Redis fit in a modern application architecture? At what point does a project genuinely benefit from introducing it? Is it something you only need at scale, or do latency problems show up earlier than people realize? He asks us to walk through the most common real-world use cases beyond simple page caching — session storage, API response caching, rate limiting, queues, pub/sub messaging, distributed locks, leaderboards, counters, and any other patterns that have made Redis a foundational building block. For each, he wants to know why Redis is especially well suited compared with using a conventional database.
And finally, the downsides. When is Redis unnecessary complexity? What applications are perfectly well served by a database alone? What operational costs, consistency concerns, or failure modes should developers understand before adding Redis to their stack? If someone is building software today, how should they decide whether Redis belongs in their architecture, and what are the common misconceptions people have about it?
That's... a lot. But it's the right set of questions.
It really is. And the origin story is the best place to start because it answers the "why does this exist at all" question before we get into data structures and use cases. So. Two thousand nine. Salvatore Sanfilippo — everyone calls him antirez — is running an Italian startup building a real-time web analytics product. Visitors, page views, all that streaming in constantly. His database is getting hammered with writes. Traditional disk-based storage just cannot keep up with the volume of incoming data, and he needs sub-millisecond responses to make the dashboards feel live.
The classic "my database is on fire" origin story.
The best kind. And here's the key detail — he doesn't set out to build a database. He builds a solution to one specific problem. He writes a single C file that holds data in memory and lets you push items onto lists and retrieve them fast. That's it. He calls it Redis — REmote DIctionary Server. And it solves his analytics problem so well that he releases it as open source, and within a few years it's everywhere.
What was happening in web development at that exact moment that made it catch fire?
The late two thousands were the Web two point oh era. Everything was becoming dynamic and interactive. Twitter was blowing up, GitHub launched in two thousand eight, Stack Overflow in two thousand eight. Pages weren't static HTML anymore — they were applications. And every time you add interactivity, you add latency. You're querying databases on every request, rendering templates, computing results. Disk-based databases that were fine for a content management system showing the same article to everyone suddenly became the bottleneck when you needed to serve personalized feeds to millions of users in real time.
So the demand for speed created the market, and antirez had already built the thing.
And he built something that was fundamentally different from what people thought of as a cache. Memcached existed, right? It was a simple key-value store — put a blob of bytes in, get a blob of bytes out. Very fast, very simple. But Redis wasn't just a dumb cache. It understood data structures. You could store a list, and push to it, and pop from it, and trim it, all with atomic operations. You could store a set and do intersections and unions. You could store a hash and update individual fields without replacing the whole thing. That's the part the "in-memory key-value store" label misses entirely.
Let's sit on that distinction. What does it actually mean that Redis understands data structures, versus something like memcached that just stores blobs?
Okay, imagine you're building a leaderboard for a game. With memcached, you'd store the entire leaderboard as one serialized blob — maybe JSON — and every time someone's score changes, your application has to read the whole blob, deserialize it, find the right entry, update it, sort the list, re-serialize the whole thing, and write it back. That's a read-modify-write cycle with no atomicity guarantee. Two players updating simultaneously can clobber each other.
The classic race condition.
Right. With Redis, the leaderboard is a sorted set. You call ZADD with the player's ID and their score. That's it. One atomic operation. Redis updates the data structure in place, maintains the sort order, and returns in microseconds. To get the top ten players, you call ZREVRANGE. It's O of log N for both operations. No serialization, no parsing, no race condition. The data structure lives in Redis's memory, and the commands operate on it directly.
So the intelligence lives on the server side, not in your application code.
That's exactly the right way to think about it. Redis is a data structure server. It's closer to a programming language's standard library than to a database. It gives you native implementations of strings, hashes, lists, sets, sorted sets, bitmaps, hyperloglogs, streams — each with a set of atomic operations that would be complex and error-prone to implement yourself on top of a generic key-value store.
How does that compare to something like PostgreSQL?
Radically different model. PostgreSQL stores data on disk in pages. When you run a query, it parses your SQL, builds a query plan, decides which indexes to use, reads pages from disk into memory, filters rows, performs joins, sorts results, and returns them. It gives you ACID transactions, complex joins, foreign keys, constraints, triggers. It's a general-purpose relational database designed to be the source of truth for your data.
And Redis just... holds things in RAM and hands them back.
With no query planner, no SQL, no joins, no transactions in the ACID sense. Redis commands are simple verbs on data structures — GET, SET, LPUSH, SADD, ZADD, INCR. Each command is implemented as a tight C function that operates directly on in-memory data. There's no parsing beyond the command itself, no planning, no disk I/O. The trade-off is you lose everything PostgreSQL gives you — relationships, constraints, durability guarantees, the ability to ask arbitrary questions with SQL. But you gain raw speed. A Redis GET takes microseconds. A PostgreSQL query, even a simple one, is measured in milliseconds at best.
And MongoDB?
MongoDB sits somewhere between. It's a document database that persists to disk, supports rich queries, indexing, aggregation pipelines, and some transactional guarantees. Documents are stored as BSON, which is a binary JSON format. You can query nested fields, do range queries, run aggregations that would be impossible in Redis. But MongoDB still touches disk. Even with its in-memory storage engine, there's a persistence layer and a query planner. Redis just blows past it on raw latency for simple operations.
So the hierarchy is: PostgreSQL gives you everything but it's slowest, MongoDB trims some relational features for more flexibility and speed, Redis trims almost everything for maximum speed.
That's a fair simplification. Now, the "why RAM" question. Memory access is on the order of a hundred nanoseconds. An SSD read is on the order of a hundred microseconds. That's a thousand-to-one difference. A spinning disk seek is ten milliseconds — a hundred thousand times slower than RAM. If you're building a feature that needs to respond in under a millisecond, you simply cannot touch disk. It's not an optimization problem, it's a physics problem.
But RAM is expensive and finite.
And volatile. That's the triad of trade-offs. Cost — a terabyte of RAM costs dramatically more than a terabyte of SSD. Capacity — you're limited by how much RAM you can physically put in a machine, or how much you can afford across a cluster. And volatility — if the power goes out, everything in RAM is gone.
Unless you configure persistence.
Right. Redis offers two persistence mechanisms. RDB snapshots — periodically write the entire dataset to disk as a point-in-time dump. And AOF, the Append-Only File — log every write command as it happens, so you can replay them on restart. You can use both together. But here's the thing — persistence adds overhead. An RDB snapshot can cause latency spikes while it's being written. AOF can grow large and slow down restarts. And even with both enabled, you can still lose data from the window between the last write and the last sync to disk.
Which is why most people use Redis as a cache where losing data is annoying but not catastrophic.
The source of truth lives in PostgreSQL or MySQL. Redis holds the hot copy. If Redis restarts and the cache is empty, your application falls back to the database, performance degrades for a bit, and the cache repopulates. That's an acceptable failure pattern for most applications. It's not acceptable for your bank balance.
So the mental model is: Redis is a performance layer, not a storage layer.
That's the right default. There are people who use Redis as a primary database with AOF and replication and all the safety nets — and you can do that — but it's not what it was designed for, and you're fighting its nature.
Let's talk about how it became the default. You mentioned Twitter, GitHub, Stack Overflow adopting it early.
The network effect was real. GitHub used Redis for job queues and caching starting around two thousand ten. Stack Overflow used it for caching and real-time stats. Twitter used it extensively. When the most respected engineering organizations in the world all adopt the same tool and blog about it, everyone else follows. But the deeper reason is that Redis is easy to deploy. It's a single binary. You download it, run it, and it works. The commands are intuitive. The documentation is excellent. There's no schema to define, no migrations to run. A developer can go from zero to a working cache in ten minutes.
Compare that to setting up a PostgreSQL cluster.
Night and day. And Redis had client libraries for every language almost immediately. So when a team hit a performance bottleneck — say their database was getting crushed by session lookups on every request — the path of least resistance was "apt-get install redis" and change three lines of code to point session storage at it. The operational cost of adding Redis was lower than the engineering cost of optimizing the database queries.
Which brings us to Daniel's question about when a project benefits from Redis. He suspects it's a scale thing — you add it once you've outgrown your database.
And he's mostly right, but with a nuance. It's not about scale in the sense of millions of users. It's about specific latency requirements that appear much earlier than people realize. A small SaaS app with a hundred users might have an admin dashboard that runs a complex report query. That query takes three seconds. The admin hits refresh a few times, no big deal. But if that same query powers a customer-facing feature that gets called on every page load, suddenly three seconds is unacceptable even with ten users.
So the trigger isn't user count, it's whether a piece of data is hot — accessed repeatedly — and latency-sensitive.
That's the framework. You add Redis when you have hot data that's expensive to compute or retrieve, and the latency matters to the user experience. The classic example is session storage. On every single HTTP request, your application needs to look up the user's session — who they are, what's in their cart, their preferences. If that lookup hits your PostgreSQL database, you're adding a disk seek and a query to every request. With Redis, it's a GET command that returns in microseconds. Even for a modest application with a few hundred concurrent users, that difference is visible.
And you get expiration for free.
Built-in TTL. You set the key to expire after an hour, or a day, or whatever your session timeout is. Redis automatically cleans it up. No cron job, no cleanup script, no orphaned session rows accumulating in your database.
Let's run through the other use cases Daniel listed. API response caching.
This is probably the most common pattern after sessions. Your application calls an external API — a weather service, a payment processor, a geocoding service. That API is slow, or rate-limited, or costs money per call. You cache the response in Redis with a TTL. Next time someone asks for the weather in London, you check Redis first. If it's there, you return it in microseconds without touching the external API. If it's expired or missing, you make the call and populate the cache. The pattern is called cache-aside, and it's the bread and butter of Redis usage.
Rate limiting.
Beautifully simple in Redis. For each user or API key, you maintain a counter with INCR. The first request in a window sets the key with a TTL of, say, sixty seconds. Each subsequent request increments it. If the counter exceeds your limit — a hundred requests per minute, whatever — you reject the request. The atomicity of INCR means you don't have race conditions. Two requests arriving simultaneously both get the correct count. Doing this in PostgreSQL requires a transaction with a SELECT FOR UPDATE or an upsert, and it's slower and more complex.
Queues.
Redis lists support blocking pop operations. You LPUSH work items onto a list, and workers call BRPOP to block until an item is available. It's a simple, reliable task queue that requires zero configuration. It's not Kafka — there's no durability guarantee, no message replay, no partitioning. If a worker crashes after popping an item but before processing it, that work is lost unless you build a reliability layer on top. But for many use cases — sending a welcome email, generating a thumbnail — that's acceptable.
Pub/sub.
Redis channels. You publish a message to a channel, and every subscriber receives it. It's fire-and-forget — there's no persistence, no delivery guarantee, no acknowledgement. If a subscriber is disconnected when a message is published, it misses it. This is great for real-time notifications where occasional message loss is tolerable — updating a dashboard, notifying connected clients that new data is available. It's not a message broker. If you need guaranteed delivery, you use Kafka or RabbitMQ.
Distributed locks.
The pattern is SET NX EX — set the key only if it doesn't exist, with an expiration. If you acquire the lock, you do your work and then delete the key. If you crash, the lock expires automatically so you don't deadlock the system. There's a whole debate around this — Martin Kleppmann wrote a famous critique arguing that Redis locks can't guarantee mutual exclusion in the presence of network partitions, and antirez responded with the Redlock algorithm. The short version is: Redis locks are fine for efficiency purposes — preventing duplicate work — but you should not use them for correctness purposes where a split-brain scenario would corrupt data.
That's an important distinction. Preventing two servers from sending the same email twice versus preventing two servers from processing the same financial transaction.
And the distributed lock use case is where people most often get themselves in trouble by not understanding the failure pattern.
Leaderboards and counters.
We touched on leaderboards with sorted sets. Counters are even simpler — INCR on a key gives you an atomic increment that returns the new value. Page view counters, like counts, vote tallies. All trivial in Redis. All would require a read-modify-write cycle with locking in a relational database.
So across all these use cases, the pattern is the same. Redis gives you atomic operations on in-memory data structures that would require complex, slow, or race-condition-prone code if you built them on top of a disk-based database.
That's the unifying thread. Now, the flip side. When is Redis unnecessary complexity?
This is the part people skip.
Because it's less fun. Here's the honest answer: most applications don't need Redis. If you're building a blog, or a small e-commerce site, or an internal tool with a dozen users, your database can handle the load just fine. PostgreSQL has its own query cache. Modern SSDs are fast. Adding Redis means adding a new service to monitor, back up, secure, keep patched, and debug when something goes wrong. That operational overhead is real, and it compounds with every service you add to your stack.
The "just use PostgreSQL" school of architecture.
Which I have a lot of sympathy for. PostgreSQL can do caching, job queues, pub/sub, and rate limiting. It won't be as fast as Redis for those things, but if you don't need sub-millisecond latency, you don't need Redis. The right approach is to start with a database alone, measure your actual performance, and introduce Redis only when you've identified a specific bottleneck that it solves.
Daniel asked about failure pattern.
Several classics. First, the single-threaded nature of Redis. Commands are executed one at a time in the main event loop. A slow command — KEYS on a large dataset, or a big SORT operation — blocks everything else. Your entire Redis instance freezes for the duration. Every application server waiting on a Redis command times out. In production, you never run KEYS. You use SCAN instead, which iterates incrementally without blocking.
What's the worst-case scenario there?
A developer runs KEYS star in production to debug something. The dataset has a few million keys. Redis is blocked for seconds. Every web request that needs a session lookup or a cached value fails or times out. The application falls back to the database, which gets hit with a thundering herd of requests all at once because the cache just disappeared. Database overloads. Site goes down. I've seen this happen.
From one command.
Another failure pattern is running Redis without a memory limit. Redis will happily consume all available RAM. When the OS runs out of memory, the OOM killer shows up, and it doesn't always kill Redis — sometimes it kills your application process instead. You should always configure maxmemory and an eviction policy.
And the persistence gotchas.
If you don't configure persistence, a restart wipes everything. The cache is cold, every request misses, and your database gets hammered until the cache warms up. If you do configure persistence, RDB snapshots can cause latency spikes because Redis forks the process to write the snapshot — the fork itself is fast, but the copy-on-write overhead can slow things down on a busy instance. AOF can grow enormous if you don't configure rewrite rules. And AOF fsync policies are a trade-off between durability and performance — every-second fsync loses at most one second of data, which is fine for a cache but might not be fine if you're using Redis as a queue.
So the operational knowledge required is non-trivial.
It's not just "apt-get install redis-server" and walk away. You need monitoring, alerting, backup strategy, failover configuration, memory management, and a plan for what happens when it goes down. None of this is insurmountable, but it's real work, and it's why "just add Redis" should be a deliberate decision, not a reflex.
Let's talk about the consistency model. Cache invalidation is famously one of the hard problems in computer science.
And Redis doesn't solve it for you. If you cache a database row in Redis, and then the row is updated in PostgreSQL, your cache is now stale. You have to decide how to handle that. You can set a short TTL and accept eventual consistency. You can explicitly invalidate the cache key in your application code when you update the row. You can use a write-through pattern where every write goes to both PostgreSQL and Redis. Each approach has trade-offs, and getting it wrong means serving stale data to users.
Which in some contexts is fine — a slightly outdated like count isn't the end of the world. In others, it's catastrophic.
And that's the judgment call. The other consistency trap is treating Redis as a source of truth. I've seen startups store user data exclusively in Redis because it was fast and easy, and then lose everything when the instance restarted. Redis is not a database in the ACID sense. It doesn't give you transactions across multiple keys. It doesn't give you referential integrity. It doesn't give you point-in-time recovery unless you've architected it carefully. Use it as a cache, a queue, a rate limiter, a leaderboard — but your canonical data should live somewhere durable.
So if Daniel's building something today, how should he decide?
Decision framework. Step one: build with a database alone. PostgreSQL, MySQL, whatever. Step two: identify your actual performance bottlenecks. Not hypothetical ones — real, measured latency problems. Step three: ask whether the bottleneck is a hot data problem. Is the same data being read repeatedly? Is it expensive to compute? Does the latency matter to users? If yes to all three, Redis is a candidate. Step four: ask whether you can solve it within the database — better indexing, query optimization, read replicas, built-in caching. Step five: if the database can't get you below your latency target, add Redis for that specific use case.
Not "add Redis because we might need it later."
Premature optimization's favorite infrastructure project. I've seen teams spend weeks setting up Redis, configuring sentinel for high availability, writing cache invalidation logic, monitoring — all before they had a single user. Meanwhile the PostgreSQL instance they already had could have handled their traffic for years.
The misconception roundup. What do people get wrong?
Biggest one: Redis is just a cache. We've spent this whole episode explaining why that's wrong, but it persists. Redis is a data structure server. The "just a cache" framing makes people miss the queues, the locks, the leaderboards, the pub/sub, the atomic counters. It's like calling a Swiss Army knife "just a blade."
Second misconception: Redis replaces your database.
It doesn't. It complements it. Different tools for different jobs. Redis doesn't speak SQL, doesn't do joins, doesn't guarantee durability by default. If you try to use it as your primary data store without understanding the trade-offs, you will have a bad time.
Third: Redis is only for big companies.
The opposite, in a way. Redis is so lightweight that it's useful for small projects with specific latency needs. A personal project with a real-time leaderboard can benefit from Redis on a five-dollar VPS. The barrier isn't scale, it's whether you have a problem that matches Redis's strengths.
And fourth: Redis is always fast.
Single-threaded. One slow command blocks everything. Persistence adds latency. Network round trips add up. It's fast in the common case, but "always" is a dangerous word in systems engineering.
Where is all this heading? Redis seven introduced functions and improved ACLs. There are forks and alternatives — KeyDB, Dragonfly. The landscape is shifting.
The interesting question is whether the line between cache and database continues to blur. We're seeing memory-optimized hardware become cheaper. We're seeing databases like PostgreSQL get better at in-memory caching. We're seeing Redis add more durability features. At some point, the distinction between "in-memory cache" and "database with a huge buffer pool" might become academic for a lot of use cases.
The forks. KeyDB is a multithreaded fork of Redis that emerged after antirez stepped back from the project. Dragonfly is a from-scratch reimplementation claiming twenty-five times the throughput.
Which is fascinating because it challenges the single-threaded assumption that's been core to Redis's design philosophy since day one. Whether those alternatives gain enough traction to unseat Redis as the default is an open question. Redis has the ecosystem — the client libraries, the operational knowledge, the Stack Overflow answers, the muscle memory. That's hard to displace.
The single C file that ate the world, and then got forked by its own father.
You said that once and I still think about it. antirez stepped away in twenty twenty, and the project is now maintained by the company Redis Ltd. The license changed, there was community drama, the forks emerged. It's a whole second act to the story.
The most common misconception about Redis is that it's just a fast cache. It's a data structure server — the speed comes from RAM, but the utility comes from the atomic operations on native data types that would be complex and error-prone to build yourself.
The correction: if you treat it as just a cache, you miss the queues, the locks, the counters, and everything else that makes it a foundational building block. But if you treat it as a database replacement, you'll lose data. Know what it is, and know what it isn't.
Thanks to our producer Hilbert Flumingtop for making this episode happen.
This has been My Weird Prompts. We're at my weird prompts dot com — send us your questions, your war stories, your Redis disaster anecdotes. We read everything.
We'll be back soon.
