So here's what Daniel wrote us. Quote — Yesterday we touched on backend optimization and caching, but one technology seems to come up over and over again whenever people discuss high-performance web applications: Redis. I'd love a deep dive from the ground up. What problem was it originally created to solve? Who created it, and what was happening in web development at the time that allowed it to become so influential? How did it go from being one of many open-source infrastructure projects to becoming almost the default answer whenever people mention caching?
Actually. The origin story is one of those rare moments where a single person just... solved the exact right problem at the exact right time.
He's not done. There's more.
Oh, keep going.
More fundamentally, what actually is Redis? People often describe it as an in-memory key-value store, but that doesn't really explain why it's so useful. How does it differ from a traditional relational database like PostgreSQL, and from document databases like MongoDB? Why is RAM such an advantage, and what trade-offs come with storing data primarily in memory? I'd also like to understand where Redis fits into a modern application architecture. At what point does a project genuinely benefit from introducing Redis? My impression has always been that it's something you weave into a stack once you've reached a certain level of scale or traffic, rather than something every small application needs from day one. Is that impression accurate, or is Redis just as useful for modest projects because it solves latency problems that appear much earlier than people realize?
He's asking the right question there. The scale threshold thing is widely misunderstood.
He goes on. Please walk through the most common real-world use cases beyond simple page caching. Talk about session storage, API response caching, rate limiting, queues, pub-sub messaging, distributed locks, leaderboards, counters, and any other patterns that have made Redis such a foundational building block. For each, explain why Redis is especially well suited to the task compared with using a conventional database. Finally, I'd like to hear about the downsides. When is Redis unnecessary complexity? What kinds of applications are perfectly well served by a database alone? What operational costs, consistency concerns, or failure modes should developers understand before adding Redis to their stack? If someone is building software today, how should they decide whether Redis belongs in their architecture, and what are the common misconceptions people have about it? End quote.
That's a whole syllabus. I love it.
It's a lot. And I think the best place to start is the origin story, because it answers about half of the "why is it like this" questions before we even get to the technical bits.
Agreed. So — Salvatore Sanfilippo. Goes by antirez online. He's a Sicilian programmer, and back in two thousand nine he was building a real-time web analytics tool called LLOOGG. The problem he kept hitting was that his MySQL database couldn't keep up with the write load for pageview counters. He needed something that could increment numbers in memory absurdly fast and persist them somehow. Nothing off the shelf did exactly what he wanted, so he wrote a single C file that held key-value pairs in RAM and flushed to disk periodically.
A single C file.
A single C file. He posted it on Hacker News, and people immediately saw it wasn't just a cache. It had data structures — lists, sets, sorted sets. That's the part that's hard to overstate. Memcached already existed and was widely used for caching, but Memcached gives you a flat key-value space. You store a blob, you retrieve a blob. Redis gave you data types you could operate on server-side. You could push to a list, intersect sets, get the rank of an element in a sorted set — all atomically, all in memory.
So the "data structure server" label he used early on was literal. It wasn't marketing.
Completely literal. And that timing — two thousand nine, two thousand ten — the LAMP stack was everywhere, MySQL was the default, and web apps were getting more interactive. People were building Twitter clones, real-time dashboards, chat systems. Every one of those needed fast counters, leaderboards, queues. Doing those in MySQL meant table locks and slow count queries. Redis made them trivial.
And the name?
Remote Dictionary Server. R-E-D-I-S. He's said in interviews he just wanted something short and pronounceable.
So we've got this Sicilian developer, a real-time analytics problem, a single C file, and a name that's essentially an acronym. And from that, it becomes... what, the most loved database on Stack Overflow's developer survey for something like five years running?
Yeah, it topped that survey for years. And it's not a database in the traditional sense. That's the distinction Daniel's asking about. PostgreSQL and MySQL are disk-first. Their entire architecture assumes data lives on disk, and memory is a cache layer on top of that. Every write involves writing to a write-ahead log, updating indexes on disk, managing pages. Reads might hit memory if you're lucky, but the system is always thinking about durability first.
And Redis flips that.
Completely. Redis is memory-first. Data lives in RAM. Persistence is the afterthought — something you bolt on via snapshots or an append-only file. The core engine is an event loop that processes commands one at a time, single-threaded, which means you never have race conditions within a command. Everything is atomic by construction.
Wait, single-threaded? That sounds like a bottleneck, not a feature.
It sounds like one, but it's the opposite. Because everything is in RAM, individual operations take microseconds. A single thread can handle a hundred thousand operations per second without breaking a sweat. And because there's no locking overhead, no context switching, no contention, you get predictable latency. The bottleneck in a Redis instance is almost always network bandwidth, not CPU.
So the simplicity of the execution model is what makes it fast, not despite being simple.
Compare that to PostgreSQL processing a simple key lookup. Even if the data is in the buffer cache, you've still got to parse SQL, plan the query, check permissions, navigate a B-tree index, handle MVCC visibility rules. Each step is fast, but they add up. Redis just hashes the key and reads the pointer. That's it.
So the RAM advantage isn't just "RAM is faster than disk." It's that the whole architecture assumes RAM.
Right. And that leads straight into the trade-off Daniel asked about. The obvious one is capacity. RAM is expensive. A terabyte of RAM costs dramatically more than a terabyte of SSD. So Redis is for hot data — the stuff you need right now. You're not storing your entire product catalog in Redis unless your catalog is small or your budget is huge.
And the second trade-off is durability.
Yes. If the power fails, anything written since the last snapshot is gone, unless you've configured the append-only file with synchronous flushing — which slows things down. Redis gives you knobs for this. You can prioritize speed and accept some data loss, or prioritize durability and accept lower throughput. But it's never going to give you the same durability guarantees as PostgreSQL with synchronous replication.
Which means the mental model isn't "Redis instead of Postgres." It's "Redis in front of Postgres."
That's the canonical pattern. Redis as an acceleration layer. Your source of truth is still the relational database. Redis holds derived data, cached data, ephemeral data. If Redis loses everything, the application might slow down or drop some non-critical state, but it shouldn't corrupt your business data.
Okay, so let's get into the use cases Daniel listed. He mentioned session storage, and I think that's actually the one most people encounter first without realizing it.
If you've ever logged into a web app, your session is probably in Redis. The alternative is storing sessions in the database, which means every single page request triggers a session lookup query. On a busy site, that's thousands of queries per second doing nothing but "is this cookie still valid?" Redis handles that with near-zero latency, and it can set TTLs on keys so sessions auto-expire. You don't need a cron job to clean up old sessions — they just vanish.
And that TTL mechanism is built in. It's not a workaround.
It's a first-class feature. Every key can have an expiry. That alone eliminates an entire class of application code.
What about API response caching? That feels related but different.
It is different, because here you're caching the output of expensive operations. Say you have an endpoint that aggregates sales data across five tables with joins and group-bys. That query might take two seconds. You cache the JSON response in Redis with a key like "sales-report-q3" and a TTL of five minutes. The first user waits two seconds. Every user for the next five minutes gets it in under a millisecond.
And the database gets to breathe.
The database loves you for it. The thing people get wrong, though, is cache invalidation. If the underlying data changes, you need to know when to bust that cache. Redis doesn't solve that for you. It gives you the tools — you can delete keys programmatically, use pattern matching, set short TTLs — but the logic of when to invalidate is still your problem.
The famous Phil Karlton quote. There are two hard problems in computer science: naming things, cache invalidation, and off-by-one errors.
And Redis handles the caching part beautifully. It's the invalidation strategy that separates the adults from the children.
Rate limiting. That one seems like such a natural fit.
It's practically the platonic ideal of a Redis use case. You need to track how many requests a user has made in a sliding window, and you need to do it for every request without adding latency. In Redis, you create a key like "rate-limit-user-1234," increment it, and set a TTL of sixty seconds on the first increment. If the count exceeds your threshold, you reject the request. That's two Redis commands — INCR and EXPIRE — and they're atomic. Doing this in PostgreSQL would mean an upsert with a timestamp comparison on every single API call. It would work at low volume, but at high volume it's a lot of write pressure for something that's fundamentally ephemeral.
And ephemeral is the word. If you lose your rate-limit counters in a crash, nobody cares. The worst case is someone gets a few extra requests in before the counters rebuild. That's a perfectly acceptable trade-off.
Right. Which is why Redis as a rate limiter is basically free operational complexity. The data doesn't matter, so the durability concern evaporates.
Queues. This one surprised me when I first learned about it. Redis as a message broker?
It's not a full-featured message broker like RabbitMQ, but for simple job queues it's remarkably good. The LIST data type gives you LPUSH and RPOP — push to the left, pop from the right — and that's a FIFO queue right there. Add BRPOP for blocking pop, and your workers can sit idle waiting for work without polling. Libraries like Sidekiq for Ruby and Celery for Python can use Redis as their backend, and millions of applications do exactly that.
What's the ceiling on that? At what point does a Redis queue become the wrong tool?
When you need advanced routing, message acknowledgments with redelivery guarantees, or when you can't afford to lose a message if Redis crashes. Redis streams, added a few years ago, address some of this — they add consumer groups and message persistence — but if you're building a financial transaction pipeline, you probably want something with stronger delivery guarantees.
So it's the eighty percent solution for queues. Good enough for most things, not for the hard cases.
That's a fair characterization.
Pub-sub. This feels like a different beast entirely.
It is. Pub-sub in Redis is fire-and-forget. You publish a message to a channel, and every subscriber currently connected gets it. There's no persistence, no queueing, no history. If nobody's listening, the message disappears into the void. This is great for things like notifying connected clients of real-time updates — a WebSocket server broadcasting "new comment on post seven" to everyone viewing that post. It's not for reliable message delivery.
So it's a signaling mechanism, not a message store.
And it's astonishingly fast at that. The pub-sub throughput in Redis is basically line rate — you're limited by how fast you can push bytes over the network.
Distributed locks. This one feels like it could go wrong in subtle ways.
Oh, it absolutely can. The basic pattern is SET NX — set the key only if it doesn't exist — with a TTL, so the lock auto-releases if the holder crashes. But there are edge cases. What if the lock expires while the holder is still doing the work? What if a network partition makes two clients think they both hold the lock? There's a whole literature on this. The Redlock algorithm, which antirez himself proposed, has been debated extensively. Martin Kleppmann wrote a detailed critique, and the back-and-forth is worth reading if you're implementing distributed locking in production.
So Redis gives you the primitive, but the correctness is on you.
Always. Redis provides atomic operations. Building correct distributed systems on top of them is a separate skill.
This one feels magical when you first see it.
Sorted sets. They're the killer feature that nothing else had at the time. You store each player's score as the score in a sorted set, and the player ID as the member. ZADD adds or updates a score. ZRANGE gives you the top hundred players, in order, instantly. ZRANK tells you where a specific player stands. All of this is logarithmic time complexity — O log N — and it's all in memory, so it's fast even with millions of entries.
Doing that in SQL is possible but painful.
SELECT with ORDER BY and LIMIT works until you have a million rows and you're running it on every page load. Then you start adding materialized views and cache tables and suddenly you've built half of Redis inside PostgreSQL.
Which is the broader pattern. People reinvent Redis features inside their relational database, poorly.
All the time. Counters are the same story. INCR in Redis is a single atomic operation. Doing an atomic increment in PostgreSQL requires SELECT FOR UPDATE and an UPDATE inside a transaction, which means row locks and potential contention. For a pageview counter on a high-traffic site, that's a disaster. Redis just... increments a number.
So to Daniel's question about scale — when does a project benefit? You're saying it's not purely a traffic threshold.
It's not. I think the better heuristic is: when you need sub-millisecond latency for specific operations, or when you're doing something that's a bad fit for the relational model. A small app with ten users doesn't need Redis for page caching. But a small app with ten users that needs a real-time leaderboard for a game? Redis from day one.
Or a small app that needs rate limiting on an API that third parties are hitting.
The scale that matters isn't total users — it's the shape of the problem. If you're doing atomic counters, sorted rankings, TTL-based expiry, or pub-sub, Redis is the right tool even at low volume. If you're just serving CRUD pages with session data, your database can handle that until the query volume actually hurts.
So the misconception is "Redis is for big scale." The reality is "Redis is for specific data shapes and access patterns."
That's it. And the other misconception is that adding Redis means you're doing "real" architecture. I've seen projects with a hundred users running Redis, a message queue, and three microservices because someone read a blog post about scaling. That's architectural cosplay.
You've used that phrase before.
I stand by it. Redis is a solution to a problem. If you don't have the problem, you don't need the solution.
Let's talk about the downsides Daniel asked about. Operational costs first.
Running Redis in production means you're running another piece of infrastructure. It needs monitoring, memory provisioning, backup configuration, failover setup if you want high availability. Redis Sentinel or Redis Cluster add complexity. Managed services like Elasticache or Redis Cloud abstract some of this, but they cost money. If your database alone is handling everything fine, adding Redis is just adding cost and failure pattern.
And the memory cost is real.
Very real. A managed Redis instance with enough memory to hold your working set might cost more per month than your entire database. You have to actually calculate whether the latency improvement is worth it. Sometimes the answer is no — just optimize your queries and add database indexes.
Consistency concerns. You mentioned these earlier, but spell them out.
The big one is the cache coherence problem. Your database says one thing, Redis says another. Maybe the cache didn't get invalidated properly. Maybe the invalidation happened but a write hadn't committed yet. Maybe you're doing write-through caching and the Redis write succeeded but the database write failed. Every piece of data that exists in two places is a potential divergence.
And debugging that at two in the morning is nobody's idea of a good time.
It's the worst. You end up writing cache warming scripts, cache busting endpoints, monitoring for staleness. Each of those is code that doesn't exist if you just query the database directly.
What about the failure pattern where Redis goes down?
Depends how you've architected. If Redis is just a cache, your application should degrade gracefully — slower responses while it falls back to the database. If you've made Redis a hard dependency — say, your session store with no fallback — then Redis going down means users can't log in. That's an operational choice, not a Redis problem, but it's a choice a lot of teams make without thinking through the implications.
And memory exhaustion. What happens when Redis runs out of RAM?
It depends on the eviction policy you've configured. You can set it to reject writes, which is safe but breaks your application. You can set it to evict keys based on LRU — least recently used — which is usually what you want for a cache. Or you can set it to evict keys with TTLs first, or randomly. The dangerous configuration is no eviction policy with no monitoring, because then Redis just... stops accepting writes, and you find out when users start complaining.
So you need to know your data and set the policy accordingly.
You need to know your data, monitor your memory usage, and have alerts. None of this is free.
Let's talk about the licensing drama. That's part of the story of how Redis got to where it is.
Right. In twenty twenty-four, Redis Labs — the company behind Redis, which had changed its name from Redis Labs to just Redis — switched the core Redis license from BSD to a dual license model, the Redis Source Available License and the Server Side Public License. The community... did not take it well.
The understatement of the century.
The reaction was swift. Within days, the Linux Foundation announced Valkey — a fork of the last BSD-licensed version of Redis, maintained as an open-source project with major backing from AWS, Google Cloud, and Oracle. And then, in a twist that nobody expected, Salvatore Sanfilippo — antirez himself — came out of retirement in early twenty twenty-five to work on a different Redis fork called Redict.
Wait, the original creator came back specifically to fork his own project?
Yes. He'd stepped away from Redis development around twenty twenty. Spent years doing other things. And the license change brought him back, because he felt strongly that Redis should remain truly open source. He's now working on Redict, which is a community-driven fork focused on stability and the original BSD license principles.
The Redis landscape right now is... what, three major variants?
At least three. The official Redis, now source-available. Valkey, which has the corporate cloud backing and is probably the most active fork in terms of development velocity. And Redict, which is antirez's more conservative, stability-focused fork. Plus a handful of others. It's a fascinating moment — the technology is more widely used than ever, but the governance is fragmented.
For someone building software today, does that fragmentation matter?
At the protocol level, they're all compatible. Your application code that talks to Redis will talk to Valkey or Redict just fine. The differences are in licensing, governance, and which features get prioritized. For most users, the managed service they pick — Elasticache, Memorystore, Redis Cloud — will determine which flavor they get, and they may never notice the difference.
That's a reassuring answer I didn't expect.
The protocol is the moat. The wire protocol is simple, well-documented, and universally implemented. That's part of why Redis became a default — the client library ecosystem is enormous. Every language has a mature Redis client. That network effect is hard to displace.
Daniel's final question — how should someone decide whether Redis belongs in their architecture? What's the decision framework?
I'd say three questions. One: do you have data that needs sub-millisecond access and fits in memory? Two: do you need data structures or atomic operations that your database handles poorly — sorted sets, counters, pub-sub, TTLs? Three: can you tolerate losing some data if things go wrong? If the answer to all three is yes, Redis is probably the right call. If you're answering no to any of them, think harder.
The "just use your database" cases?
If your application is CRUD with no exotic access patterns, if your query volume is well within what your database handles comfortably, if you don't have real-time features, if your data set is large and needs complex joins — just use PostgreSQL. It's a remarkable piece of software. It will serve you well for a long time before you need to add anything else.
There's something almost reassuring about that. The boring stack works for most things.
The boring stack works for most things. Redis is for when the boring stack stops being boring in a specific, identifiable way. Not before.
The through-line here, from a single C file in Sicily to three competing forks and a protocol that underpins a huge chunk of the internet... it's the right abstraction at the right time. A data structure server that just happens to be fast because it lives in RAM.
The fact that antirez came back to defend the open-source version of his own creation — there's something almost poetic about that. The technology outgrew the license, and the creator stepped back in to say "no, this should belong to everyone."
A single C file that ate the world, and then got forked by its own father.
That's... actually a pretty good summary.
We should probably leave it there before I try to turn that into a movie pitch. Thanks to our producer Hilbert Flumingtop for keeping this show running. This has been My Weird Prompts. If you want to send us your own questions — and clearly Daniel has set the bar for thoroughness — email the show at show at my weird prompts dot com.
We'll be back soon.
