Most people never think about their kernel. It arrives, it runs, it updates itself in the background, and that's the end of the relationship.
And yet it's the one piece of the operating system you can swap out entirely without reinstalling anything else. Same Ubuntu, same files, different kernel underneath.
Which is what Daniel wants to dig into. He wrote in with a long one, so let me compress it. His starting point is that on something like an Ubuntu LTS, you're not actually locked to the kernel the distribution gave you. You can pull a newer upstream one, keep an older one, or go get a kernel somebody has modified for a particular kind of work. Most people run what they're given, and reasonably so, but there's a crowd that deliberately doesn't.
And he wants the whole map.
The whole map. What actually changes between kernel versions, how a distribution like Ubuntu decides what to ship, whether the alternative kernels are delivering real gains or just better hardware support, what Liquorix and XanMod are optimizing for, whether compiling for your specific CPU buys you anything, where desktop and server motivations split apart, and what the benchmarks actually say when you line them up.
That last one is the one I want to get to.
The question underneath all of it is whether the default kernel is already close to optimal, or whether there's a real opportunity sitting there that most people ignore.
Here's the thing that makes the question harder than it looks. The kernel isn't one thing. It's the scheduler, the memory manager, the I/O layer, the driver host, and the power manager all in one binary. Change the kernel and you change all five at once, in ways that interact.
So you can't tune one without moving the others.
And the landscape has layers to it. You've got distribution kernels, which is what almost everybody runs. You've got upstream mainline, straight from kernel.org. You've got the distribution's own variants, the low-latency one, the hardware enablement one. And then you've got the enthusiast kernels, Liquorix, XanMod, and the real-time kernel that used to be a patchset and now isn't.
Give me the shape of the episode.
First what changes between versions and how Ubuntu decides what to ship. Then why anyone would go elsewhere, what those specialized kernels optimize for, whether CPU-specific tuning pays off, how desktop and server differ, and what the benchmark numbers actually show. The numbers are where the marketing takes a hit.
Start at the top. What actually changes when the version number moves?
Upstream Linux doesn't have a release date. It has a release rhythm. Every two to three months, when it's ready. No fixed calendar. Ubuntu is the opposite, a hard six-month cadence no matter what state upstream is in.
Those two schedules don't line up.
That's the whole problem. Canonical's rule of thumb is that they need about a month of stabilization between an upstream release and a stable Ubuntu kernel. So when Ubuntu freezes for a release, whatever upstream kernel exists at that moment gets shipped, even if it's still a release candidate.
They ship release candidates to users.
Under the policy they adopted in 2024, yes. And they split it into categories. If the candidate is at release candidate four through six, that's a Tight release. If it's at one through three, that's Unstable. In the Unstable case they ship what they call a Bridge Kernel, a kernel from the previous release, to carry people until the new one settles down.
Naming a kernel "Unstable" and shipping it anyway is a choice.
It's honest labeling, at least. And this is where the version story gets interesting, because the GA kernel for a release is the latest upstream kernel at the time of that release, and then it stays on that version for the entire life of the release. Four years, in an LTS case, on one kernel version.
With security fixes going in.
That's the hardware enablement kernel's whole reason to exist. HWE backports a newer kernel to an older LTS, and it updates every six months until it matches the GA kernel of the next LTS.
So on an older LTS you get a choice. Stay on the original kernel, or follow the HWE stack and get newer hardware support.
And Corn, you made this point before, that the real story of LTS is the hardware enablement stack absorbing most of the reason people used to jump to interim releases.
I stand by it.
The variant list is worth hearing, because it shows how much curation is happening. Ubuntu ships a low-latency kernel for things like audio work, generic HWE, generic HWE edge, cloud kernels for AWS, Azure and GCP, platform kernels for Raspberry Pi and Snapdragon, and silicon-partner kernels built with the chip vendors.
And people think of the default as just an older upstream.
That's the misconception to kill. A distribution kernel isn't upstream frozen in amber. Canonical backports the CVE fixes itself. That's why an Ubuntu stable kernel is named with a four-part scheme ending in ABI, rather than kernel.org's three-part version. The version number itself tells you it's a different product.
Say that plainly. What do you give up if you go install the newest upstream kernel on your desktop?
The maintenance pipeline. You get the new scheduler behavior, the new driver support, the new everything. You also get every regression upstream hasn't fixed yet, and nobody backporting security patches to the version you're running. You've opted out of the curated product and into the raw one.
So the default kernel is the maintained one, and the newest one is the unmaintained one.
For that version, yes. Which is not an argument against ever doing it. It's an argument for knowing what you're doing, and for the people who do it, most of them are on a distribution where they can roll back.
What actually changes inside the kernel between versions, the things that would move performance?
A few big levers. The CPU scheduler. The memory manager, where multi-generational LRU changed how pages get reclaimed. The I/O schedulers. Power management. Driver support. The scheduler is the biggest one for anything interactive, and in kernel 6.6 it changed in a way that matters.
EEVDF.
EEVDF replaced the completely fair scheduler. It's a different way of deciding which process runs next, built around virtual deadlines instead of the older fairness model. On most workloads you wouldn't notice. On some you would.
So a kernel upgrade can change how your machine feels without touching a single other package.
It can. But it won't make your CPU faster. What it changes is how the CPU's time gets handed out, how memory gets reclaimed, how I/O gets queued. If none of those is your bottleneck, the upgrade does nothing you can measure.
Which sets up the question Daniel actually cares about. Why would somebody go get a different kernel on purpose?
Two reasons, and the balance between them is the interesting part. Hardware support is the boring one. New laptop, new WiFi chip, new GPU, the drivers landed upstream after your distribution froze. That's not folklore. That's just a new kernel containing code your old kernel doesn't have.
And the other reason.
Performance, responsiveness, latency. Which is where it gets contentious, because those three words are not synonyms and the kernels that sell on them are trading one for the others.
Define what's actually tunable.
CPU scheduling, obviously. Memory management. I/O scheduling, where you've got choices like Kyber for multi-queue devices and BFQ for single-queue. Power management, the CPU frequency governors. And system latency, which is the preemption model, how aggressively the kernel will interrupt what it's doing to let something else run.
Every one of those is a trade.
Every one. Give more, smaller time slices to threads and your latency improves while your throughput drops. That's the axis. On a desktop you might want the latency. On a database server you almost always want the throughput.
And that's the tension the specialized kernels are built around.
Take Liquorix. Their own description is "designed for uncompromised responsiveness in interactive systems," and they name their audiences. Low-latency audio and video production, and reduced frame-time deviation in games. Not average frame rate. Frame-time deviation. The stutter, not the speed.
What do they change to get there?
Zen interactive tuning, which by their own account trades throughput and power for responsiveness. A different CPU scheduler, either PDS or BMQ depending on the build. A thousand-hertz tick, which is the timer interrupting the system a thousand times a second instead of two hundred and fifty. Timeslices shortened from four milliseconds to two. Hard kernel preemption. Preemptible RCU. Kyber or BFQ for I/O. TCP BBR3 for networking. Multi-generational LRU on by default.
So it's a stack of deliberate choices, each one buying responsiveness with something else.
And XanMod is a different philosophy wearing similar clothes. They call it general-purpose. They're built with LLVM software pipelining and polyhedral compilation, and they tune core scheduling, load balancing, caching, the virtual memory manager and CPU frequency for heavy workloads rather than for interactivity.
Heavy workloads meaning what.
Sustained compute. They default multi-generational LRU, they support sched_ext, they ship BBRv3, they carry Valve's Steam Deck drivers, and they take part of the Clear Linux patchset.
And they ship builds for different CPU generations.
x86-64 version one, two and three. Which is exactly where Daniel's question about CPU-specific compilation lands, and XanMod's own documentation is the answer.
Say it.
They label x86-64 version four, the AVX-512 build, as having no kernel benefit. Their words. The people shipping the CPU-optimized kernel are telling you the most aggressive CPU optimization doesn't help the kernel.
That's the whole enthusiast argument, undermined by the vendor.
A kernel is mostly memory management, scheduling and bookkeeping. Those aren't vector math. AVX-512 is for multiplying big matrices. The kernel doesn't multiply big matrices. There's no loop for the compiler to vectorize because the hot paths aren't that kind of code.
So how much of the enthusiast kernel story is the placebo you've been warning about for years?
Less than you'd think, more than the marketing admits. The preemption changes are real, they're just not free. Which takes us to the third one, the real-time kernel, because that's the cleanest example of a kernel optimizing for something that isn't speed at all.
Real-time doesn't mean fast.
Real-time means bounded worst case. It means you can guarantee the system responds within a window, not that it responds quickly on average. The way PREEMPT_RT does that is by replacing spinlocks with preemptible, priority-inheriting mechanisms, so a high-priority task can't get stuck behind a low-priority one holding a lock.
And that costs you throughput.
It costs you throughput. Which is fine, because if you're running a robot arm or a synthesizer you don't care about averaging slightly better, you care that the worst case never blows your deadline.
Where does that patch live now?
In the mainline kernel. It was merged in 6.12, in September of 2024, after roughly twenty years out of tree. First release with real-time baked in for x86, ARM64 and RISC-V. The out-of-tree patch still exists and they still recommend it, for extra architectures and ongoing improvements. But running real-time on mainstream hardware is now a configuration option, not a patchset you apply.
Two decades of work, and the payoff is that it stopped being interesting.
That's the success condition. Something that was a specialized kernel became infrastructure. And there's more of it coming. Linux 7.1 is bringing real-time support for 32-bit ARM with no out-of-tree patches at all.
Stay on the scheduler, because that's where the actual news is.
Two things happened. One, EEVDF in 6.6, which is a general-purpose scheduler that works fine for most things and is admittedly not optimal for many. The Linux community says that out loud, which I appreciate.
And two?
sched_ext, in 6.12. This is the one that changes the shape of the whole argument. sched_ext lets you write a CPU scheduler as a BPF program and load it into a running kernel. Dynamically. You don't rebuild anything.
And if your custom scheduler is broken?
There's a watchdog. If the loaded scheduler misbehaves, the kernel boots it and falls back. The pitch is that it's safe, it cannot crash your system the way a bad in-tree scheduler change would.
So the scheduler becomes something you load and unload, like a driver.
And Valve is the proof of concept. They worked with Igalia on a sched_ext scheduler for the Steam Deck and reported more consistent frame rates, and they've been moving toward enabling it by default.
More consistent, not faster.
More consistent. Which is the right target for a handheld. You'd rather have a stable sixty than a jittery eighty. And it means the Steam Deck doesn't need its own kernel to get that. It needs its own scheduler, loaded on top of a normal one.
Now the benchmarks, because Daniel asked specifically and this is where the story turns.
Phoronix is the reference here, and the December 2024 comparison is the one to know. They took Liquorix 6.12.5 and upstream 6.12.5, same version number, and ran them head to head on a Ryzen 9 9950X. A hundred and one benchmarks.
What was the result?
Upstream won fifty-six of them. And on a geometric mean across the whole suite, upstream was twenty-one percent faster.
The responsiveness kernel was twenty-one percent slower.
On throughput. And that's not a contradiction, it's the entire point. Liquorix is not trying to win those benchmarks. It's trading throughput for latency on purpose, and the benchmark suite measures throughput, so of course it loses.
Did it win anything?
It did well on the synthetic kernel micro-benchmarks, which is a bit circular, and on Darktable and some video and image encoding. Mostly it lost the heavy compute tests, which is exactly where throughput matters most.
So the marketing and the benchmark aren't in conflict. They're measuring different axes and one of them got put on the wrong axis.
Which is the most useful thing to take from it. "Faster" and "more responsive" are different claims. A kernel can be worse at one and better at the other, and the benchmark only tells you which question it was asked.
There are older comparisons too.
2021, a Ryzen 5 laptop, a hundred and eighty-eight benchmarks. XanMod, both the stable and edge builds, had a small geometric-mean lead over mainline. Liquorix came in behind the others.
Small lead, so within noise-ish.
Small enough that I wouldn't rebuild my system for it. And then there's the 2020 test, which is the funniest one, because it's a case where a default choice actively hurt. Liquorix defaulted to the BFQ I/O scheduler, and Phoronix ran SQLite with eight threads on NVMe.
And BFQ is built for spinning disks.
BFQ is built to be fair to interactive processes on a single queue. NVMe is a multi-queue device with enormous parallelism. Putting BFQ in front of it on a database workload was a significant disadvantage. They've since moved the default, but the lesson holds. The optimization is a bet, and sometimes the bet is wrong for your machine.
Which is why the rule should be that you measure before you switch.
Measure before you switch. And the measurement is the hard part, because Daniel asked whether the gains show up in real applications or only in synthetic benchmarks, and the honest answer is: sometimes one, sometimes the other, and it depends entirely on whether your actual workload stresses the thing the kernel changed.
If you're compiling, the scheduler barely matters and throughput dominates.
If you're compiling, you care about throughput. If you're doing live audio, you care about the worst-case scheduling latency and throughput is irrelevant. Same kernel, opposite verdict.
Server versus desktop.
This is where deliberate kernel selection gets more defensible, because servers have measurable, stable workloads and skilled operators. If you're running a latency-sensitive trading system you already know you care about the tail, not the mean. If you're running a database you care about throughput and page cache behavior. If you're virtualizing, you might care about I/O scheduling. And you can actually benchmark your own workload, which is the rare advantage a desktop user doesn't have.
The desktop user is guessing.
Their workload is whatever they happen to do today, which makes benchmarking it nearly impossible and makes the default kernel the correct choice almost every time.
So most people should just leave it alone.
Most people should leave it alone. The default kernel is a curated, maintained, security-backported product that the distribution has tested against an enormous range of hardware. Swapping it for something from a website is trading that for a specific behavior you probably haven't measured.
When is it not a waste of time?
Hardware support. New chip, old distribution, HWE kernel, clean justification. Or a specialized workload where you know your latency or throughput requirement and can measure it. Or you want to tinker, which is a legitimate reason and not one I'd argue with.
Say the trade-offs on the other side.
Stability, because specialized kernels lag on regression fixes. Security, because they don't have Canonical's backport pipeline. Driver compatibility, because a proprietary driver may not build against a kernel it wasn't tested on. And support, because when something breaks the first question anyone asks is what kernel you're on, and the answer "a build from a forum" ends the conversation.
Herman, you once predicted that the specialist kernel community shrinks as features get absorbed into distributions. Still hold that?
More than ever, and the real-time merge is the example. When PREEMPT_RT became a config option, a whole category of specialist kernel quietly went away. The same thing is happening with sched_ext. You don't need a gaming kernel if you can load a gaming scheduler onto a stock one.
Which reframes the whole debate.
It does. If the future is loading a scheduler per workload instead of choosing a kernel per workload, then the kernel version matters less and less. The Steam Deck story is that future arriving early.
And XanMod's own documentation saying AVX-512 has no kernel benefit.
That's the tie-breaker for the CPU-specific argument. If the most aggressive build option does nothing for the kernel, then the "compiled for my exact CPU" pitch is mostly a feeling.
So where does this leave the default?
Close to optimal for most users, meaningful for a specific slice, and mostly misunderstood in the middle. The people who switch and are happy are usually the ones who knew exactly what they wanted and measured it. The people who switch and are disappointed are the ones who wanted "faster" and got "different."
Ninety-nine percent of people should never touch it.
Ninety-nine percent. And the one percent who should know exactly why.
It's a hundred and twenty hertz, not a thousand.
The tick rate. He said a thousand. On most stock builds it's two hundred and fifty, sometimes a hundred. Liquorix uses a thousand. But the number that matters is the one on your own machine and most people never look at it.
That's a fair correction.
I ran a custom kernel once because my neighbor claimed his machine booted faster.
Of course you did.
So we sat there with a stopwatch. I had mine at a thousand hertz, he had his at two hundred and fifty, and we spent the whole weekend opening applications and timing them. Same apps, same order.
With a stopwatch.
He'd say go, and we'd click, and we'd watch the clock.
And who won?
He did. His system was faster. But he had a newer SSD, so it wasn't the kernel, and I told him that.
So you kept the kernel.
I still run it. It's a laptop I use to check email and watch videos. But every time a new release comes out I rebuild it from source, and that takes most of a weekend.
Every release.
I've got a spreadsheet of the compile times going back years. And the times are getting faster, which I'd like to think is my configuration, though I did get a new CPU at some point.
A spreadsheet of compile times.
Column for the version, column for the time, notes column. I track my notes.
And the tinkering is the point, not the speed.
The speed was never the point. Knowing you built the thing you're running, that's the point. The email doesn't load any faster. I just know how it got there.
Hilbert, let me set the levels before we keep going. Your mic's a touch hot on the plosives.
Noted. Anyway, the spreadsheet's the useful part. Every rebuild the spreadsheet gets a new row, and I can see the machine getting quicker, which is exactly what you'd expect from a newer CPU and nothing to do with the kernel at all. I know that. I still keep the spreadsheet.
So the tinkering is the reward, and the benchmark is incidental.
That's it. The benchmark is what you do to justify the tinkering.
Which is a better answer than ours. We told people the default is fine. You're telling them why the people who ignore that advice are still getting something out of it.
They're getting a spreadsheet.
One thing I want to pick up from Hilbert, before we land this. The compile times getting faster because of a new CPU and not the configuration, that's the same mistake as the benchmark, just harder to see. The thing you changed gets the credit for what something else did.
The stopwatch and the spreadsheet are the same instrument. You're measuring, but you're not controlling for anything.
Going back to the sched_ext question, because I think it's the one Daniel's really circling. If you can load a scheduler per workload instead of choosing a kernel per workload, does the kernel version stop mattering?
Less, but not never. You still need a kernel new enough to support sched_ext, which means 6.12 or later. And sched_ext only covers the scheduler. It doesn't touch memory management or I/O scheduling or power management, so if your bottleneck is one of those you're back to choosing a kernel.
So it's a partial decoupling.
The most useful one available. And the real-time merge is the pattern repeating. Twenty years as a specialized patchset, then one release where it becomes a config option, and suddenly nobody has to run a custom kernel to get bounded latency.
Which ones follow it?
The real-time support for 32-bit ARM lands in 7.1 with no out-of-tree patches, so that's the next one. And I'd bet on a bunch of the smaller latency patches from the enthusiast kernels following the same path over the next few years.
The cutting-room floor, one thing. XanMod's build list, dated this month. They've got a main build on 7.2, an LTS build on 6.18, and an RT build on 6.18 from the third of October. A specialty kernel distribution tracking the real-time branch right alongside its stable builds.
Which tells you real-time has stopped being a separate world. It's a flavor now, the same way low-latency is a flavor of Ubuntu's kernel. That's the merge showing up in a build list.
The question I'm left with is whether Valve ships that sched_ext scheduler to every Steam Deck, because if a per-workload scheduler becomes the default on a mass-market device, the whole alternative-kernel category gets smaller overnight.
And the answer is nobody outside Valve knows yet, which is why it's worth watching. The one thing I'd tell Daniel is that his instinct about the second axis is right. "Faster" and "more responsive" are two different questions, and most of the confusion in this space comes from people asking one and getting an answer to the other.
Which is exactly what Hilbert's spreadsheet was doing. Thanks to our producer Hilbert Flumingtop for holding the levels and the spreadsheet. If this was your kind of episode, go back for episode seven twenty-nine, The Surprising Family Tree of Modern Operating Systems; episode thirty-seven eighty-nine, What Virtualization Actually Costs on two thousand twenty-six Hardware; and episode eleven, How Does Fine Tuning Work Anyway. This has been My Weird Prompts, the human-AI collaboration podcast.
If you enjoyed this episode, leave us a review and subscribe. It helps other listeners find the show. We'll be back soon.
