Daniel's got a whole taxonomy of a question this week, and it starts with something that sounds simple but isn't. He says the received wisdom on CPU inference has hardened into a kind of dogma — that anything serious needs a GPU, that if you're running on CPU you're either doing a toy demo or you've made a mistake. And his pushback is that this framing is at least partly wrong, and almost certainly much too coarse. He wants us to start by being precise about what GPUs are actually giving us, because "it's faster" is not an answer. Is it raw parallel throughput on big matrix multiplies? Is it memory bandwidth? Is it batching amortisation across concurrent requests? His hunch — and I think he's onto something — is that memory bandwidth is doing more of the work in that story than people admit, and that the "GPUs are for parallelism" explanation is a slightly outdated folk model.
It's a good hunch. And the way he's framed it forces us to separate three things that get mashed together constantly. Parallel compute throughput — how many floating point operations you can do per second. Memory bandwidth — how fast you can move weights from RAM into the compute units. And batching amortisation — spreading the cost of loading those weights across many requests at once. They're different bottlenecks for different workloads, and the GPU advantage isn't one thing, it's a bundle. Some workloads need all three. Some need exactly one.
So let's unbundle them.
Right. The core operation in a transformer — any transformer, large or small — is a matrix multiply. You've got your token embeddings, your attention projections, your feed-forward layers. Every single one of those is, at bottom, multiplying two big matrices together. A GPU has thousands of cores designed to do exactly that — multiply and accumulate — in parallel. An H100 has something like eighteen thousand CUDA cores. A server CPU has, what, ninety-six cores if you're being generous. The raw throughput gap on dense matrix math is enormous. That's the parallelism story.
And that's the part everyone knows.
That's the part everyone knows. But here's where it gets more specific. During prefill — when you feed the model your entire prompt at once — you're doing one enormous matrix multiply. Every token in the prompt gets processed simultaneously. That's massively parallel, and the GPU chews through it because it can throw thousands of cores at the problem. The CPU chugs along doing it in much smaller chunks. Prefill is GPU territory.
But then there's decode.
Then there's decode. And decode is a completely different animal. During decode, you're generating one token at a time. Each new token depends on the previous one — it's sequential by definition. You can't parallelise across the sequence. So what are you actually doing? For each token, you load the entire model — every weight, every parameter — from memory, do a relatively small amount of math, and spit out one token. Load the whole model. Produce one token. Load the whole model. Produce the next token.
So the bottleneck shifts.
The bottleneck becomes memory bandwidth almost entirely. How fast can you get those weights from RAM into the compute units? The math per token is trivial compared to the data movement. This is the roofline model — you're memory-bound, not compute-bound. And here's the thing. A modern server CPU with eight or twelve channels of DDR5 has memory bandwidth on the order of four to five hundred gigabytes per second. An H100 has over three terabytes per second. That's a six-to-one gap, roughly. Significant, absolutely. But it's not the hundred-to-one gap you see in raw FLOPS. The GPU advantage compresses dramatically when you're memory-bound at batch size one.
So Daniel's hunch about memory bandwidth is basically right. For single-user, single-request inference — which is the deployment shape for an enormous number of real applications — the GPU's massive parallel compute advantage is largely irrelevant, and you're left with a bandwidth gap that's real but much narrower.
And that gap narrows further when you quantise. Take a model like Llama three point two three B, or Phi three mini, or Qwen two point five seven B. Quantise those down to four bits, maybe three bits. The model that was seven gigabytes at sixteen bits is now around two gigabytes. At two gigabytes, even a laptop with LPDDR5 at a hundred gigabytes per second can push that through at — what — fifty tokens per second? On a CPU. Using llama dot cpp. That's not a toy. That's a usable chat experience.
And that's where the batching question comes in. Daniel asked about batch size one specifically. If you're OpenAI and you're serving ten thousand concurrent users, you can batch their requests together, load the model once, and amortise that weight-loading cost across a hundred queries. The GPU's memory bandwidth gets multiplied by your batch size, effectively. That's where the economics of GPU serving really shine — high concurrency, dense batching, everyone's requests getting processed in the same matrix multiplies.
But if you're a company running an internal chatbot for fifty employees, your concurrency might be three. Maybe five at peak. You can't amortise across a big batch because there's nobody to batch with. At that point, the GPU is sitting there loading the model for each tiny batch, and most of its compute is idle. You're paying for parallel throughput you can't use.
And a GPU sitting idle is just a very expensive space heater.
With a very loud fan.
So let's get into the specific workloads Daniel asked about. He mentioned classical ML — gradient-boosted trees, XGBoost, random forests, linear models. He even asked whether including those is cheating.
It's not cheating at all. And he's right that an enormous amount of production ML at real companies is exactly that stuff. Tabular data, fraud detection, credit scoring, demand forecasting, churn prediction. XGBoost and LightGBM are not matrix-multiply-heavy. They're doing tree traversals, which are inherently sequential and branchy — exactly the kind of workload CPUs are designed for. A GPU trying to do tree inference is miserable. The control flow diverges, the branches don't map to SIMD lanes, and you end up with terrible utilisation. There's been academic work on GPU-accelerated tree inference and it's... fine, for very specific cases. But in practice, everyone runs XGBoost on CPU and it screams.
Because the model itself is tiny. A trained XGBoost model for fraud detection might be a few megabytes. Loading it is instantaneous. Inference on a single row is microseconds. You don't need bandwidth, you don't need parallel throughput, you just need branch prediction and cache.
And the same goes for the whole layer of what Daniel called unglamorous models doing actual work. Embedding models — if you're running something like BGE small or all-MiniLM-L6-v2, that's a hundred million parameters or less. It fits in cache. You can run it on a CPU core in single-digit milliseconds per sentence. Rerankers, same story. A cross-encoder reranker like BGE reranker base or Cohere's rerank model — these are small transformer models, sure, but at batch size one on CPU they're fast enough that the bottleneck in your RAG pipeline is almost certainly somewhere else. Probably retrieval, probably the vector search, probably the network call to your vector database.
And vector search itself — that's CPU territory. FAISS, HNSW, all the approximate nearest neighbour libraries are CPU-first. Some have GPU backends, but for most deployments the index fits in RAM and the CPU does the distance computations faster than you can shuttle the vectors to a GPU and back.
The round-trip latency kills you. The PCIe transfer alone can be longer than the search time on CPU.
So we've got this whole category — classical ML, small transformers, embeddings, rerankers, vector search — where the GPU advantage is either zero or negative once you account for data movement. That's not a compromise. That's the right call.
Let me add one more to that list. Sparse models and mixture-of-experts. This is where things get interesting. A mixture-of-experts model like Mixtral eight by seven B or DeepSeek V2 — for each token, you only activate a fraction of the parameters. Mixtral activates two of its eight experts per token. So even though the model has forty-seven billion total parameters, you're only touching about thirteen billion per token. The rest sits in memory and does nothing.
Which means the effective model size per token is much smaller than the total.
And on CPU, that's a huge advantage. You load the active expert weights, compute, move on. On GPU, you still have to keep all the experts in VRAM, and you're still memory-bandwidth-bound during decode. The sparsity doesn't reduce the bandwidth demand as much as you'd hope because the active experts still need to be fetched. But on CPU with good quantisation, a Mixtral model can run at reasonable speeds because the total data moved per token is a fraction of the full model size. There are people running Mixtral eight by seven B quantised on a MacBook with decent performance. Not blazing, but usable.
Daniel asked about Whisper specifically. What's the honest picture there?
Whisper is fascinating because it's a perfect case study in CPU viability. The model itself is an encoder-decoder transformer. The encoder runs once over the entire audio input — that's parallelisable, GPU-friendly. The decoder generates tokens sequentially, same as any autoregressive model. But here's the thing: Whisper's models are not that large. Whisper large V3 is about one point five billion parameters. Whisper medium is around seven hundred and fifty million. Whisper small is two hundred and fifty million. Whisper tiny is thirty-nine million.
Thirty-nine million parameters. That's practically a rounding error by current standards.
And Whisper tiny, running on a modern CPU with something like faster-whisper or whisper dot cpp, can transcribe faster than real-time. Real-time factor below one. Accuracy is not amazing — it'll get the gist but it'll make mistakes. Whisper small, still faster than real-time on a decent CPU, and the accuracy is good for clean audio. Whisper medium starts to push it — you might be at one point five to two times real-time, depending on the CPU. Whisper large V3, you're probably at three to five times real-time on CPU. That's slow for live transcription but perfectly fine for batch processing overnight.
And for a lot of use cases — podcast transcription, meeting notes, anything asynchronous — you don't need real-time at all. You need acceptable latency and low cost.
Right. If you're transcribing a hundred hours of audio a week, and you can do it on a CPU instance that costs a dollar an hour instead of a GPU instance that costs three dollars an hour, and the CPU finishes in eight hours instead of two — who cares? It ran overnight. You saved money. That's the economics Daniel's getting at.
Let's talk about that crossover. If a workload is intermittent, is CPU cheaper per unit of work even when it's slower per unit of time?
Almost certainly. And the reason is utilisation. A GPU that's sitting idle between requests is costing you money. If you're on cloud, you're paying for that instance whether it's computing or not. If you're on-prem, you've sunk capital into hardware that's drawing power and doing nothing. CPU instances are cheaper per hour, and for spiky workloads, the total cost of ownership tilts hard toward CPU. The crossover point depends on your request volume and your latency requirements, but for anything with low to moderate concurrency, CPU often wins on total cost.
Even if the per-request latency is higher.
Even then. Because the GPU's speed advantage only matters if you're saturating it. A GPU that's ten times faster but only utilised twenty percent of the time is effectively only twice as fast in throughput terms — and it costs more than twice as much per hour. The math is not complicated, but people skip it constantly.
Daniel also asked about deployment shapes where CPU is the right call for reasons beyond pure economics. Edge and embedded, air-gapped environments, anything where the model has to live next to data that can't move.
Air-gapped is a huge one. If you're doing inference in a classified environment, or a hospital with patient data that legally cannot leave the building, you're running on whatever hardware is in that room. That hardware is almost certainly CPU-based servers. You can't just spin up a GPU cluster in a SCIF — well, you can, but it's expensive and the procurement cycle is measured in years. Meanwhile, the CPU servers are already there, and they can run quantised LLMs, they can run Whisper, they can run embeddings. The model comes in on a USB stick, gets loaded, and you're done.
Cold start is another one Daniel flagged. If you're in a serverless environment — Lambda, Cloud Run, whatever — your container might be sitting cold. A GPU container cold start is brutal. You've got to provision the GPU, load the driver, load the model into VRAM. That can be thirty seconds, a minute. A CPU container with a small quantised model can start in under a second and be serving requests immediately. For anything event-driven with low traffic, that's the difference between a viable service and a timeout.
And that connects to the economics again. If you're paying per invocation, or per millisecond of compute time, the GPU's cold start penalty is eating your margin on every single request that isn't batched.
So we've established a pretty clear picture of where CPU works well. Classical ML, small transformers, sparse models, embedding and reranking, vector search, Whisper up to medium, quantised small LLMs at low concurrency, air-gapped and edge deployments, spiky workloads, cold-start-sensitive environments. Where does it fall off a cliff?
Large models at high concurrency. If you're serving Llama three seventy B to thousands of concurrent users, you need GPU memory bandwidth and you need batching. A CPU cluster can technically do it — you can shard the model across many nodes, use distributed inference — but the latency per token becomes unacceptable for interactive use, and the total cost of all those CPU nodes starts to approach or exceed the GPU cluster cost anyway. At that scale, GPU is the right tool.
And large-batch prefill-heavy workloads. If you're doing bulk embedding of millions of documents, or batch processing thousands of prompts through a large model, the GPU's parallel throughput on prefill matters. The CPU will get there eventually, but the GPU gets there in a fraction of the time and at lower total cost if the utilisation is high.
Training. We haven't mentioned training because Daniel's prompt is about inference, but it's worth saying: training is still GPU territory, almost exclusively. The backward pass requires keeping activations in memory and doing enormous amounts of parallel computation. CPUs are not competitive for training anything beyond toy models. That boundary is not moving anytime soon.
There's a middle ground — models that are too large to run comfortably on CPU but don't need a full GPU. That's where Apple Silicon has carved out a niche. The M series chips have unified memory with bandwidth in the hundreds of gigabytes per second — comparable to mid-range GPUs — and they can run models like Llama three point one seventy B quantised at usable speeds. That's not strictly CPU inference, it's a different architecture, but it's blurring the line.
That brings us to Daniel's last question: is the CPU-viable envelope growing? I think the answer is clearly yes. Three things are happening simultaneously. CPU vector units are getting wider — AVX-512, AMX on Apple Silicon, SVE on ARM — so the raw throughput per core is climbing. Quantisation techniques are getting better — we're seeing viable two-bit and three-bit quantisation that preserves most of the model quality, which shrinks the memory bandwidth demand proportionally. And NPUs are showing up in ordinary laptops and phones, offloading the matrix multiply work to dedicated hardware that sits alongside the CPU.
The NPU story is interesting because it's not CPU inference in the traditional sense, but it's CPU-adjacent. It's inference without a discrete GPU. And the NPU in something like a Snapdragon X Elite or an Intel Meteor Lake chip is doing the same kind of matrix math a GPU does, just at lower power and with tighter integration to the CPU's memory system.
The Snapdragon X Elite NPU is rated at forty-five TOPS at INT8. That's not nothing. For comparison, an RTX 4090 is around thirteen hundred TOPS at INT8. So you're talking about a thirtieth of the throughput, but at a fraction of the power, and it's in a laptop that costs twelve hundred dollars. For small models, that's perfectly viable. Microsoft is shipping Phi Silica, a three point eight billion parameter model that runs entirely on the NPU in Copilot Plus PCs. That's a real product, shipping now, doing local inference on what is effectively a CPU package.
The boundary is moving. The question is how fast, and for which workloads.
I think the biggest shift in the next few years is going to be in the five to fifteen billion parameter range. Right now, those models are in an awkward spot — too large for comfortable CPU inference on most hardware, too small to justify a dedicated GPU. They're the sweet spot for Apple Silicon and for upcoming NPU-equipped laptops. As quantisation improves and memory bandwidth on consumer hardware creeps up, I think we'll see those models running locally on ordinary machines at fifty-plus tokens per second within a couple of years.
Which covers an enormous amount of use cases. A fifteen billion parameter model, well-trained, can do summarisation, classification, extraction, basic reasoning. That's the work most companies actually need done, as opposed to the frontier stuff that makes headlines.
That's the point Daniel's really driving at, I think. The discourse about AI infrastructure is dominated by the people doing the hardest thing — frontier training, large-scale serving, pushing the envelope. That's an unrepresentative corner of the field. Most ML in production is not that. Most ML in production is a gradient-boosted tree predicting customer churn, or a small transformer classifying support tickets, or an embedding model powering semantic search. And most of that runs on CPUs, happily, and has for years.
The "everything needs a GPU" belief is partly genuine engineering truth for a specific set of workloads, and partly just the fact that the loudest voices are the ones with the biggest clusters.
The ones selling GPUs.
That too.
Hilbert: We had this exact conversation in two thousand four. Not about GPUs. About dedicated DSP chips for audio processing. Same argument, same shape.
I'm listening.
Hilbert: I was doing install work for a studio in Manchester. The owner had spent forty thousand pounds on Pro Tools HD cards — dedicated DSP, each one did a fixed number of tracks, very expensive, very impressive. And the resident engineer kept saying, look, the host CPUs are getting faster. In a few years you won't need the cards. The owner wouldn't hear it. The cards were the serious kit. The CPU was for amateurs.
How'd it play out?
Hilbert: About three years later the cards were in a skip. The CPUs caught up. The software caught up. The dedicated hardware was faster on paper but the flexibility of running it all native — no card limits, no proprietary cabling, no waiting for the DSP vendor to update their drivers — that won. The engineer was right.
There's a pattern there. Dedicated hardware wins when the performance gap is enormous and the workload is stable. General-purpose hardware wins when the gap narrows and the workload keeps changing.
Hilbert: That's what he said. I remember because I had to carry the skip. Those cards were heavy.
The GPU isn't going in a skip. But the set of things you need it for — that's a smaller set than the marketing suggests.
Hilbert: The other thing he said — the engineer — was that the DSP cards made sense if you were running a commercial studio with bookings all day. If you were tracking at ten in the morning and mixing at two and mastering at six, you needed the guaranteed track count. But most studios weren't that. Most were one room, one project at a time, and the DSP sat idle eighty percent of the time.
That's exactly the utilisation argument. If you're not saturating the hardware, you're paying for capacity you never use.
Hilbert: He had a spreadsheet. Showed the break-even. I still have the spreadsheet somewhere. It's in a box.
The spreadsheet survived the skip.
Hilbert: The skip took the hardware. The spreadsheet was paper.
The principle's the same though. If your workload is spiky, if your concurrency is low, if you're not batching heavily — the economics tilt toward general-purpose compute. That was true for audio DSP in two thousand four, it's true for GPU inference now.
The envelope where general-purpose compute is good enough keeps expanding.
Hilbert: It does. Though I'll say this. The people who insisted on the DSP cards — they weren't stupid. They'd been burned before by promises that the CPU would be fast enough. For years it wasn't. Then one day it was, and the switch happened fast. The hard part is knowing which year you're in.
That's the question, isn't it. Are we in the year where CPU inference is finally good enough for the mainstream, or are we still in the years where the promise is ahead of the reality?
I think we're in the year where it's good enough for a lot more than people admit, and the people insisting otherwise are the ones who bought the DSP cards.
Hilbert: Fair.
The boundary Daniel wanted drawn — I think it looks something like this. If you're doing classical ML, small transformers, embeddings, rerankers, or vector search, CPU is the right call, full stop. If you're doing quantised small LLMs at low concurrency, CPU is viable and often cheaper. If you're in air-gapped, edge, or cold-start-sensitive environments, CPU is often the only practical option. If you're doing large-model inference at high concurrency, or training, or bulk prefill-heavy processing, you need GPUs. And there's a growing middle ground — models in the five to fifteen billion parameter range — where the line is blurring fast, and where Apple Silicon and NPUs are making GPU-optional a real thing.
The economics are the through-line. It's not about whether GPU is faster — it usually is, in isolation. It's about whether that speed translates to lower cost per unit of work given your actual workload pattern. For a lot of real-world deployments, the answer is no.
Daniel's hunch about memory bandwidth was right. At batch size one, during decode, you're bandwidth-bound, and the GPU's massive compute advantage is largely irrelevant. The gap is real but it's single-digit multiples, not orders of magnitude, and quantisation shrinks it further.
The other thing I'd add — and this is where I think the folk model really is outdated — is that the "GPUs are for parallelism" story assumes the workload is parallelisable. Decode isn't. Not in the way prefill is. You can batch across requests, but you can't parallelise within a single sequence. So if you're a single user chatting with a model, the parallelism story just doesn't apply to most of the tokens you're generating. You're doing sequential work on hardware designed for parallel work, and most of that hardware is waiting on memory.
Which is why a MacBook with good memory bandwidth can run a quantised seventy-billion-parameter model at reading speed. The bandwidth is there. The parallel compute isn't, and it doesn't matter.
We should probably name some specific models and tools, since Daniel asked. llama dot cpp is the obvious starting point — it's the engine that makes most of this possible, with its quantisation formats and CPU optimisations. Ollama wraps llama dot cpp in a nice API. For Whisper, whisper dot cpp and faster-whisper are the CPU-friendly options. For embeddings, the sentence-transformers library with a small model like all-MiniLM-L6-v2 runs beautifully on CPU. For classical ML, XGBoost, LightGBM, and scikit-learn are CPU-native and always have been.
On the model side — Llama three point two three B and one B, Phi three mini and Phi three point five mini, Qwen two point five in the three B and seven B sizes, Gemma two in the two B and nine B sizes. All of these run on CPU with quantisation at usable speeds. Mistral seven B, Mixtral eight by seven B if you want to push it. DeepSeek's smaller models. Whisper tiny, base, small, and medium for speech.
The ceiling right now for comfortable CPU inference on a decent machine is probably around eight to ten billion parameters at four-bit quantisation. Above that, you're making tradeoffs on latency. Below that, you're in the sweet spot where CPU is good.
That sweet spot covers an enormous amount of ground.
It does. And I suspect a lot of teams are spending money on GPU instances for workloads that would run fine on the CPU servers they already have. Not because they've done the analysis and concluded GPU is cheaper — because the default assumption is that anything AI needs a GPU.
Default assumptions are expensive.
They really are.
This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop.
You can find us at my weird prompts dot com. Email us at show at my weird prompts dot com.
We'll be back soon.
