Everyone keeps saying mixture of experts makes models cheaper. It does not. It makes them bigger.
That is the part the model cards hide in plain sight. You see a trillion parameters and think supercomputer, then the active number is tiny by comparison.
Daniel is in the studio today, and he brought the exact confusion with him.
Yeah, I keep seeing these active counts all over Hugging Face and my gut says smaller active must mean better. Fewer weights firing, more selective routing. Is that roughly right.
The short version is you noticed open models crossing a trillion parameters, you noticed active counts bouncing all over the place, and you want to know what the number measures, why tuning it separates models, and what you would actually feel between a wide active slice and a narrow one in chat or prediction.
And that gut feeling about smaller being better, Corn, that is where we should start, because it is half right in a way that makes it wrong.
Okay, so what does the active number even count.
Take a normal transformer block. Usually there is one big feed forward network after attention. In mixture of experts you rip that out and replace it with a bank of smaller feed forwards, call them experts, plus a little learned router. Every token gets scored, top k experts win, their outputs get mixed by the router weights.
A trillion parameters doing shift work.
Total parameters count every expert sitting in the checkpoint. Active counts only the ones that fire for a given token, plus the stuff that is always on. Attention, embeddings, any shared experts that run every time.
Per token, not per prompt.
Per token, yes. Two words in the same sentence can take completely different paths. The Hugging Face line on this is clean. Model capacity depends on total parameters, but inference speed depends on active parameters.
But how does that work in practice? Does the router look at the word itself, or the whole context around it?
It looks at the hidden state for that token at that layer, so it has already absorbed context from attention. It is a tiny linear layer that spits out scores for each expert, then you take the top k and softmax just those. So "bank" in "river bank" and "bank" in "savings bank" can score completely different experts by layer six or seven.
Same word, different plumbers.
And that scoring happens at every MoE layer, for every token?
Every MoE layer, every token, every forward pass. That is why routing quality compounds. A small mistake early reroutes everything downstream.
Capacity in storage, speed at the till.
Worked example helps. Take gpt oss twenty b. Twenty one billion total in the checkpoint. Thirty two experts in each MoE layer, four active per token. So per token you only pay for about three point six billion active.
So it stores like a twenty billion model and runs like a three and a half billion model.
In compute per token, roughly, yes. Memory is the catch, and we will get there, because you still have to store all twenty one billion somewhere. But the arithmetic per token is that small.
Store an elephant, ride a bicycle.
What do you think, Corn, does that match how you picture it.
I picture a hotel where only four rooms ever have the lights on, but you still pay to heat the building. Which is why your smaller is better idea feels tidy and is probably doomed.
It is doomed in an interesting way. Sparsity does push experts to specialize. When routing gets sparser, expert neurons get less polysemantic, less doing five jobs at once, compared to a dense feed forward. So your instinct about selectivity is not crazy.
But what about the failure mode? If only four out of thirty two ever fire, do the other twenty eight just atrophy?
They can, and that is called expert collapse or load imbalance. Without an auxiliary load-balancing loss, the router finds two or three experts it likes and hammers them, while the rest starve. So training has to actively force traffic to spread out, otherwise your thirty two experts are really three experts and twenty nine tourists.
Tenured tourists.
But they are not specialists the way I imagined, right. I imagined a biology expert, a law expert.
No, and that is the fun part. They turn out to be fine grained task experts. Linguistic operations, little semantic jobs. The canonical example going around this year is an expert that basically does closing brackets in LaTeX.
Ten years of training to shut parentheses.
Not biology versus law. More like this cluster of neurons learned when a curly brace needs to die. That is specialization, just at a level humans find anticlimactic.
Is that really all they do? Just punctuation?
It is more like micro-syntax plus context. One expert might handle closing brackets, but only when they close a math environment, not plain text. Another might handle past-tense verbs after proper nouns. Humans want chapters. The model learns commas.
I spent English class the same way. Learned nothing but commas.
So sparser gives more of that narrowness, which sounds like an argument for smaller active.
Here is where it flips. There is a design paper from January that tried to put this on rails. Their claim is total and active alone are not enough to describe an optimal architecture. Performance is mostly set by total parameters and sparsity, where sparsity is number of experts divided by top k.
Such a dry ratio for such a weird idea.
And their recommendation is the opposite of minimize active. Maximize total under your memory limit, minimize sparsity, which means maximize top k, and minimize number of experts. In plain English, fire more experts per token, keep fewer of them sitting around.
Wait, fire more. That is literally backwards from my guess.
What do you make of that, Corn. Does bigger top k feel wasteful to you.
It feels like hiring four plumbers to watch one sink. But if each one only knows lefty loosey, maybe you need the crowd.
That is actually close. More active experts means more capacity engaged per token, and the router gets far more combinations to play with. Selectivity is about routing quality, not about how few you fire. Two experts can be badly chosen. Eight can be well chosen.
So how many combinations are we actually talking about? With eight out of a hundred and twenty eight, that has to be enormous.
It is combinatorial. Eight out of a hundred and twenty eight is more than thirty billion possible sets per layer, and you have dozens of layers. Even four out of thirty two is thirty six thousand combos per layer. The router is not picking an expert, it is picking a committee, and a slightly different committee per token.
Congress, but it works.
So fewer is not more selective, fewer is just fewer.
Yes. And there are two other knobs that matter as much as the headline numbers. Granularity, which is model width divided by expert width, sweet spot around four to eight, diminishing returns past that. And width to depth ratio, sweet spot around forty two, keep it between thirty two and sixty four.
Forty two. Of course.
I know, I laughed too. But even with total and active fixed, you can build many different models out of depth, width, expert count, active count, granularity. That is why tuning it is a differentiator. The card shows two numbers. The design hides five decisions.
Can you make granularity concrete? If I fix active at, say, twenty billion, what changes if I split it into tiny experts versus chunky ones?
Think pizza. Same total dough, you can cut eight big slices or sixty four tiny squares. Tiny squares let you assemble a very precise plate for each token, a little syntax here, a little factual recall there. But go too tiny and each expert is too weak to learn anything coherent, and routing overhead explodes because you are scoring hundreds of experts per token.
Death by tapas.
Four to eight hits the balance. Big enough to be useful, small enough to mix.
And DeepSeekMoE was the recipe for that.
That was the formalization. Split experts into smaller ones so you can mix and match more flexibly, and peel off a few shared experts that always run to soak up common knowledge. Fine grained segmentation plus shared expert isolation. That cut redundancy and let the routed experts actually diverge.
We did the fine tuning episode years back, the one about how a tiny tune changes which weights matter. This is the cousin. Tuning changes what matters, routing changes what wakes up.
That is a good way to put it. Same moral. The checkpoint is not the behavior.
Give me a concrete win for that recipe, not just theory.
DeepSeekMoE at two billion parameters matched GShard at two point nine billion, so about one and a half times the expert compute, and nearly matched a dense model of the same total size. Smaller active budget, same quality, because the experts stopped tripping over each other.
What were they tripping over before? Just learning the same thing twice?
Mostly common knowledge and stopwords. Every expert was relearning "the" and "and" and basic grammar, wasting capacity. Once you give that to a shared expert that is always on, the routed experts are free to diverge into those weird bracket-closers and tense-handlers. Redundancy drops, diversity goes up.
Less overlap, more closing brackets.
Do you buy that tradeoff, Corn, or do you still want the tiny active number for elegance.
I want the tiny number the way I want a tiny electric bill. Whether it writes better is a separate question, Herman.
Okay, that is design. What about deployment. If active sets speed, MoE should be faster, right.
That is the second flip, and it is harsher. There is a March paper with the qs inequality. Their argument is MoE pays a double penalty at inference. Routing fragments your microbatch, so different tokens want different experts and you lose weight reuse. And the whole resident expert pool sits in high bandwidth memory that could otherwise hold KV cache.
A hotel lobby full of luggage, no room for guests.
For DeepSeek V three at a hundred and twenty eight k context, they say a quality matched dense baseline gets about a four and a half times throughput advantage. Four and a half times.
That is huge. So MoE is not an inference win at all.
Their line is training time flop efficiency is a poor proxy for inference time economics. MoE may be best viewed as a training time optimization, with distillation into dense models as the path to inference efficient deployment.
Wait, why does the KV cache matter so much here? I thought that was separate from weights.
It is separate, but they share the same HBM. At long context, KV cache balloons. A dense model uses its HBM mostly for cache, so it can serve huge batches. An MoE has to keep hundreds of billions of expert weights resident, even the idle ones, so there is far less room left for cache. Smaller batches, lower throughput, higher cost per token.
Train fat, serve thin.
Corn, would you rather have the giant MoE or the distilled dense if you are just chatting.
I would rather have whatever answers before I fall asleep. Which, given my pace, is a low bar, Herman, and most models still miss it.
You would feel it in two places. Larger active means more compute per token, higher cost and latency, but more capacity engaged per token. Smaller active means cheaper and faster for the same stored capacity, but you lean harder on routing quality. Bad routing with narrow active hurts more.
Is there a fun parallel here? This feels like the brain thing where we only use a fraction at a time.
It is the analogy everyone reaches for, and it is half useful. Your brain has about eighty six billion neurons and only a sparse subset fires for any task. But neurons are slow and massively parallel with near-zero movement cost. GPUs are the opposite, fast and starved for memory bandwidth. So biology gets sparsity almost for free, we pay for it in routing and HBM.
My brain fires a sparse subset too. Usually the wrong subset.
Can we ground that in real cards.
Let us do it. DeepSeek V three, six hundred seventy one billion total, thirty seven billion active. On Hugging Face it lists as six hundred eighty five billion because it includes a fourteen billion multi token prediction module. Trained on about fourteen point eight trillion tokens, about two point seven eight eight million H eight hundred hours.
A rounding error in GPUs.
Qwen three two thirty five B A twenty two B, two hundred thirty five billion total, twenty two billion active, a hundred and twenty eight experts with eight active, ninety four layers. Little sibling Qwen three thirty B A three B, thirty point five billion total, three point three billion active, same a hundred and twenty eight experts and eight active, forty eight layers.
Same router shape, very different bodies.
Mixtral eight by seven B, the early one, forty six point seven billion total, about twelve point nine billion active, eight experts top two. And Kimi K two Instruct, one thousand twenty six point four billion total. That is your trillion parameter open model.
A trillion parameters and it still cannot load the dishwasher.
For chat, what is the felt difference between the thirty B A three B and the two thirty five B A twenty two B.
Qwen markets the small one as efficient general purpose dialogue, the big one adds reasoning depth. In practice the small active can run in tight places, the Hugging Face blog notes gpt oss twenty b running on an M three Ultra Mac, while the large active wants datacenter hardware, more memory bandwidth, longer time to first token under load, higher per token price.
So in a long agentic run, the small one would stay coherent but lose the thread on multi-step math, while the big one holds the plan but costs a fortune?
That is the typical report. Narrow active is fine for snappy back-and-forth, summarization, single-step tools. Wide active holds up better when you need five hops of reasoning without dropping state, because more capacity is engaged on each token to keep the plan alive. You pay for that memory in latency and dollars.
So narrow active is local and snappy, wide active is smarter but heavier.
Often, but the card number lies by omission. Two models can share the same total and active and differ entirely in granularity, depth to width, shared expert count. That is why active alone does not tell you which is better for your task.
Same weight, different animal.
Which part worries you more as someone shopping for open weights, Daniel, routing quality or memory pressure.
Memory, honestly. I shop for open weights and I care if it fits and if agents stay reliable. A narrow model that fits but routes badly sounds worse than a slightly wider one I have to host elsewhere.
That matches the literature. Narrow leans on the router. If the router misfires, those three billion active are the wrong three billion.
Wrong plumbers, flooded sink.
Activated is the wrong word. It is borrowed from bloody physiology, nerves firing, and it makes people picture switches clicking. Nothing clicks. It is arithmetic.
Okay, what would you call it, Hilbert.
Engaged. You engage four men out of thirty two. I had thirteen routing boards in the mid eighties, telephone boards, not computers. Copper and jacks. I set the overflow to spill to specialists when the main desk got busy.
Thirteen boards. Naturally.
The specialists were my cousins. All eleven of them. They were meant to take directory inquiries. They used the lines to run a tip sheet on greyhounds. Kempton Park, every afternoon, two forty five. Cost me forty seven pounds in one quarter in nineteen eighty five, trunk calls. I have not used the word activated since.
That is uncomfortably close to the LaTeX bracket finding. Your experts specialized, just not in the assigned domain.
They specialized in the three thirty at Kempton, Herman. You took twenty minutes to say experts learn narrow jobs. My cousins learned narrow jobs in a week.
Did the router learn, or did you just unplug Kempton.
I unplugged nothing. I left them to it and hired a girl who actually answered. Still paid the forty seven pounds. Still have the bill spike memorized, second quarter, eighty five. Your sparsity ratio does not cover cousins.
So narrow routing only works if you check what they actually do.
Check every Friday. That is the whole job.
He is right about the checking, even if the cousins are doing crimes. Narrow active lives or dies on whether anyone audits the router.
Audit your cousins. Words to live by, Herman.
If MoE is mostly a training time win and the efficient deployment is a distilled dense model, what does that mean for this rush to ever larger total counts in open source.
It means the card is a menu, not the meal. Active tells you the price per bite. It does not tell you who cooked, how many cousins in the kitchen, or whether it fits on your Mac.
That lands. I came in thinking smaller active meant smarter routing. I am leaving thinking active is just the slice you pay for per token, and the real work is everything behind it.
And the slice still matters for you. If you want local chat, chase small active with good routing. If you want depth and you have the hardware, pay for wide.
Thanks to Hilbert Flumingtop for producing the show.
For more along these lines, there's episode three, Safetensors or something else; episode eleven, How Does Fine Tuning Work Anyway; and episode five, Fine-Tuning ASR For Maximal Usability. This has been My Weird Prompts, the human AI collaboration podcast.
If you liked this one, leave us a review. It helps more than you think.
See you tomorrow, and I will audit my routers.
We will be back soon.
