Here's what Daniel sent in this week. He says the idea that we can't fully understand large language models, and that unexpected capabilities just emerge over time, is a fascinating one. And he wants us to go find more examples.
More examples of what, exactly?
Of properties that were scientifically validated, or at least seriously proposed and still debated. Things nobody saw coming. And he's tying it back to that linguistics episode we did, where the question was whether models follow grammatical rules they were never formally taught. He wants to know how that debate actually resolved. What surprised the people who built these systems. And what's becoming more apparent as the models scale.
That's a big ask. That's like four episodes.
It's one episode, and he asked nicely. So let's start by pinning down what we even mean when we say a capability is emergent.
The technical definition is narrower than the colloquial one. It comes out of the BIG-Bench work from 2022. An ability is emergent if it is not present in smaller models but is present in larger models. That's it. The key clause is the second half, because it means you could not have predicted it by extrapolating the performance of the smaller models you already had.
So it's a statement about our ignorance as much as about the model.
Exactly that. There are two features that make it striking. The first is sharpness. The ability seems to go from absent to present almost instantaneously as you scale. The second is unpredictability. It shows up at scales nobody could have forecast from the curve.
And there's a distinction from physics that's worth dragging in here, because it gets abused a lot.
Weak versus strong emergence. Weak emergence is a system-level property that follows from lower-level rules but is hard to predict from them. Strong emergence is irreducible, where the higher level can't be derived from the lower level even in principle. Almost everything in machine learning that gets called emergence is weak emergence.
Which is a much less mystical claim.
Much less mystical. And even the weak version is contested. That's the fight we're going to have for the next twenty minutes. Is emergence a property of the model, or a property of how we measure it?
So the definitional groundwork is done. Now let's get into the fight over whether any of this is real.
The single most important paper here is Schaeffer, Miranda and Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?" It came out in 2023 and landed at NeurIPS. Their argument is that for a fixed model family and fixed outputs, apparent emergence is an artifact of the researcher's choice of metric. Not a fundamental change in what the model is doing.
Walk me through the mechanism, because that's the part that actually bites.
Take exact-match accuracy. A model either gets the answer fully right or it gets zero. That metric is discontinuous. It sits at zero, zero, zero, and then at some scale it jumps. Now take token-level log-likelihood, which is continuous. The same model, on the same outputs, shows smooth, predictable improvement all the way up. Same behavior, different yardstick, completely different story about whether something emerged.
So the jump is in the ruler, not the thing being measured.
That's their claim. And they didn't just assert it. They confirmed it three ways. They made predictions on the InstructGPT and GPT-3 family. They did a meta-analysis of the BIG-Bench emergence claims and showed most of them were metric artifacts. And then the provocative one. They manufactured never-before-seen emergent abilities in vision tasks simply by choosing the right metric.
They invented emergence.
They demonstrated that you can produce the appearance of it on demand, in a domain where nobody was claiming anything mystical was happening. That's the strongest version of the argument.
Okay. So what's the defense?
The best rebuttal is the loss perspective. Du, Zeng, Dong and others, "Understanding Emergent Abilities of Language Models from the Loss Perspective," NeurIPS 2024. Their finding is that models with the same pre-training loss show the same downstream performance, regardless of how you split parameters versus data.
Two models can have wildly different sizes and wildly different training sets. If their pre-training loss is the same, their downstream task performance is the same. Emergent abilities appear when pre-training loss falls below a task-specific threshold. Before that threshold, performance is at random-guessing level.
So the variable isn't size at all.
It's loss. And here's the part that rescues the original definition. They argue this still cannot be predicted by extrapolating the performance trends of models with higher pre-training loss. You can't look at the weak models and see it coming, because the thing that predicts it is a quantity you weren't tracking.
That's a different claim from the mirage argument. Schaeffer says the jump is in the metric. Du says the jump is real, it's just indexed to loss instead of parameters.
And both can be true. They're answering different questions.
There are other reconciliations worth putting on the table.
Wu and Lo, ICLR 2025, on U-shaped scaling. Group questions by difficulty. Hard questions show U-shaped scaling, meaning performance gets worse before it gets better. Easy questions show an inverted U. The two curves initially offset each other, which produces apparent stagnation. Then the easy-question curve reverts, and overall performance soars.
So the aggregate metric was hiding two opposing trends underneath it.
And they propose a pipeline they call Slice-and-Sandwich to predict the emergence threshold. That's the thing to watch. Prediction is the whole ballgame.
Then there's the infinite-resolution argument.
Hu and others. The claim is that small models do improve smoothly. Our conventional metrics just lack resolution to see it. So they used massive sampling, a method they call PassUntil, and identified a strict task scaling law. Then they predicted a 2.4 billion parameter model's code-generation performance within 0.05 percent deviation. Before training began.
Half a percent of one percent.
Zero point zero five percent. They called the shot before the model existed and were right to that precision.
That's the kind of result that should make the mirage people nervous.
It should make everyone nervous, in the good way. There's also a circuits-level account from Huang and others that unifies grokking, double descent and emergence as a single phenomenon. Competition between memorization circuits and generalization circuits inside the network.
Same underlying process, three different names depending on how you were measuring.
That's the pitch, yes.
Where does the field sit as of now?
The Berti, Giorgi and Kasneci survey is the best current overview. Revised August 2026. It says plainly that the field still has inconsistent definitions. Emergence depends on scaling laws, task complexity, pre-training loss, quantization, and prompting. And it extends the whole discussion to large reasoning models, the RL-trained, inference-time-compute-heavy systems.
So the question migrated rather than settled.
It migrated. And there's a practitioner skepticism worth carrying, because it comes from people actually training these things. The argument is that a lot of supposed emergent abilities dissolve once you inspect the training data. Instruction-following looks magical until you remember the model was trained on a massive instruction-following corpus. Chain-of-thought looks magical until you remember it was trained on step-by-step reasoning.
That's not a refutation. That's a reminder that we built the thing that does the thing.
It's a reminder that "emergent" was always a statement about our surprise, not about the model's magic. And one more honest note from that crowd. Capability gains look roughly linear. What was surprising was the visibility of progress, not the rate of it.
The mirage debate is about metrics. But there's a whole other thread here that goes back to linguistics, and that's where the consensus is actually clearer.
This is the thread from our earlier conversation, and the peer-reviewed literature is unusually clean on it. Mahowald, Ivanova, Blank and others, "Dissociating language and thought in large language models," in Trends in Cognitive Sciences, 2024. They split the problem in two.
Formal versus functional competence.
Formal linguistic competence is knowledge of linguistic rules and patterns. Syntax, morphology. Functional linguistic competence is using language in the world. Reasoning, world knowledge, social understanding.
And the finding?
LLMs are surprisingly good at formal competence. They master hierarchical syntax and dependency relations without any explicit grammatical instruction. But their functional competence remains spotty and often requires specialized fine-tuning or coupling with external modules.
So the consensus is that they do learn grammar they were never taught.
That part of the debate has largely settled. The live question is whether that competence is functionally meaningful, and whether the internal representations are the actual causal mechanism.
That second question is the one people skip.
It's the one that matters. There's work on hierarchical structure in latent space. H-Probes, 2026, extract depth and pairwise distance from the model's internal representations. The hierarchy-containing subspaces turn out to be low-dimensional, causally important, and they generalize out of domain. And they show up not just at the syntax level but inside the reasoning process itself.
That sounds like a clean win for the representation side.
It would be, except for the disconnect. Agarwal, Jian, Manning and others tested thirty-two open-weight transformers and found that syntactic features found via probing fail to predict downstream syntactic performance. A substantial disconnect between what's in the latent space and what the model actually does.
So the structure is there, but it might not be what's driving the output.
That's the uncomfortable reading. Finding a representation is not the same as finding a mechanism. You can find a pattern in the activations that correlates with the behavior without it being the thing that causes the behavior. Probing is a correlational tool.
And the brain-alignment work?
Real, but diverging. HFTP, 2025, finds that LLMs process syntax in analogous layers and align more with the left hemisphere, which is the language-dominant one in humans. But newer models diverge. Gemma 2 is more brain-like than Gemma. Llama 3.1 is less brain-like than Llama 2.
So capability is going up while human-likeness goes down.
In at least one case, yes. Which raises the question of whether we're building something that thinks like us or something that thinks in a way we don't have vocabulary for.
There's a theoretical reconciliation worth mentioning. Murphy, 2024, argues that artificial language models will contribute to the neuroscience of horizontal morphosyntax. The linear, predictive stuff. But much less to hierarchically compositional structure. A hybrid neurosymbolic view.
That's a deflationary reading, and it's probably closer to right than either extreme.
If the linguistics debate is largely settled on formal competence, what about the capabilities nobody saw coming at all?
This is where it gets fun. The most striking class is internal world models. Othello-GPT is the canonical case. A model trained only on move sequences from the game developed a linear representation of the board state. Nobody trained it to track the board. It was trained to predict the next move.
And the board state fell out.
It fell out, and it was linearly decodable. There's a 2025 paper, "Linear Spatial World Models Emerge in Large Language Models," that extends this. LLMs implicitly encode linear representations of physical space and object configurations. Decodable via probes, geometrically consistent, and causally used by the model, which they verified by intervention.
Intervention meaning they reached in and changed it.
They altered the representation and the behavior changed accordingly. That's the difference between a correlation and a mechanism. They demonstrated the causal link.
Grokking is the other one.
Grokking is delayed generalization. A model memorizes a task, sits there performing at chance on held-out data, and then after extended training suddenly transitions to perfect generalization. It's a phase change that is completely invisible in the training loss. The loss curve looks smooth the whole time.
And the reasoning result?
Wang, Yue and Su, NeurIPS 2024. They show transformers learn implicit reasoning only through grokking. And a fully grokked transformer achieves near-perfect accuracy on a hard reasoning task where GPT-4-Turbo and Gemini-1.5-Pro fail badly. Regardless of prompting or retrieval.
A small model that grokked beats the frontier models.
On that specific task, yes. Which tells you the capability isn't about scale at all. It's about whether the training process found the generalizing circuit or the memorizing one.
And there's recent nuance on grokking.
Two pieces. One argues grokking is a spectral recoding of an existing distributed circuit rather than a module switch. So the circuit was already there and got reorganized. The other reports an anti-grokking phase. Late-stage collapse of generalization back to chance while training accuracy stays perfect.
So the model can unlearn the general solution and keep the memorized one.
And you'd never see it in the training loss. That's the part that should unsettle anyone building on these systems.
What about the affective work?
Romeo and Testolin, Royal Society Open Science, 2025. Multimodal models' emotional ratings of images correlate surprisingly well with average human ratings, despite never being explicitly trained to match human affective reactions.
Emotion concepts emerging from statistical learning over image-language pairs.
That's the suggestion. And it's a strange one, because emotion is exactly the kind of thing you'd assume requires embodiment or at least a nervous system.
There's also the in-context learning result, which complicates the size story.
Muckatira and others, NAACL 2024. They trained thirty-six models from one million to one hundred sixty-five million parameters on simplified language. Those models produced zero-shot capabilities comparable to models six times larger trained on unrestricted language.
So it's not scale. It's data complexity.
At least partly. Emergence is tied to what the model was trained on, not just how big it is.
Those are the validated surprises. What do the people who actually built these systems say about them?
The honest answer is that they're surprised too, and they say so. The recurring theme is what I'd call the next-token-predictor paradox. To predict the next token well, the model had to build internal representations of the world. What is a giraffe. What's the board state. Where are the objects in the scene.
It wasn't asked to model the world. It was asked to guess the next word.
And the world model fell out as a side effect. One practitioner put it well. Bread is not just yeasty wheat. In the task of training a next-word guesser, there are emergent capabilities that go beyond what people expect when they look at the method.
I like that. You put in flour and water, you get something with structure.
You get something with structure you didn't put in. And then the honest counterpoint, which is well-upvoted and worth taking seriously. We don't understand emergence. We don't understand how optimizing one task leads to emergent ability on another task. Even the existence of emergent ability is under debate.
So the builders are the first to say they don't have the theory.
They're the first to say it. That's the state of the science. We have a pile of empirical results and no theory that covers them.
As models grow, what's becoming more apparent?
Reasoning models are the new frontier. RL-trained, inference-time-compute-heavy. The 2026 survey treats them as a distinct category, which means the emergence question isn't settled, it's migrating.
And predictability is improving.
Multiple papers from 2024 and 2025 claim emergence thresholds can now be forecast. Proxy tasks, infinite-resolution evaluation, U-shaped scaling. If that holds, it undercuts the unforeseeable half of the original definition. An ability you can predict isn't emergent in the 2022 sense. It's just an ability you hadn't measured yet.
That's a real shift. The definition might get eaten by its own success.
It might. And then there's the capability-versus-reliability gap. Models can track code state over two hundred fifty thousand tokens and still fail at chess or Sudoku piece-tracking. Capability is domain-specific and training-regime-specific. It's not a general emergent tide lifting everything.
And the divergence from human cognition.
As models improve, brain alignment is not monotonically increasing. Llama 3.1 is less brain-like than Llama 2. Scaling may be producing non-human-like mechanisms. Which is the strangest result in the whole pile, because we assumed getting better meant getting more like us.
We assumed the ladder was ours.
And it might not be.
Hilbert: The crumb tray was the tell. Nobody ever cleaned the crumb tray.
Sorry, the what?
Hilbert: The crumb tray. On the deck ovens. I did six months as a quality-control technician at a plant that made industrial baking ovens. My job was to watch the internal temperature profiles. Thermocouples in the deck, chart recorder running, I'd log the deviation from spec every half hour.
And the crumb tray?
Hilbert: The ovens were controlled by a PID loop. Simple thing. Proportional, integral, derivative. And every so often an oven would turn out a better crust on the bread than the profile specified. The engineers called it oven personality.
Oven personality.
Hilbert: They'd shrug and say it had personality. I never got a straight answer. I asked the controls guy and he said the loop was doing what it was told. I asked the baker and he said the oven was just like that. Six months and nobody would tell me why one deck browned better than the one above it.
So what was it?
Hilbert: The loop was doing exactly what it was told. The personality was the interaction between the control loop and the thermal mass of the oven. The brick, the steel, how much heat was stored in the deck. None of that was in the model. The model was air temperature. The bread was getting what the brick had to give it.
So the behavior was real, but it was in the system, not in the controller.
Hilbert: That's what I've been listening to for the last twenty minutes. You're arguing about whether the jump is in the model or in the ruler. The oven didn't emerge anything. The loop did what it was told and the result came out of the loop and the oven together. I think your language models are the same. Not a new capability. An interaction between the training objective and the data that nobody bothered to model.
So you're saying the surprise is in the system.
Hilbert: I'm saying it's more interesting that way. If the model did it, you'd have to explain how. If the system did it, you just have to admit you didn't model the oven.
Did you ever figure out the crumb tray?
Hilbert: The crumb tray was a separate thing. It sat under the deck and caught the ash. If you didn't clean it, the airflow changed and the bottom deck ran hot. That was in the manual. Nobody read the manual.
Of course.
Hilbert: I've got a thing at four.
The panel on causal representation was making a narrower claim than it sounded like. They showed the representation is used, not that it's sufficient.
Which is the same thing Hilbert just said about the oven. The thing you measured isn't the whole system.
That's the honest position to leave this on. We have validated surprises. Othello-GPT building a board it was never asked to build. Grokking, where a small model beats the frontier on a task it learned to generalize on. Emotional ratings that track human averages without anyone training for it. Those are real.
We have a pile of papers arguing about whether the jump is in the model or in the ruler.
We have a theory gap. We don't have an account of why optimizing one task produces capability on another. That's the thing nobody has.
If emergence thresholds can now be predicted, does that change what we even mean by emergent? An ability you can forecast isn't a surprise. It's a measurement you hadn't taken yet.
If brain alignment is diverging as models improve, then we're building something that thinks in a way we don't have a word for. That's the open question I'd leave people with. Not whether it's real. What it is.
That's the show. Thanks to Hilbert Flumingtop, our producer. This has been My Weird Prompts, the human-AI collaboration podcast. If you want more of this, leave us a review wherever you're listening. It helps.
We'll be back soon.
