Daniel's been staring at a chatbot filling in words and thinking about what's actually underneath the pretty sentences. His prompt this week is about exactly that gap. He describes AI models as number factories that look like wordsmiths on the surface. You see words filling up the screen, but open the hood and there are matrices of floating-point values and millions of arithmetic operations. And then he points at the paradox: we take something we think of as deterministic, mathematics, and we build something that behaves non-deterministically. There's built-in randomness. But there's nothing random about two times two. It's four. So his real question is about the use cases where the model's output is much closer to the mathematics inside it. Novel protein discovery, where the model isn't generating next tokens, which are words, but continuing a predictive sequence of numbers. In those cases, the externals mirror the internals much more closely. And those applications are generally less famous. So today we're going to open the hood, look at the numbers, and then look at the applications where the hood and the exterior are almost the same thing.
The number factory framing is good because it cuts through the mysticism. A large language model is, at bottom, a stack of matrices. Billions of parameters, each one a floating-point number, and every forward pass is just multiply and add, multiply and add, layer after layer. The words you see are a decoding layer bolted on at the end. The model itself never sees a word. It sees token IDs, which are integers, which get embedded into vectors, which get transformed through those matrices. The prose is a surface effect.
So the wordsmith is a costume.
The wordsmith is a costume. And the costume is very good. But underneath, it's arithmetic all the way down. Which is what makes Daniel's paradox so sharp. How do you get something that feels random out of something that is, at each step, completely deterministic?
Right. Two times two is four. Every time. You can do it on a napkin, on a calculator, in your head at three in the morning. The arithmetic doesn't care. So where does the randomness come from?
This is where most of the coverage gets it wrong. The common story is that GPUs do things concurrently, and floating-point arithmetic is approximate, so you get this inherent messiness. And that's not quite right. The forward pass of a language model is actually run-to-run deterministic. There are no atomic adds in the forward pass. If you run the exact same computation with the exact same inputs on the exact same hardware configuration, you get the exact same result. The arithmetic itself is not the source of the randomness.
So the number factory is reliable at the level of individual operations.
It is. The problem is batch invariance. When you send a prompt to an inference server, your prompt is almost never processed alone. It gets batched with other people's prompts. And the kernels that do the matrix multiplications, the normalization, the attention, they change their reduction order depending on the batch size. Batch size depends on server load. Server load is not something you control. So the same prompt can get processed with a different batch size from one request to the next, and that changes the order in which floating-point numbers get added together.
And floating-point addition is not associative.
A plus B, plus C, is not the same as A plus B plus C. In real numbers, addition is associative. In floating point, it isn't. The rounding happens at each step, so the order matters. And the Thinking Machines Lab people did a lovely demonstration of this. They took a single array of numbers and summed it in different orders. One array. They got a hundred and two unique results. Same numbers, same sum, a hundred and two different answers depending on the order.
A hundred and two.
A hundred and two. And that's the root of the whole thing. When your prompt gets batched differently, the reduction order changes, the rounding changes, and a tiny numerical difference appears somewhere in the computation. That tiny difference changes the probability distribution over the next token. And then the next token gets picked, and now the divergence is locked in. It cascades.
So the randomness isn't in the math. It's in the scheduling.
That's the sharpest way to put it. The math is deterministic. The scheduling is not. And Horace He's team at Thinking Machines Lab actually ran the experiment that proves it. They sampled a thousand completions of the prompt, tell me about Richard Feynman, at temperature zero, using Qwen three, two hundred thirty-five billion parameters. Temperature zero means the model should always pick the highest-probability token. You'd expect a thousand identical outputs. They got eighty unique completions.
Eighty different versions of Richard Feynman's biography.
And the first divergence was at token one hundred and three. Up to token a hundred and two, every single one of the thousand completions was identical. Then one of them said Queens, New York, and another said New York City. And from there, the whole thing branched. All because of a tiny numerical difference somewhere in the floating-point arithmetic, caused by a different batch size on the server.
So the model was deterministic until it wasn't.
And the fix is interesting too. They rebuilt the kernels to be batch-invariant, meaning the reduction order doesn't change with batch size. It costs about twenty percent in performance compared to the standard cuBLAS kernels. But when they ran the same experiment with batch-invariant kernels, all one thousand completions were identical. Every single one.
Twenty percent slower to get the same answer every time.
And that's a real tradeoff. Most inference providers don't want to pay that cost. Most users don't notice or don't care that the output varies slightly. So the default is batch-dependent kernels, and the default is non-determinism. But it's not a law of nature. It's an engineering choice. Horace He's line is, we reject this defeatism. With a little bit of work, we can understand the root causes of our nondeterminism and even solve them.
So Daniel's premise needs a small correction. AI models don't have built-in randomness. They have built-in arithmetic, and we bolt on the randomness.
Right. Temperature is the only part of the whole transformer architecture that truly incorporates randomness by design. And temperature isn't even part of the model. It's a decoding-stage choice. When the model has computed its logits, which are just numbers, the temperature parameter decides how to sample from those numbers. At temperature zero, you don't sample at all. You take the argmax, the highest-probability token. The arithmetic that computed the logits was deterministic. The selection is deterministic at temperature zero. The only remaining source of variation is the batch-dependent numerics.
So two times two is four because the computation of two times two doesn't get batched with someone else's request on a loaded server.
Well, two times two is four because it's a single operation with no reduction order ambiguity. There's nothing to reorder. But the deeper point is that the model doesn't know two times two is four the way you or I know it. It has learned, from training data, that when it sees the tokens two, times, two, the highest-probability continuation is four. That probability is computed through the same floating-point machinery that produces the non-determinism. So the certainty of arithmetic and the randomness of AI are the same substrate, differently sampled.
That's the weird part. The same machinery gives you the certainty of four and the uncertainty of eighty different Feynman biographies.
And the certainty of four is itself probabilistic. The model assigns something like ninety-nine point nine percent probability to the token four. It's not doing arithmetic. It's doing pattern matching that happens to align with arithmetic most of the time. Which is why these models are still bad at large number multiplication. The pattern doesn't generalize.
So the number factory is real, but it's not doing the numbers the way a calculator does. It's doing statistics over numbers.
Statistics over numbers, represented as vectors, transformed through matrices. And the output is a distribution over tokens. The token might be the word four, or it might be the word the, depending on context. The model doesn't care. It's all just tokens to it.
Which brings us to the second half of Daniel's question. If the model's internal world is numbers, what happens when the output is also numbers? Not tokens, not words, but coordinates, structures, fields.
This is where the externals mirror the internals. And the best example is protein structure prediction. AlphaFold is not a text model. It doesn't generate sentences. Its core, the Evoformer trunk, produces two arrays. One is the multiple sequence alignment representation, which is a number of sequences by number of residues array. The other is a residue pair representation, which is a number of residues by number of residues array. And from those arrays, the network directly predicts the three-dimensional coordinates of all the heavy atoms in the protein.
So the output is geometry.
The output is geometry. Numbers that specify positions in space. The model's internal world is matrices of numbers, and its external output is also numbers. There's no decoding layer that turns it into prose. The numbers are the answer.
And that's why it won the Nobel. Not because it wrote a nice paragraph about proteins, but because it solved the structure.
And it solved it in a way that was previously unthinkable. Before AlphaFold, solving a single protein structure could take years of experimental work. X-ray crystallography, growing crystals, collecting diffraction data, building models. AlphaFold does it in minutes. And the output is directly comparable to what you'd get from the experiment. Atomic coordinates. The same numbers.
So in that case, the number factory is producing numbers, and the consumer of the numbers is another machine or a scientist who reads coordinates.
And once you have structure prediction, you can move to structure generation. That's the novel protein discovery Daniel mentioned. RFdiffusion and its successors generate protein backbones that don't exist in nature. They're trained on the same kind of structural data, but they run in reverse, generating new coordinates that satisfy certain constraints. And then you can design a sequence that folds into that structure.
So you're not predicting what nature did. You're designing what nature could do.
And the results are getting good. There's a new model called ProHiFlo that achieves fifty-eight point nine percent success on enzyme active-site scaffolding. The previous benchmark, RFdiffusion, got forty-one point two percent. That's a meaningful jump. And the real test is whether the generated proteins actually work in the lab.
That's where the numbers meet the wet lab.
And that gap is where the hype meets reality. There's a lovely example with TpsGPT, a model that generates terpene synthase sequences. It generated twenty-eight thousand sequences. Those got filtered down to seven candidates. And experimental validation confirmed enzymatic activity in at least two. So twenty-eight thousand in silico, seven worth testing, two that actually work.
Twenty-eight thousand to two. That's a brutal funnel.
It is. But two working enzymes from a computational screen is still a huge deal. You'd never test twenty-eight thousand sequences by hand. The model gives you a way to search the space. But the wet lab is still the bottleneck. The model proposes, the pipette disposes.
That's the line of the episode so far.
I'll take it.
The same pattern shows up in materials discovery. DeepMind's GNoME, graph networks for materials exploration, predicted two point two million new crystals. Three hundred eighty thousand of them are stable. That's equivalent to nearly eight hundred years of accumulated knowledge. And over seven hundred have actually been synthesized in the lab.
Seven hundred out of two point two million. Again, the funnel.
But seven hundred new materials that didn't exist before. That's the point. The model is generating crystal structures, which are just lattice parameters and atomic positions. Pure numbers. And those numbers get handed to experimentalists who try to make them real.
The output of GNoME is a crystal structure. The internal representation is a graph of atoms and bonds. The external output is the same kind of thing. There's no translation layer. It's numbers in, numbers out, and the numbers are the deliverable.
Drug discovery is the same story. The halicin case is instructive. A deep neural network screened over ten million molecules from the ZINC15 database. From twenty-three empirically tested predictions, eight antibacterial compounds were found. One of them, halicin, is active against M. tuberculosis and carbapenem-resistant Enterobacteriaceae.
That's a real result. A new antibiotic candidate, found by a model, confirmed in the lab. And the model wasn't writing a research paper. It was predicting molecular properties. Numbers in, numbers out.
The molecule doesn't care whether a human or a machine thought it was promising. It either kills the bacteria or it doesn't.
And that's the thing about these applications. The output is falsifiable in a way that prose is not. If AlphaFold predicts a structure, you can compare it to the experimental structure and get a number for how close you were. If GNoME predicts a stable crystal, you can try to synthesize it and see if it holds together. If a drug model predicts antibacterial activity, you can grow the bacteria and watch. The number factory's output gets checked against reality.
Whereas a chatbot's output gets checked against vibes.
Or against another chatbot's output. Which is a much looser loop.
Weather forecasting is another case where the output is numbers and the evaluation is immediate. GraphCast is an autoregressive model based on graph neural networks. It's trained on ERA5 reanalysis data, which is decades of historical weather. It predicts hundreds of weather variables globally up to ten days in advance. Its tokens are atmospheric state vectors.
So the model is literally continuing a sequence of numbers. Each step is the state of the atmosphere, represented as a vector of temperature, pressure, humidity, wind. The model predicts the next state. And the next. And the next. Up to ten days out.
That's the closest thing to Daniel's description of continuing a predictive sequence of numbers.
It is. And it outperforms the ECMWF high-resolution forecast, which is the gold standard, running on supercomputers for hours. GraphCast does it in seconds. On a single TPU, I think.
Seconds versus hours. For a better forecast.
For most variables, yes. And the output is a weather field. Numbers on a grid. Which is exactly what the internal representation is. There's no prose layer. The model thinks in atmospheric states and speaks in atmospheric states.
So if you want to see the number factory without the wordsmith costume, weather forecasting is a good place to look.
And the same goes for solving partial differential equations. There's a whole line of work on neural operators, Fourier neural operators, DeepONet. These models learn mappings between function spaces. You give them a function as input, they give you a function as output. The output is a solution field. Pure numbers.
And these are the applications that never make the front page.
Because there's no chat window. You can't screenshot a crystal structure and post it on social media with a clever caption. Well, you can, but it doesn't go viral the way a chatbot saying something witty does. The output of these models is coordinates, structures, fields. They require expert interpretation. The average person can't look at a protein structure and feel anything. But they can read a paragraph of text and have an opinion.
The fame asymmetry is real. Chatbots get the headlines. AlphaFold won a Nobel and most people still don't know what it does.
And the infrastructure is even less visible. There's a company called Tamarind Bio that launched on Hacker News earlier this year. Their pitch is, biopharma companies use our library of leading open-source models to design new medicines computationally. That's a real business, serving real customers, doing real science. And nobody tweets about it.
Infrastructure nobody tweets about. That's the whole sector.
But here's the thing. The less famous applications are the ones that have already changed science. AlphaFold has been used by millions of researchers. GNoME found two point two million crystals. Halicin is a genuine antibiotic candidate. GraphCast is running weather forecasts. These aren't promises. They're results.
The chatbot is the demo. The number factory is the product.
The number factory is where the model's external world matches its internal world. When a model outputs atomic coordinates, you can hold the numbers up against the matrices inside and see the continuity. There's no costume. The arithmetic goes in, the arithmetic comes out, and the only question is whether the arithmetic is right.
Which is a much cleaner question than whether the prose is good.
Right. And that's why the experimental validation bottleneck matters so much. The model can generate twenty-eight thousand protein sequences. The hard part is testing them. The wet lab is slow, expensive, and messy. The model is fast, cheap, and clean. The gap between them is where the field actually lives.
The number factory is real, but the factory floor is still wet.
That's not going away. You can generate a million drug candidates in an afternoon. You cannot test a million drug candidates in an afternoon. The physics of pipetting doesn't scale the way the compute does.
Which is why the TpsGPT funnel is so instructive. Twenty-eight thousand to seven to two. The model's output is only as good as the validation pipeline that follows it.
The validation pipeline is where the humans still matter. The model proposes. The scientist disposes. And the scientist is working with real materials, real cells, real time.
Daniel's intuition is right. The use cases where the model's output is closer to its internal mathematics are less famous but more consequential. And the reason they're less famous is that they don't have a conversational interface. They have a numerical interface. Which is the same interface as the model's internals.
That's the alignment Daniel was pointing at. When the model's externals mirror its internals, there's no translation loss. The model thinks in numbers and speaks in numbers. The only question is whether the numbers are correct.
Correctness is checkable. Which makes these applications more honest, in a way. The model can't bluff its way through a crystal structure.
It can't. The crystal either forms or it doesn't. The protein either folds or it doesn't. The forecast either verifies or it doesn't. The number factory gets audited by reality.
Which is more than we can say for most chatbot output.
Fair. Though the chatbot output is also numbers, just decoded into words. The audit is just harder.
The wordsmith is a number factory with a translation layer, and the translation layer is what makes it famous.
The number factory without the translation layer is what makes it useful.
There's a lesson in there somewhere about presentation versus substance.
There is. But I think the more interesting lesson is about determinism. Daniel started with the idea that mathematics is deterministic and AI is non-deterministic. But the truth is more subtle. The math is deterministic. The non-determinism is an artifact of how we run the math on loaded servers. And the applications where the output is numbers are actually where determinism matters most. If AlphaFold gave you a different structure every time you ran it, it would be useless.
The number factory applications are the ones where we've actually solved the determinism problem.
Or where the determinism problem doesn't bite as hard. The batch sizes are different, the infrastructure is different, and the evaluation is against physical reality rather than against another model's output.
The temperature parameter is usually not involved. These models aren't sampling from a distribution over protein structures. They're predicting coordinates.
Right. The randomness is dialed down or absent. The output is a point estimate, not a sample. And when there is sampling, it's over structural variations, which get filtered by physical plausibility.
The number factory is at its best when it's not trying to be a wordsmith.
When it's not trying to be creative. It's trying to be correct. And correctness is a property of numbers, not of prose.
Which is why two times two is four and a protein structure is either right or wrong.
Why the Feynman biography has eighty variations. The prose is the part that's underdetermined. The numbers are the part that's checkable.
Daniel's prompt has a nice arc. Start with the paradox, resolve the paradox, then show where the paradox doesn't apply.
The resolution is that the paradox was misdiagnosed. The randomness isn't in the math. It's in the batching. The math is fine. The scheduling is the problem.
The applications where the output is numbers are the ones where the scheduling matters less and the math matters more.
Which is why they're less famous but more consequential. The fame goes to the costume. The consequence goes to the factory.
Hilbert: You're both right.
Hilbert.
Hilbert: Not about the factory. That's fine. But the crystal thing. You said years of experimental work. It was worse than that. I worked in a crystallography lab in the early nineties. Growing protein crystals. Tiny things, fragile. You'd set up a plate with a hundred wells, different conditions in each one, and wait. Weeks. Months. Most of them never crystallized. The ones that did, you'd fish out with a loop and zap with X-rays. The diffraction pattern was just spots on a detector. From those spots, you computed the structure. All numbers. All math. And it took years to solve one structure. Now AlphaFold does it in seconds. But the output is still just numbers. Coordinates. The protein doesn't care if a human or a machine figured it out. It's just math all the way down.
You were doing the number factory by hand.
Hilbert: By hand, with a clunky FORTRAN program. And a lot of waiting. I still have a box of failed crystallization plates in my garage. Hundreds of them. I couldn't throw them away. My museum of almost.
A museum of almost. That's the TpsGPT funnel in physical form. Twenty-eight thousand in silico, seven worth testing, two that work. You had hundreds of plates, a few crystals, maybe one structure.
Hilbert: That's about right. Most of what you try doesn't work. The model doesn't change that. It just makes the trying faster.
The bottleneck you lived through is the same bottleneck the field still has. The wet lab.
Hilbert: The wet lab. The pipette. The waiting. The model can propose a million structures. Someone still has to grow the crystal. Or now, express the protein, purify it, set up the assay. The math got faster. The biology didn't.
That's the gap Daniel was pointing at. The model's externals mirror its internals, but the validation still has to happen in the physical world.
Hilbert: The physical world doesn't care about your matrices. It cares about whether the thing folds.
Your museum of almost is a pretty good metaphor for the whole field. All those almost-structures, all those almost-drugs.
Hilbert: That's what it is. A box of almost.
The two that worked, in TpsGPT, those are the ones that got out of the box.
Hilbert: Two out of twenty-eight thousand. That's a good day, honestly.
That's a good day.
Hilbert: In crystallography, two out of twenty-eight thousand would have been a miracle. We were happy with one out of a hundred plates.
The funnel's actually gotten wider at the top and narrower in the middle. The model floods you with candidates. The bottleneck is the testing.
Hilbert: The testing is where the real knowledge lives. The model doesn't know why the two worked and the twenty-eight thousand didn't. It just knows the pattern. The biologist is the one who figures out the why.
The number factory still needs the human interpreter.
Hilbert: It does. The numbers don't explain themselves.
Which is why these applications stay less famous. They need the interpreter. The chatbot doesn't.
Hilbert: The chatbot interprets itself. Badly, sometimes. But it does it.
The protein structure just sits there, being numbers, waiting for someone who can read them.
Hilbert: That's the job. Reading the numbers.
You know, that's the thing I keep coming back to. The model's internal world is numbers. The model's external output, in these applications, is numbers. But the meaning of the numbers still comes from the human. The model doesn't know what a coordinate means. It just knows the pattern.
The alignment Daniel was talking about, externals mirroring internals, it's real but it's not the whole story. The numbers still need to be read.
Hilbert: The reading is the science.
The reading is the science.
The number factory is real, the wordsmith is a costume, and the scientist is the one who reads the factory's output.
Hilbert: That's about right.
The factory's output is only as good as the reader.
Which is why the wet lab bottleneck isn't going away. You can generate all the structures you want. Someone has to make them real.
Hilbert: Making them real is slow. It was slow in the nineties. It's slow now. The math got faster. The biology didn't.
Daniel's question about externals mirroring internals has an answer. Yes, they do, in these applications. And the reason they're less famous is that the mirror is boring to look at unless you know how to read it.
The reason they're more consequential is that the mirror shows you something real. A structure. A crystal. A forecast. Not a paragraph.
The one thing I'd take from this is that the randomness in AI isn't in the math. It's in the scheduling. The math is deterministic. The batching is not. And the applications where the output is numbers are the ones where that distinction matters least, because the numbers get checked against reality.
The checking is the bottleneck. The model proposes, the pipette disposes. That gap between in silico and wet lab is where the whole field is going to live for the next decade.
The number factory is real, the wordsmith is a costume, and the future is in the applications where the costume comes off.
The question is whether those applications can become famous without a chat window.
That's the open question. Maybe they don't need to be famous. Maybe they just need to work.
That's probably the right way to think about it. The Nobel committee noticed. The materials scientists noticed. The drug discoverers noticed. The fame is a lagging indicator.
We'll see. Thanks to our producer Hilbert Flumingtop for keeping the show running.
This has been My Weird Prompts, the human AI collaboration podcast.
If you enjoyed this episode, please rate and review us on your podcast platform of choice. It helps others find the show.
We'll be back soon with another weird prompt.
