Daniel's been staring at vendor spec sheets again, and he's hit the wall that every business user eventually hits. Some model boasts a one-million-token context window, and his honest reaction is — I don't know what that means for my work. He's not confused about tokenisation. His problem is that tokens have become this weird currency where a coffee costs two thousand units. Technically correct, not legible to anyone who doesn't live inside the machinery. So he's asking three things. One, are we heading toward a way of stating AI capability that's derived from tokens but legible to ordinary people — and what would that unit actually look like? Two, has anyone seriously attempted this, and what are the candidate answers? Three, why hasn't it happened yet — is it marketing inertia, or is there a genuine reason the honest unit resists translation?
That's the right set of questions. And the fact that he's framing it as a measurement-and-communication problem rather than a tokenisation explainer — that's exactly where the interesting part lives.
So let's start by unpacking why the token, as a unit, fails at the job we're asking it to do — not because it's inaccurate, but because it's illegible.
Right. A token is the fundamental unit of text that a model processes — roughly a word part. And it's honest. It's what the model actually consumes. The problem isn't dishonesty, it's that token counts tell you the size of the input the model can hold, and absolutely nothing about what it can do with that input.
It's like describing a truck by the volume of its fuel tank. It's a real number, it matters, but you still don't know what it can haul or how far it can go.
And the bigger the numbers get, the less informative they become. A million tokens — roughly seven hundred and fifty thousand words, depending on the language and the tokeniser. That sounds enormous. But what does it actually mean for someone who needs to know whether this model can review a five-hundred-page legal contract and catch the liability clause buried on page three hundred and twelve?
The conversion to words is the thing Daniel specifically said he doesn't want, and he's right. "Seven hundred and fifty thousand words" — great, now I know how many words it can hold. I still don't know if it can reason across them.
And this is where the needle-in-a-haystack benchmark comes in, because it's the perfect illustration of the gap. That benchmark tests whether a model can find a specific piece of information placed somewhere in a long document. Every frontier model now hits ninety-nine percent on it. It's saturated. But finding a fact is not the same as reasoning across a document. A model can locate the sentence "the contract terminates after sixty days" perfectly, and still fail to connect it to the clause on page forty-seven that modifies the termination conditions.
So you've got a benchmark that's maxed out, giving the impression that long-context problems are solved, while the actual reasoning capability lags far behind. The token count is being used as a proxy for something it doesn't measure.
And it's being used that way across models that don't even tokenise the same way. Different models use different tokenisers. The same sentence might be thirty tokens in one model and forty-five in another. Code tokenises differently from prose. Languages tokenise wildly differently — a model that handles English efficiently might chew through three times as many tokens for the same content in Japanese or Arabic.
So "one million tokens" doesn't even mean the same amount of text across models. It's a unit that varies by what you're measuring.
It's a mess.
And yet it's the unit every vendor puts on the box.
Because it's the one number they can all agree on how to count. Everything else requires judgment.
Which brings us to the core tension Daniel's really poking at. Vendors love big numbers because they're impressive and easy to compare — "our context window is twice as large as theirs." But the thing the buyer actually wants to know — "can this handle my workload?" — isn't answered by that number at all.
Let me put it in clinical terms. You can tell a patient their white blood cell count is twelve thousand, and that's a real number, it's honest, it was measured precisely. But it doesn't tell them whether they have an infection or leukaemia or just went for a run. The number is input to a diagnostic process, not the diagnosis itself.
That's good. And the vendors are selling the lab value as if it were the diagnosis.
Yes. And Google's own materials are instructive, because they're trying to have it both ways. Their Cloud team published a piece that frames long-context windows in terms of business use cases — analysing entire codebases, processing legal documents, handling multi-year financial records. They're selling the capability, but they're still quoting the token count. "Our model has a one-million-token context window, which means it can process an hour of video or a seven-hundred-thousand-word novel." They're reaching for the human unit, but they can't quite let go of the token.
So what would a better unit actually look like? Daniel asked for candidate answers, and specifically for real proposals with names and origins where they exist.
There are a few serious attempts, and they fall into roughly three buckets. The first is what you might call "effective context" — trying to measure not how much text the model can hold, but how much relevant information it can actually use. This shows up in research papers that test whether model performance degrades as you add irrelevant filler. The idea is: I give you a question, I embed the answer somewhere in a document, and then I pad the document with unrelated text. How much padding can I add before you stop getting the right answer?
That's the broader family that needle-in-a-haystack belongs to. And the problem is, as we said, that specific test is saturated. But researchers have been developing harder variants — multi-needle tests where you have to find and synthesise several pieces of information, or reasoning tests where the answer depends on connecting facts from different parts of the document. These are attempts to measure effective context rather than raw context.
So the unit there isn't a number of tokens, it's something like "can synthesise information across X pages" or "maintains reasoning coherence up to Y thousand words of relevant content." Task-based.
And that's the second bucket — task-based benchmarks. Instead of stating a token count, you state what the model can do. "Handles a hundred-page contract review." "Processes a full day of meeting transcripts and produces accurate summaries." "Can analyse an entire codebase and identify security vulnerabilities across files." These are legible. A business user understands what a hundred-page contract review means.
The downside being that "a hundred-page contract review" isn't a standard unit. One vendor's claim about contract review might mean something completely different from another's.
Right. And that's the third bucket — the attempt to create standardised capability scores. Something like an F1 score for long-context tasks, where you've got a standardised test suite and every model gets a number that means the same thing regardless of who built it. There have been academic proposals — the Long Range Arena benchmark, the SCROLLS benchmark, and more recently the RULER benchmark which specifically tests effective context length across multiple task types.
RULER being?
Retrieval, Understanding, and Long-context Evaluation benchmark. It was proposed around late twenty twenty-three. The key insight was that you need to test multiple dimensions: retrieval, multi-hop reasoning, aggregation, question answering. A model might be great at finding a single fact and terrible at synthesising four facts from different sections.
So instead of a single number, you get a profile. "This model scores ninety-two on retrieval, seventy-eight on multi-hop reasoning, eighty-five on aggregation."
Which is more honest. But it's also harder to put on a marketing page.
Well, that's the whole problem. Marketing wants one number. Bigger is better. And tokens deliver that perfectly — the number only ever goes up, and it's always impressive to someone who doesn't know what it means.
And there's a genuine structural reason this is hard, beyond marketing inertia. Tokens are universal in a way that tasks aren't. Every model consumes tokens. You can count them the same way regardless of what the model is being used for. But the moment you try to create a task-based unit, you have to decide which tasks matter. Is contract review the standard unit? What about code generation? What about translation?
The unit fragments because the use cases fragment.
Yes. And that's not just a marketing problem, it's a genuine measurement problem. The reason tokens persist isn't only that vendors like big numbers — it's that they're the one thing every model has in common.
So we're stuck between a unit that's universal but illegible, and units that are legible but not universal.
That's the tension. And I think what Daniel's really asking is whether there's a way out of that trap.
Let me try a framing. In computing, we used to sell processors by clock speed. Megahertz, then gigahertz. It was the universal number. And for a while it worked — a three-gigahertz chip was faster than a two-gigahertz chip. Then chip architecture got more complex, and a lower-clocked chip with a better design could outperform a higher-clocked one. Clock speed became a bad proxy for performance.
And the industry shifted to benchmarks. Things like SPECint, or Geekbench scores.
Right. But notice that the benchmarks didn't replace clock speed on the spec sheet. They supplemented it. The box still says "three point two gigahertz." The benchmark score is somewhere else, for the people who know to look for it.
And I think that's probably where AI is heading. The token count stays because it's the raw spec. But alongside it, you start seeing capability statements that are more legible. The question is whether those capability statements ever get standardised into something you can compare across vendors.
What would it take for that to happen?
Pressure from buyers. Enterprise procurement teams are already asking these questions. When a company is deciding between models for a document-processing pipeline, "one million tokens" doesn't help them choose. They want to know which model actually performs better on their specific task. And as AI gets embedded further into business systems, the people making purchasing decisions are increasingly not the people who know what a token is.
They're the legal ops manager, or the CFO, or the head of customer support. They don't care about the unit. They care about whether the thing works.
And that creates demand for a legible unit. Whether that demand produces a standardised answer or just a bunch of competing vendor-specific claims — that's the open question.
There's another angle here worth pulling on. The token count isn't just illegible — it's actively misleading in a specific way that matters for business users. A model with a million-token context window can technically hold a million tokens. But the quality of its output degrades as you approach that limit. The attention mechanism gets diluted. The model starts forgetting things at the beginning of the context, or gets confused about where information is located.
This is the "lost in the middle" problem. There's been solid research showing that model performance is best at the beginning and end of the context window, and worst in the middle. So if you've got a million-token window and you put a critical fact at token five hundred thousand, the model is less likely to retrieve it accurately than if you put it at token ten thousand or token nine hundred and ninety thousand.
Which means the usable context is smaller than the advertised context. On two axes now — reasoning capability and simple retrieval both degrade.
And some of the newer benchmarks try to capture this. There's a metric called "effective context length" — the point at which model performance drops below some threshold, like ninety percent accuracy on a retrieval task. For many models, the effective context length is significantly shorter than the advertised context window.
So if you were designing a legible unit from scratch, you'd want it to reflect effective capability, not theoretical maximum. "This model reliably handles documents up to two hundred pages" rather than "this model has a one-million-token context window."
And you'd want it to be task-specific, because a model might handle two hundred pages of prose summarisation fine but fall apart at fifty pages of multi-step legal reasoning.
Which brings us back to the fragmentation problem. The more accurate the unit, the less universal it becomes.
I don't think there's a way around that. The world is complicated. Any unit that's simple enough to put on a billboard is going to be misleading. The question is whether we can find something that's legible enough to be useful without being so simplified that it's deceptive.
Daniel mentioned the currency analogy — a coffee costing two thousand units. That's a perfect parallel, because when currencies hit that point, countries sometimes redenominate. They knock off zeros. A thousand old pesos becomes one new peso.
And we can't exactly redenominate tokens, because the token is tied to the actual computational unit. You can't just declare that a thousand tokens are now one "super-token" and have that mean anything real.
Unless you define the super-token as something like "the amount of context required to reliably perform task X." Which is back to task-based units.
Right. We keep circling back to the same place. The honest answer to Daniel's question — "what would that unit actually be?" — is that it would probably be a family of task-specific benchmarks, with some attempt at a summary score for high-level comparison, and the raw token count would persist alongside it as the technical spec.
Has anyone actually tried to ship this? Not a research benchmark, but a product-facing capability statement that replaces or supplements the token count?
Anthropic has been interesting on this. When they talk about Claude's context window, they tend to frame it in terms of use cases — "you can upload entire books" or "process a full codebase." They still quote the token count, but the marketing emphasis is shifting toward what you can do with it. Google's doing the same thing. "An hour of video" or "a seven-hundred-thousand-word novel" — those are attempts to create human units.
But they're not standardised. "An hour of video" depends on the video. "A novel" depends on the novel. War and Peace is not The Great Gatsby.
No, and that's the problem. They're reaching for legibility but they're not reaching for comparability. And comparability is what you need if you're making a purchasing decision between two models.
So what would comparability require? Some kind of standards body?
That's one path. Something like an IEEE working group that defines standardised long-context benchmarks and reporting requirements. Or it could come from a major enterprise buyer — a big bank or a government agency that says "we won't purchase any AI system that doesn't report performance on these five standardised tasks."
The procurement angle is interesting. The federal government has driven standardisation in plenty of industries just by being the biggest customer and demanding specs.
And enterprise RFPs are already getting more sophisticated about AI. Companies are asking for specific accuracy thresholds on their own test sets, not just token counts. They're building internal evaluation pipelines because they don't trust the vendor numbers.
Which is the rational response to an illegible unit. You build your own measurement system.
And that's expensive. It creates an advantage for large enterprises with the resources to do their own evaluation, and leaves smaller buyers relying on marketing claims they can't verify.
So the absence of a legible unit isn't just an annoyance, it's a market structure problem. It creates information asymmetry.
Yes. And information asymmetry in markets tends to produce bad outcomes — adverse selection, lemons problems. If buyers can't tell good models from bad ones on the dimensions they care about, the market doesn't reward quality on those dimensions.
Which is a knock-on effect Daniel might not have been thinking about, but it's probably the most important reason to care about this. As AI gets embedded in actual business processes, the cost of picking the wrong model goes up, and the cost of not being able to compare models accurately goes up with it.
Let me try to put some structure around what a good capability unit would need to do. I think there are four requirements. One, it has to be legible to a non-specialist. Two, it has to be comparable across models — the same unit means the same thing regardless of who's reporting it. Three, it has to be task-relevant — it actually correlates with performance on the things the buyer cares about. And four, it has to be hard to game — vendors can't trivially optimise for the metric without actually improving the underlying capability.
That fourth one is the killer. Any metric that becomes a target ceases to be a good metric.
Goodhart's law. And we've seen this play out in every industry that's tried to standardise performance measurement. Standardised tests in education, credit ratings in finance, CPU benchmarks in computing — the moment the metric matters, people optimise for the metric rather than the thing it's supposed to measure.
So even if we get a legible unit, it'll have a half-life. It'll be useful for a while, then it'll get gamed, then we'll need a new one.
That's probably the cycle. But that doesn't mean it's not worth doing. A useful metric that lasts three years is better than no metric at all.
Fair. So let's talk about what vendors are actually going to say to business users in the near term. Daniel asked that specifically — as AI gets embedded further, somebody has to state these capabilities to people who will never care what a token is. What do those conversations look like?
I think we're already seeing the shape of it. The pitch is shifting from "our model has a million-token context window" to "our model can process your entire document set" or "our model can analyse a thousand-page report in seconds." It's the same underlying capability, but the unit is the customer's own workload.
Which is smart, because it sidesteps the comparison problem. You're not comparing to a competitor's number, you're describing what the thing does for this specific customer.
Right. And it's also vague enough to be hard to dispute. "Process your entire document set" — what does "process" mean? Summarise it? Answer questions about it? Extract structured data from it? The vendor gets to define success on their own terms.
It's legible without being verifiable. Which is maybe the worst of both worlds — the customer thinks they understand what they're buying, but they actually don't.
That's where the pressure for standardisation is going to come from. Not from vendors — they're perfectly happy with the current state of affairs. It'll come from buyers who've been burned. Companies that bought a model based on a capability claim, deployed it, and discovered it couldn't actually do the thing they bought it for.
There's a class-action-shaped hole in this conversation.
I was trying not to say it, but yes. If a vendor says "handles hundred-page contract review" and a law firm buys it and it misses a material clause, that's a dispute waiting to happen. And disputes create demand for clear definitions.
The timeline might be: vendors keep using tokens until the lawsuits start, and then suddenly there's an industry standard.
That's a cynical read, but it's not an unrealistic one. Standards often follow disasters.
Before we move on, I want to touch on something you mentioned earlier — the tokenisation variance across languages. If the same content is thirty tokens in English and ninety in Japanese, then a "one-million-token context window" means radically different things depending on what language you're working in.
It's a huge issue, and it gets almost no attention in the English-language AI discourse. If you're building a system for a Japanese law firm, the effective context window is a third of what it is for an American law firm, just because of how the tokeniser works. And that's before you even get to the question of whether the model reasons as well in Japanese as it does in English.
Which means any legible unit would need to be language-specific, or at least language-aware. "Handles two hundred pages of English text" isn't the same as "handles two hundred pages of Japanese text."
That's another reason the problem is hard. There are so many variables that collapsing it all into a single number feels almost dishonest.
Almost.
Well, it is dishonest. But it's also what markets demand. Buyers want a number. They want to rank things and make a decision and move on with their day.
The tension is between accuracy and decisiveness. An accurate capability statement is a multidimensional profile that requires interpretation. A decisive capability statement is a single number that's probably wrong.
The history of measurement in every field is the history of navigating that tension. We build simplified models of complex reality, we use them until they break, we build better ones. It's not a problem that gets solved, it's a problem that gets managed.
Which is maybe the most honest answer to Daniel's question. The unit he's looking for doesn't exist yet, and when it does, it'll be imperfect and temporary. But the pressure to create it is real and growing.
The candidates are visible. Effective context length, task-based benchmarks, capability profiles — these are all attempts to answer the same question. None of them is the answer, but they're pointing in the direction the answer will come from.
I think the thing I'd add — and this is where I land on Daniel's "why hasn't it happened yet" question — is that the token has one enormous advantage that no alternative has yet matched. It's generated automatically by the system. You don't need a human to evaluate anything, you don't need a benchmark suite, you don't need a standards committee. The tokeniser just counts them. That zero-cost measurability is a genuinely useful property, and any replacement unit is going to be more expensive to produce.
The token count is essentially free. A capability score requires running evaluations, maintaining test sets, updating them as models improve, defending against gaming. That's real ongoing cost.
The token persists not just because of marketing inertia, but because it's cheap. And cheap beats good in a lot of market contexts.
Until the cost of being wrong exceeds the cost of measuring properly.
Which is where we came in.
Before we wrap up, I want to flag one more candidate from research circles. There's a concept called "useful context ratio" — the proportion of the context window that the model can actually use effectively for a given task. So instead of saying "one million tokens," you'd say "achieves ninety percent useful context ratio on document summarisation up to five hundred thousand tokens, dropping to seventy percent at one million."
That's a profile, not a number. Which is more honest, but harder to market.
Right. And it requires defining the task and the threshold, which brings back all the standardisation problems.
We've got a landscape of partial solutions, none of which fully solves the problem, all of which improve on the status quo in some dimension while introducing new problems in others. That feels like the honest state of play.
I think that's exactly right. And the most likely near-term future is a hybrid — token counts persist as the raw spec, supplemented by increasingly standardised task benchmarks, with vendors also making capability claims in customer-specific language. It'll be messy, but it'll be more informative than what we have now.
The long term depends on whether the buyer pressure for standardisation outweighs the vendor incentive to keep things vague. I don't think we know the answer to that yet.
Speaking of people who've dealt with exactly this kind of measurement problem — Hilbert, you've been quiet back there. You were a technical writer, right? This has got to be familiar territory.
Hilbert: Mid-nineties. I wrote documentation for a company that made backup software. We had to explain to small business owners how much disk space the program needed. Megabytes meant nothing to them. Absolutely nothing.
What did you use?
Hilbert: Floppy disks. We'd say "this program takes up fourteen floppy disks." People understood that. They had floppy disks on their desk. They could hold one in their hand and think, okay, fourteen of these.
That's exactly the human unit we've been talking about.
Hilbert: It worked for about eighteen months. Then hard drives got bigger, floppy disks got less common, and the unit stopped meaning anything. But while it worked, it was the only thing on the spec sheet that customers actually read.
Did you try to standardise it? Get other companies to use the same unit?
Hilbert: We tried. Got three other software companies in a room, tried to agree on a standard conversion. One floppy disk equals one point forty-four megabytes. But then someone asked how many pages of text that was, and the whole thing fell apart. Nobody could agree on what a page was. Single-spaced, double-spaced, what font, what margins. We spent six hours arguing about margins.
That's perfect. That's exactly the token problem. You get one level of translation that works — tokens to words, or megabytes to floppy disks — and then someone asks what it actually means for their work, and the whole thing fragments.
Hilbert: We ended up shipping a calculator with the software. Little program that asked you what you were backing up — documents, spreadsheets, whatever — and estimated how many floppy disks you'd need. It was wrong about half the time, but people liked it because it gave them a number.
A number that was probably wrong, but decisive.
Hilbert: They'd rather have a wrong number they understood than a right number they didn't.
That's a pretty damning summary of where we are with AI capability measurement.
Hilbert: The thing is, you're all talking about whether the unit will come from standards bodies or enterprise buyers or whatever. It won't. It'll come from whoever figures out what the customer is actually trying to do and describes it in those terms. "Backs up your entire accounting system." That's what we should have said. Instead we said "requires eighteen megabytes of disk space" and then tried to explain what a megabyte was.
The vendor that wins is the one that says "processes your entire contract portfolio" instead of "one million tokens."
Hilbert: That's what I'd buy. I don't know what a token is. I don't want to know.
You just spent twenty minutes listening to us talk about tokens.
Hilbert: I was checking the levels. You two were fine.
Hilbert, before you go — the calculator that was wrong half the time. Did anyone ever complain?
Hilbert: Once. A dentist in Ohio. He said it told him he needed eight floppy disks and he only needed six. Wanted a refund for the two disks he bought. We sent him a box of ten and he wrote us a thank-you letter.
The cost of being wrong was two floppy disks.
Hilbert: Different world. Now if one of these AI things gets it wrong, someone loses a contract or misses a diagnosis. The unit matters more when the stakes are higher.
That's the whole episode in four words.
Hilbert: I'm going home. The levels are fine.
The unit matters more when the stakes are higher. That's where we should land. Daniel's question isn't just about measurement aesthetics — it's about what happens when AI moves from demos to decisions, and the people making those decisions can't read the spec sheet.
The cutting-room floor detail I wanted to mention — there's a paper from late twenty twenty-three that proposed something called "context utilisation efficiency," which is essentially a measure of how well a model's performance scales as you give it more relevant context. Some models improve linearly, some plateau quickly, and the slope of that curve tells you more about real-world usefulness than the raw token count ever will. It's not a consumer-ready unit, but it's the kind of thinking that gets us there.
The open question — will the industry converge on something standardised, or will it stay a mess of competing claims until a disaster forces the issue?
My bet is on the mess persisting longer than it should, and then a high-profile failure — a model that was sold as "handles your entire document set" missing something critical in a way that costs real money — creates the pressure for standardisation. That's the pattern. Standards follow scandals.
Which means the unit Daniel's looking for will probably arrive. Just not until we've proven we really need it.
Thanks to Hilbert Flumingtop for producing, and for the floppy disk calculator story which I'm going to be thinking about for the rest of the day.
This has been My Weird Prompts. If you've got a question about AI that's been bothering you — measurement problems, capability claims, anything else — email the show at show at my weird prompts dot com.
We'll be back soon.
