The AI industry has a leaderboard addiction. Every new model launch comes with a score on MMLU, or HumanEval, or HellaSwag — some number that's supposed to tell you how good it is. But here's the thing Daniel's been thinking about: what happens when those scores stop meaning anything useful? Not because they're wrong, exactly, but because they're answering a question nobody in the real world is actually asking.
Daniel wrote in with a whole thing about this. He sees a gap opening up between broad benchmarks — the kind that test whether a model can do a hundred different things adequately — and the specific, high-stakes questions organizations actually need answered. Can this model flag adverse drug interactions in pediatric oncology notes? Can it review contracts under Louisiana insurance law without hallucinating a clause that doesn't exist? Those aren't benchmark questions. They're deployment questions. And he thinks answering them is going to become a distinct career — in-house specialists or external consultants who build bespoke test suites for niche, domain-specific tasks. He calls it laser-focused evaluation.
He's got a few questions baked in here. What exactly is this role, and how does it differ from traditional benchmarking? What skill set should someone be building if they want to walk into this? And how do you keep up when the model landscape changes basically every week? So let's start with the problem that makes the role necessary — the broken state of AI benchmarking.
The core issue is that benchmarks like MMLU and HumanEval are broad capability proxies. They test whether a model can do many things adequately — answer a med school question, solve a logic puzzle, write a function that reverses a linked list. That's useful for comparing models at a high level, but it tells you almost nothing about whether the model can do one specific thing reliably enough to deploy.
And the scores look precise. Ninety-two point three percent. Eighty-seven point six. Numbers that feel solid.
Right, and they're not. The 2024 study that put MMLU under a microscope found something like thirty percent of benchmark questions already appeared in training data. The models had seen the test. That's data contamination, and it's widespread — labs train on enormous web crawls, and benchmark test sets get sucked in. Nobody's necessarily doing it on purpose, but nobody's exactly rushing to clean it up either.
So the score is measuring memorization, not reasoning.
Partially. And then there's prompt sensitivity. You can take the same model, same benchmark, just rephrase the question slightly — same meaning, different words — and the score swings ten or fifteen points. That's not a measurement, that's a slot machine. Labs publish the run that looked best, not the distribution of runs. Selective reporting.
Which means the leaderboard is a highlight reel.
It's the dunk contest, not the season. And for a hospital deciding whether to deploy a model that reads radiology reports, the season is what matters. One missed tumor because the prompt was phrased slightly differently than the test set — that's not an acceptable failure mode.
So that's the problem. Benchmarks are gamed, they're broad, and they don't predict specific task performance. Daniel's point is that fixing them at scale is basically impossible, because the number of possible domain-specific tasks is effectively infinite. You can't have a benchmark for every niche.
That's the fundamental insight. There cannot be a benchmark for "does this model correctly flag adverse drug interactions in pediatric oncology notes under Israeli Ministry of Health guidelines." There just can't. The number of combinations of domain, jurisdiction, use case, and risk tolerance is combinatorially enormous. So you need someone who builds that test from scratch, for that specific question, and runs it across the models you're considering.
Which is the role Daniel's describing. Not a benchmark maintainer — an evaluator.
And the distinction matters. A benchmark maintainer curates a test set that's supposed to be general. An evaluator designs a test set that's laser-focused on one deployment decision. The evaluator has to understand the domain well enough to know what good looks like, the models well enough to know how to probe them, and statistics well enough to know whether the results mean anything.
Walk me through what that actually looks like. Say I'm a law firm, and I want to know whether I can use a frontier model for document review. What does the evaluator do?
First, they don't reach for LegalBench. LegalBench is a broad legal reasoning benchmark — it'll tell you which models are generally good at reading statutes and applying precedent. But my firm doesn't practice general law. We handle insurance defense in Louisiana. Our documents reference specific state statutes, specific court rulings from the Fifth Circuit, specific policy language that shows up in hurricane claims. A model that aces LegalBench might still hallucinate a Louisiana statute that doesn't exist, or misapply Texas law because it's statistically similar.
So the evaluator builds a custom test set.
They work with the firm's senior partners to identify the highest-risk error types. What would actually expose the firm to malpractice? Hallucinating a case citation. Misclassifying a claim type. Missing a statute of limitations. Then they build a test set of, say, two hundred document review tasks that specifically probe those failure pattern, using real documents from the firm's own archives — redacted, obviously.
And they run it across every model the firm is considering.
Claude 4, GPT-5, Llama 4, Mistral Large 3 — whatever's current. But they don't just get a score. They produce a report that says: Model A missed three statute of limitations issues in the test set, which is unacceptable for deployment. Model B got all the statutes right but hallucinated case citations in four percent of responses. Model C performed worse on Louisiana-specific questions than on general insurance law, suggesting it's under-trained on state-level material. Here's the ranking, here's the risk profile, here's our recommendation.
That's a very different deliverable from "Model B scored eighty-seven on LegalBench."
Completely different. And it's worth real money. A fifty thousand dollar evaluation engagement that prevents a two million dollar deployment failure — that's an easy conversation. The firm doesn't need to understand perplexity or sampling temperature. They need to know which model won't get them sued.
So the business model makes sense. What about the skill set? Daniel specifically asked what someone should be arming themselves with if they want to do this.
I think there are four pillars. The first is real understanding of LLM architecture and behavior — and I don't mean "I took an API tutorial." I mean knowing how sampling temperature changes output distributions, how system prompts interact with few-shot examples in the context window, how different models handle long-context retrieval differently. If you're evaluating a model for document review, you need to know that some models get worse at retrieving information from the middle of long documents — the "lost in the middle" problem — and you need to design your test set to catch that.
Because the client won't know to ask.
They won't. They'll say "the model read the contract and gave an answer," and they won't know it missed a clause on page fourteen because the context window handling dropped it. The evaluator has to know that's a thing that happens and test for it explicitly.
What's the second pillar?
Statistical literacy. Designing a test set that's actually valid. How many test cases do you need to detect a five percent error rate with confidence? If you're running twenty different evaluations and looking for the best score, you're doing multiple comparisons — are you correcting for that? A lot of what passes for evaluation right now is just vibes. "We tried a few prompts and the model seemed good." That's not evaluation, that's an anecdote.
Sample size, confidence intervals, avoiding p-hacking. The stuff that clinical trials have been dealing with for decades.
And I think my old career is actually a useful reference point here. In medicine, we don't deploy a drug because a few doctors tried it and liked it. We run controlled trials with pre-registered endpoints and statistical analysis plans. AI evaluation needs to grow that same muscle. The evaluator who can say "we're ninety-five percent confident the error rate is below two percent" is offering something qualitatively different from "the model did pretty well in our tests."
Third pillar?
Domain expertise. You cannot evaluate a model for radiology report generation without understanding radiology. You don't need to be a radiologist — you might pair with one — but you need to know enough to design meaningful test cases and interpret errors. Is a hallucinated anatomical detail catastrophic or cosmetic? Only someone who knows the domain can answer that. This is why I think the role will often be a consultant pairing a domain expert with an ML engineer. The evaluator is the bridge.
And the fourth pillar?
There's an emerging ecosystem here. EleutherAI's LM Evaluation Harness is the most widely used open-source framework — it lets you build custom evaluation tasks programmatically. Anthropic has an evals platform. LangChain has evaluation tooling. But the tooling is only half the story. The evaluator also needs version control for test sets — because test sets evolve as you discover new failure pattern — and automated CI/CD pipelines that re-run evaluations whenever a new model drops.
That last part is the moving target problem Daniel mentioned. New models release constantly. An evaluation from last month might already be stale.
And this is where the workflow gets concrete. The evaluator maintains what I'd call a model registry — a running log of every frontier model release, with evaluation results for each model version against their core test suites. They subscribe to model release trackers: the Hugging Face model hub, the LMSYS Chatbot Arena leaderboard, academic pre-print servers. And they run a weekly automated sweep — every new model that appears gets run through the test suites, and any significant performance change gets flagged.
The skill isn't knowing what released. It's knowing which releases matter for your clients' domains.
Right. If you're evaluating for a legal client, you don't need to care about a new code generation model. You care about the model that just improved long-context retrieval or reduced hallucination rates on legal text. Filtering the signal from the noise is part of the job.
How much of this is automatable? If the evaluation pipeline is automated, what's the human evaluator actually doing?
Designing the test. That's the part that doesn't automate. Deciding what to test, what constitutes a pass, what error types are catastrophic versus acceptable — those are judgment calls that require domain knowledge and risk reasoning. The pipeline runs the tests. The evaluator decides what the tests should be and interprets what the results mean for a specific deployment decision.
So the evaluator is part engineer, part statistician, part domain consultant, and part risk analyst.
And part skeptic. A healthy suspicion of all scores — including your own. Which brings me to something I think Daniel's framing doesn't quite capture. There's an ethical dimension here that matters a lot.
Say more.
Evaluators have power. Their reports determine which models get deployed. If an evaluation consultancy develops a reputation for favoring certain model providers, that distorts the market. If a single evaluation methodology becomes the de facto standard, it can stifle innovation — model providers optimize for the test, not for real performance, and we're right back to the benchmark gaming problem, just with a different test.
The evaluator becomes the new benchmark.
And the incentives can get ugly. Imagine a bank hires an evaluation consultancy to pick a model for loan underwriting. The consultancy knows the bank's CTO already likes a particular model. Do they design the test set in a way that's genuinely neutral, or do they — consciously or not — weight it toward the model the client wants?
So the same gaming problem that plagues benchmarks can infect bespoke evaluation.
It can. The field needs norms. Transparency around test sets, scoring rubrics, and raw results. Maybe eventually third-party accreditation — an auditor who reviews evaluation methodologies the way financial auditors review accounting practices. If this becomes a real profession, it needs professional standards.
Let me pull on the market implications for a second. If bespoke evaluation becomes standard practice, what does that do to the model provider landscape?
It could shift it pretty dramatically. Right now, the frontier model providers compete on broad benchmark scores — who's got the highest MMLU, the best HumanEval. But if enterprises start making deployment decisions based on bespoke evaluations, the competition shifts to specific task performance. And that opens the door for smaller, specialized models.
A model fine-tuned for legal reasoning could beat a general frontier model on its home turf.
And the evaluation report would prove it. You could have a boutique model provider that only does medical literature summarization, and if they consistently beat GPT-5 on that specific task in independent evaluations, hospitals will buy it. The evaluation reports become the market signal, not the leaderboard.
Which means the evaluators become gatekeepers in a way that's worth paying attention to.
They do. And that's why the ethical dimension isn't a footnote — it's central to whether this role develops in a healthy way or just replicates the problems it's supposed to solve.
Before we go further, Hilbert's been quiet back there.
I saw him nod at something a few minutes ago.
Hilbert: Two thousand twenty-two. I worked three weeks as a prompt evaluator for a startup that claimed to benchmark AI customer service bots.
What did that involve?
Hilbert: Reading chat logs. Rating responses on a one-to-five scale. The rubric was a Google Doc that changed every time the CEO got nervous. Which was often.
What was the rubric based on?
Hilbert: Vibes. The CEO's word, not mine. "Does this response feel helpful?" No definition of helpful. No consistency across raters. I gave a response a three, the guy next to me gave the same response a five, and the CEO averaged them and called it a score.
So the numbers looked precise but there was nothing underneath.
Hilbert: I quit after three weeks. The passing threshold kept moving. Every time a competitor released a benchmark score, our threshold dropped just enough to keep our product on top. The test set wasn't measuring the model — it was measuring whatever the CEO needed it to measure that afternoon.
That's exactly the gaming problem, just at the evaluation layer instead of the benchmark layer.
Hilbert: It gets worse. I had another client later — this was freelance, I was doing data labeling for an insurance company. They were evaluating models for claims processing. Had a whole test suite built, proper methodology, the works.
Sounds promising.
Hilbert: They'd already bought the model. This was six months after deployment. The evaluation wasn't to choose a model — it was to justify the choice they'd already made. Every time a test case made their model look bad, they'd argue it was unrealistic. "That claim type is rare." "That phrasing would never come from a real adjuster." They shaved the test set down until their model looked great, then showed the report to the board.
They gamed their own evaluation.
Hilbert: Bespoke evaluation is only as honest as the person writing the check. You can have the best methodology in the world. If the client wants a specific answer, they'll find a way to get it. Or they'll find an evaluator who'll give it to them.
So the independence problem isn't theoretical. It's already happening.
Hilbert: It happened the moment someone realized you could sell a number. The number's always for sale. The question is who's buying and what they want it to say.
That's a pretty sobering counterpoint to everything we've been building up here. The whole premise of bespoke evaluation is that it's more trustworthy than broad benchmarks because it's specific and transparent. But if the client controls the test design, the transparency doesn't matter — they're transparently gaming it.
Hilbert: I'm not saying don't do it. I'm saying build the audit trail. Every test case, every score, every version of the rubric, timestamped and immutable. If you can't show your work — all of it, not just the final report — you're not evaluating, you're marketing.
That's the accreditation question, isn't it? Whether this professionalizes to the point where there's a standard for what counts as a valid evaluation, and some mechanism for verifying that standard was followed.
And who does the accrediting. If it's the model providers, we're back to square one. If it's the evaluators themselves, you get regulatory capture — the established players define the standards to lock out competitors.
Hilbert: Same problem every industry has. The people who need watching are usually the ones writing the rules.
So the field needs independent audit — third parties who review evaluation methodologies but don't perform evaluations themselves. Separation of powers, basically.
Which is a much bigger institutional ask than "some consultants start offering evaluation services." It implies a whole ecosystem: evaluators, auditors, standards bodies, maybe eventually regulators.
That's where I think Daniel's career prediction gets interesting in a way he might not have intended. He's picturing a technical role — someone who runs evals and produces reports. But if Hilbert's right, the role that actually matters is the person who can look at an evaluation report and say whether it was done honestly.
The meta-evaluator.
That's a different skill set again. Less engineering, more forensic. You're not designing tests, you're interrogating test design.
Which brings us back to the skill set question. If you're building a career in this space, statistical literacy isn't just about designing valid tests — it's about being able to spot invalid ones. Reading an evaluation report and noticing that the sample size is too small, or the confidence intervals are missing, or the test set was clearly cherry-picked.
The evaluator who can't audit someone else's work is only half-trained.
The client who can't tell the difference between a real evaluation and a vibes-based sales pitch is going to get burned. That's part of the value proposition, honestly. A good evaluator doesn't just produce reports — they teach the client what a good report looks like.
Where does this all land? Daniel asked what skills someone should build, and I think we've got a clearer picture now. Deep LLM knowledge — not API-level, architecture-level. Statistical rigor — sample sizes, confidence intervals, multiple comparison correction. Domain expertise — either your own or through partnership. Tooling fluency — evaluation frameworks, version control, automated pipelines. And a layer we didn't start with but clearly need: the ability to audit, to spot gaming, to maintain independence when the client wants a specific answer.
The keeping-up problem. Maintaining a model registry, running weekly sweeps, knowing which releases matter for which domains. That's not a one-time skill, it's an ongoing practice. The evaluator who stops tracking new models becomes obsolete in about six weeks.
The open question I keep coming back to is whether this role professionalizes into something with recognized credentials — a certified AI evaluator, some kind of license or accreditation — or whether it stays a boutique consultancy niche where reputation is the only currency.
I suspect both. Large enterprises and regulated industries will demand certification — they need someone to point to when the auditor asks who approved the model. Smaller firms will hire based on reputation and word of mouth. The market will segment.
The model providers — will they get into this business? Build their own bespoke evaluation services? "Buy our model, and we'll also tell you it's the best one for your use case."
Some already are, in a soft way. "We evaluated our model on your behalf and it performed great." The conflict of interest is obvious, but if the client doesn't know enough to spot it...
That's the whole game, isn't it? The evaluation industry exists because the client can't evaluate the models themselves. But the client also can't evaluate the evaluators. There's always a trust problem one layer up.
Which is why the best evaluators will be the ones who combine deep technical skill with genuine domain expertise and a healthy skepticism of all scores — including their own. The evaluator who's never surprised by their own results isn't paying attention.
Hilbert mentioned immutable audit trails. I think that's the practical takeaway. If you're building an evaluation practice, build it so that everything is reproducible. Every test case, every model version, every prompt, every score. If someone wants to tear your report apart, they should be able to reconstruct exactly what you did and run it themselves.
Reproducibility as credibility. That's a good standard.
Daniel, if you're listening — I think your instinct is right. This is a real role, and it's going to grow. The skill set is deeper than it looks from the outside, and the independence question is going to be the thing that separates the real evaluators from the salespeople. But the demand is there. Organizations need to know which model works for their specific problem, and broad benchmarks can't tell them. Someone has to build the test.
Thanks to our producer Hilbert Flumingtop for keeping us honest, as always.
This has been My Weird Prompts. If you want to dig deeper into any of this, the show notes are at my weird prompts dot com. We'll be back soon.
See you then.
