AI Evaluation
522 episodes · Page 2 of 22
#6159: When a Speech Model Can't Decide How to Write Dollars
One dataset says "$20," another says "twenty dollars." Fine-tuning a speech model means finding out which convention wins — and why it matters.
#6158: Naming the ASR Sub-Tasks Nobody Mapped
A 2001 linguistics paper already named Daniel's problem — but it points the other way. So what do you call a fine-tune set that expands "F Y I" to ...
#6154: Who Checks the Labels? Weak vs Strong Supervision
Human labels, rule-based labels, or a teacher model writing the exam — what actually makes supervision "strong," and who verifies it?
#6153: Decision Models: Typed Answers in One Forward Pass
Clef, Jev, and OpenAI's Decisions API return typed answers with probabilities in a single forward pass — no generated text to parse.
#6149: 4 Hard Fails in 84 Answers: Testing AI Bias on Israel-Palestine
A 19-prompt, 6-model eval tested whether AI models push back evenly on false claims about Israel and Palestine. One model failed four times.
#6146: Grounding Is Not Solved: DIY vs Gateway vs Native Search
Bolt-on search, gateway search, or native grounding? The plumbing didn't vanish — it just moved to someone else's bill.
#6145: Do LLMs Nod Along With Conspiracy Theories?
A 98-prompt eval asks one question: when a believer brings a conspiracy theory, does the model push back, or help them dig?
#6142: We Gave 4 AI Models the Same Toast Prompt
Four models, one buttered-toast physics question. Three invented studies that never existed — and the oldest model was the most fun.
#6141: Overheard at the Laundrette
A seven-minute laundrette scene about socks hides a real sentence. Can you spot the null cipher before it's revealed?
#6139: Which AI Writes Our Scripts Best? We Tested Five
Five models, three scripts each, one rubric. Sol won on length, Grok on voice — and the runner-up title hides a catch.
#6137: Two Ways a Chinese Open Model Gets Censored
Same weights, different landlord: how to tell a serving-time filter from censorship baked into the model itself.
#6136: How We Chose Our New Script Writer
Ten models audition to write the show. Seven fail the censorship gate — including the workhorse that wrote three episodes in four.
#6135: How Many Runs Does Your AI Eval Actually Need?
Reruns shrink one kind of noise and leave another untouched — so where should your eval budget really go?
#6134: Cheap Tokens, Expensive Scripts
A model with worse list prices can cost less per script — if the cache hits. The bill hides in tokens, not rates.
#6133: One Rater Is Enough: Designing Daniel's Taste Test
When the ground truth is one person's taste, agreement with strangers is the wrong bar. Stability with yourself is the right one.
#6132: How to Run a Model Eval on a Budget
Daniel is teaching himself to run evals on a real budget. Successive halving, paired comparisons, and why three clean scripts prove almost nothing.
#6131: Judging the Judges: How AI Evaluates Creative Writing
LLM judges match human raters — but the human ceiling is low. What that means for evaluating creative writing.
#6130: Picking a Human Labeling Tool for LLM Benchmarks
Building a human review layer for LLM benchmarks? Here's what Argilla, Label Studio, Prodigy, and Potato actually cost in money and maintenance.
#6129: How to Build an AI Preference Dataset
Pairwise comparison, forced choice, or just "this one or that one" — the engineering behind collecting human preferences on AI outputs.
#6127: Haiku 5.5 at 10 Cents: What the Cheap Model Actually Does
Haiku 5.5 costs 10 cents per million input tokens. We break down where it holds up, where it fails, and whether it can write this show.
#6126: Can You Train a Machine to Be Funny?
Comedy fights everything next-token prediction is built for. Turns out you can train a model to land a joke — and then measure it.
#6115: Claude Cracked the NSA's 1997 Curve Seeds
A 25-year crypto mystery solved in under 30 minutes — by finding a bit that wasn't there.
#6114: We Reviewed Our Own Worst Episode
Two AI hosts listen back to episode fifteen and critique the voices, the writing, and the mistakes they can't unhear.
#6111: Translation Models That Know Drama From Contracts
Specialist translation models beat general ones on benchmarks — then lose the human vote. What actually ships when you need a register, not just a ...