AI Evaluation
522 episodes
#6216: What Do You Call the Middle Tier Model?
Sonnet, Sol, Flash, Medium — the daily-driver models nobody can name, and why they're squeezed from above and below.
#6215: Can Anyone Still Spot an AI Voice?
Clones now pass as human. Generic AI voices still don't. And the detectors? They fall apart in the wild.
#6214: Nobody Trains an Uncensored Model From Scratch
Every "uncensored" model is a renovation, not a new build — and the political lean lives in the agreement layer, not the refusals.
#6213: Ten Weirdest AI Evals Ever Published
A pelican on a bicycle, a badger for Groundhog Day, and a benchmark where passing is the bad outcome — the strangest AI evals ever run.
#6212: Woolly Sedate Snail and Other Model Failures
Most of Hugging Face's 2.96M models sit unread. But the dust has a personality — snails, pirates, and a matchbox that resigns.
#6207: Two WERs, One Model: Inside ASR Eval Harnesses
Same model, same audio, two different word error rates. The gap lives in the normaliser — and that's just layer two of three.
#6202: Israel Votes, Part 6: Asking the Decision Models
Two decision models, identical evidence, one table of betting prices — and both flip their pick for Israel's next leader.
#6201: Inside Cloudflare's Clef: A Language Model That Can't Speak
Cloudflare's Clef reads text, images, and video but outputs only probabilities — no words. Here's what's actually under the hood.
#6199: Running AI Experiments Before an Election
What MLflow, Braintrust and notebooks actually do when you have five days and one forecast to get right.
#6196: Fine-Tuning ASR: When Your Model Memorizes You
Your fine-tuned speech model aces your training recordings — but does it actually work on fresh audio?
#6194: Teaching a TTS Model His Own Accent
One man fine-tuned a text-to-speech model on his own voice to get Irish-accented English that can say Hebrew words mid-sentence.
#6192: What Actually Happens When You Train a Model
Data in, model out — except the "data" is pairs, the split is the whole game, and one leaked speaker can fake your score.
#6190: When Your AI API Quietly Changes Under You
Same API name, different model behind it. How to tell when the thing you tested stops being the thing you're using.
#6189: What "Real" Improvement Actually Means in AI Evals
A p-value isn't the probability your result was luck. Here's what chance actually means in AI benchmarks — and why bold numbers need cross-examinat...
#6185: Why AI Labs Sell a Tier Above the Flagship
Fable, Deep Think, SuperGrok Heavy — what the $300/month top tier actually does that the flagship can't.
#6180: Mapping Code-Switching: 113 Pairs, No Community
Researchers count 113 code-switched language pairs — but a pair is just a label on a file, not a community of speakers.
#6176: Do Audio Models Hear How You Said It?
A benchmark called LISTEN found audio models score near chance when emotion lives in the voice, not the words. What does that mean for real calls?
#6175: Video Models Can't Tell Time: What Motion Benchmarks Reveal
Frontier models score 48% on tasks where humans hit 98% — the gap is motion, speed, and sequence, not scene description.
#6174: What Your Voice Reveals: Accent, Distance, and Movement in Audio AI
What can AI actually hear in your voice? Accent detection, speaker distance, whale songs, and the gap between deployed audio models and chatty gene...
#6173: Auditing a Sycophantic Model From the Inside
A stumped lab, a fawning model, and the diagnostic playbook nobody has a name for — from eval slicing to persona vectors.
#6172: Recipes vs. Regeneration: AI Photo Editing in 2026
Two ways AI edits photos now: models that write a readable recipe of slider values, and models that just repaint the whole frame.
#6171: AI Audio Restoration: What Actually Ships in 2026
Noise removal and bandwidth extension are shipping products today. Matching a clip to a target EQ is still mostly a research prototype.
#6169: What It's Like to Be a Token Inside an LLM
A token's-eye view of an LLM's forward pass — no cone, no squeezing, just a hallway where every layer scribbles on you.
#6164: We Gave Each Host Its Own AI Agent
We split our two hosts into separate AI agents with private context windows. The bill came to 78 cents an episode.