AI Evaluation
522 episodes · Page 4 of 22
#6023: How Much Voice Data Is Enough?
Two hours of phone recordings took Hebrew code-switching from 20% to 60%. But when does more data stop mattering?
#6022: Fine-Tuning for Code-Switched Hebrew on 2 Hours of Audio
Two hours of one man's voice beat models trained on 120,000 hours — but only for code-switched Hebrew.
#6009: Vertex AI Is Now the Gemini Agent Platform
Vertex AI just became the Gemini Enterprise Agent Platform. Here's the family tree, the five products, and when to actually move.
#6008: Why Codex Writes Code for a Dead API
Models don't hallucinate your API — they quote real docs from the wrong year. Here's why, and what actually fixes it.
#5994: Hebrew in Latin Letters: A TTS Pipeline Fix
A token classifier that finds Hebrew words written in Latin letters so TTS stops mangling them — built for under $3.
#5989: Twenty Tiny Beats That Make a Podcast Stick
Twenty novel recurring segment pitches — and the test each one has to pass: does it add context, or just noise?
#5982: Teaching a Voice Engine to Say "Yalla" Right
A small token classifier that spots Hebrew words hiding in English sentences — and why finding the word is the easy half.
#5962: How Word Error Rate Lies to Us
WER is the number everyone reports for speech recognition — and the one that agrees with human judgment least. Here's what it actually measures.
#5960: There's No Such Thing as Tokens Per Second
Why one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working.
#5959: What RTF Actually Tells You About Speech Models
Accuracy gets the headlines, but speed decides what ships. How RTF, latency percentiles, and memory shape real ASR deployments.
#5957: The Scientific Plugins Hiding in Plain Sight
ChatGPT's plugin directory has 5,723 entries — but the real scientific AI work is happening in arXiv papers, not plugins.
#5949: Inside Deep Research Agents: Claims, Citations, and the 84.7% Orchestrator Problem
Why 84.7% of citation errors originate at the orchestrator — and how a claim ledger fixes it.
#5946: Inside Humanity's Last Exam: 2,500 Questions AI Can't Ace
2,500 questions, ~1,000 experts, and a top score that jumped from 2.7% to 54.8% in 18 months. What is HLE actually measuring?
#5945: How a Reward Model Actually Trains an LLM
One number, no gradients, four models in the room. What a reward model actually does inside the RLHF loop.
#5941: Five Minutes Reading Your Dictation Prompt
A listener's speech-to-text cleanup prompt, read out in full — then stress-tested against what ASR actually gets wrong.
#5937: The Fine-Tuning Career: Titles, Skills, and What It Pays
The job above the labeling layer exists — it just doesn't have one name. What it takes, what it pays, and how to sell it.
#5922: Teaching a Model Your Voice: SFT vs DPO
Three ways to teach a model your writing voice — and the evidence says one of them quietly makes things worse.
#5917: The Women Who Were Computers Before Computers Existed
Before machines did the math, people did — and most of them were women. Reconstructing the desk of the human computer.
#5915: How Multi-Agent AI Simulations Keep a Shared World State
Concordia keeps the world in a dictionary an LLM writes to. Generative Agents keeps it in a spreadsheet. CAMEL keeps it in the chat log.
#5903: Video Tokens, Omni, and the Local VRAM Math
Video costs ~100 tokens per second. That single number explains why ChatGPT hides video upload and why local inference hits a KV cache wall.
#5899: Same GPU, Same Weights, Different Answers
Identical hardware, identical weights, different engine — and answers that shift by up to 16.6 points.
#5859: Slow Labs, Fast Ships: What Release Cadence Actually Tells Us
Anthropic ships premium models every 22.5 days — the fastest lab in the industry. So much for the slow-and-careful story.
#5858: Routing AI Models: When 3 Rules Beat a Trained Router
A deep dive into AI model routing — and why a learned router may not beat three rules you wrote by hand.
#5857: Building a Scorecard for Our Own AI Writers
Daniel wants a real scorecard for the agent that writes this show — repetition, dialogue quality, and whether the retrieval harness actually works.