AI Evaluation
522 episodes · Page 5 of 22
#5845: The $700 Pin and Other AI Flops
Five AI misfires, from a $700 pin that overheats doing nothing to a flagship model nobody remembers.
#5842: When Your TTS Model Updates Itself
Resemble pushed new Chatterbox weights to Hugging Face. Our pipeline started using them without a single config change.
#5832: When Your GPS Is Lying: Aviation's Spoofing Problem
GPS spoofing is feeding pilots fake positions — and the panel stays green. How crews detect it, and why the airport stand sign can't save you.
#5817: When Agents Don't Need Your Pretty Buttons
Agents are becoming major software users — and they prefer raw command-line tools to polished interfaces. What replaces UX when your user can't see?
#5805: Freeze the Model, Move the Labels
Your classifier isn't stale — your label set moved. Why retraining is cheap, and why not retraining might be smarter.
#5804: Text Classifiers: Local Models vs Always-On Endpoints
A specialist classifier costs $5.70 per million messages. A general LLM costs $142. So why is the obvious product so hard to find?
#5799: Two Agents, One Approval Gate: Wiring a Dictation Workflow
A dictated prompt becomes an episode — if one agent resolves facts, a second tightens prose, and a human approves the dispatch.
#5777: Lorebooks: The Hobbyist RAG That Beat the Labs
How a chat frontend fork built a full retrieval system — keywords, vectors, recursion — so fictional characters would stop forgetting who they are.
#5770: Talking to Your Data: What MCP Doesn't Solve
MCP standardized the pipe. It has no opinion about what flows through it — and that's where the hard part lives.
#5769: Stack Overflow for Agents: SOFA's Trust Model
SOFA is an API-first knowledge exchange where AI agents search, post, and verify fixes — with reputation earned from verification, not upvotes.
#5743: Why Claude Keeps Saying "Prior Art
Daniel noticed "prior art" creeping into his Claude reports. Is it wrong — or just a prestige term the model learned to love?
#5697: Decision Models That Return Probabilities, Not Text
A new class of model skips text generation entirely and returns calibrated probabilities in one forward pass. Here's what that changes.
#5678: Why Voices Are Harder to Design For Than Music
Every word arrives clearly on a phone speaker — so why is it exhausting? The science of listening effort, and how to design audio that isn't.
#5673: Why Lowering Temperature Broke Daniel's Scripts
Lowering temperature should make output safer, not incoherent. Daniel's DeepSeek scripts say otherwise — and the reason may be stranger than he thi...
#5626: What Multimodal Models Actually Learn From Video
Late fusion, early fusion, and vision laziness — what actually happens when models train on video, audio, and text together.
#5625: Omni and the Marble That Rolls Uphill
Google's Omni claims an intuitive grasp of gravity and momentum. Can a video model really learn physics, or just remember what physics looks like?
#5624: One Forward Pass: Building a Bounded Classifier
No generation, no parsing — just label scores in a single pass. How to build a small encoder classifier that actually works.
#5623: When Your AI Cites Sources It Didn't Actually Use
Your RAG pipeline cites a source — but did the model actually use it? New interpretability research says you can't tell from the trace.
#5621: Constrained Decoding for AI-Generated Podcast Tags
A working AI podcast pipeline has one quiet failure: the tagging. Here's how constrained decoding and a stable taxonomy fix it.
#5608: When Your ISP Bot Won't Let You Talk to a Human
A caller with one simple request gets trapped in a diagnostic loop he doesn't need — and the escape hatch won't open.
#5570: Why Phone Trees Break and AI Voice Agents Can Fix Them
AI is good at routing calls and bad at resolving them. That single distinction explains why menu-replacing voice agents work and human-replacing on...
#5517: What 5,000 AI-Written Podcast Episodes Reveal
Daniel archived every AI-generated episode with the model that wrote it. Now he wants to know which models loop — and how to catch it.
#5482: When AI Edits Your Words: The Off Switch Problem
A tool that works reliably — and still gets switched off. What over-editing studies reveal about why AI rewrites more than you asked.
#5480: Writing a Personal Instruction for ChatGPT
Daniel's personal ChatGPT prompt gets a line-by-line critique — and the case for shorter, sharper custom instructions.