Reasoning Models
227 episodes
#6500: Why LLM Leaderboards Need Psychometrics
Applying item response theory to AI benchmarks reveals up to 31 percent of leaderboard positions shift when raw accuracy is adjusted for difficulty.
#6499: Why Simple Rules Beat Value Models in Reasoning RL
DeepSeek R1-Zero boosted AIME accuracy from 15.6% to 77.9% using only two deterministic checks and no learned critic.
#6479: Why Research Agent Agreement Is Not Independence
When two distinct language models fail a task, they give the exact same wrong answer sixty percent of the time.
#6397: Why Model Benchmarks Fail at Wit
When prompted for a joke over 1,000 times, ChatGPT repeated the same 25 jokes more than 90% of the time.
#6393: Why Model Verbal Tics End Up Driving Script Benchmarks
A pattern search across forty recent episodes found host emotional forecasting in 55% of scripts generated by GPT-6.1 Sol.
#6384: Why LLM Output Tokens Cost More Than Input
Decode computes at just 1.4% utilization on an H100 because reading 140 gigabytes of weights takes 280 times longer than the arithmetic.
#6309: Why DeepMind Systems Live Outside the Gemini API
AlphaEvolve saved 0.7% of Google's compute, but running it requires an enterprise cloud environment to evaluate candidate code.
#6288: What Happens Between Model Checkpoints
Extending reinforcement learning on Olmo 3 Think for 21 days yielded a 20-point leap on IFBench without starting fresh pretraining.
#6285: Why Coding Benchmarks Split Across Models
GLM 5.3 claims 88.2 on Terminal Bench 2.1 against Opus 4.8 at 85, but newer evaluations reveal a very different order.
#6283: Why Terminal Benchmarks Miss Real Desktop Faults
A hardness audit revealed that over a third of universally failed benchmark tasks were broken rather than genuinely difficult.
#6278: Why Model Benchmarks Miss Small Workflow Friction
Switching Claude Code to Sonnet broke an everyday shell alias request that Opus solved simply by recognizing where to look in local context.
#6245: Why Inference Engines Change Benchmark Scores
A study found a 16.6 percentage point spread on GSM8K across different inference engines running identical weights on the same GPU.
#6235: When Your AI Agent Has Too Many Skills
Your agent has 55,000 skills installed — and it's spending the whole task just reading the menu.
#6216: What Do You Call the Middle Tier Model?
Sonnet, Sol, Flash, Medium — the daily-driver models nobody can name, and why they're squeezed from above and below.
#6213: Ten Weirdest AI Evals Ever Published
A pelican on a bicycle, a badger for Groundhog Day, and a benchmark where passing is the bad outcome — the strangest AI evals ever run.
#6202: Israel Votes, Part 6: Asking the Decision Models
Two decision models, identical evidence, one table of betting prices — and both flip their pick for Israel's next leader.
#6185: Why AI Labs Sell a Tier Above the Flagship
Fable, Deep Think, SuperGrok Heavy — what the $300/month top tier actually does that the flagship can't.
#6184: Anthropic's Unreleased Model Two, Explained
Anthropic trained a model better than what it sells — and never released it. What that gap actually tells us.
#6173: Auditing a Sycophantic Model From the Inside
A stumped lab, a fawning model, and the diagnostic playbook nobody has a name for — from eval slicing to persona vectors.
#6172: Recipes vs. Regeneration: AI Photo Editing in 2026
Two ways AI edits photos now: models that write a readable recipe of slider values, and models that just repaint the whole frame.
#6164: We Gave Each Host Its Own AI Agent
We split our two hosts into separate AI agents with private context windows. The bill came to 78 cents an episode.
#6153: Decision Models: Typed Answers in One Forward Pass
Clef, Jev, and OpenAI's Decisions API return typed answers with probabilities in a single forward pass — no generated text to parse.
#6149: 4 Hard Fails in 84 Answers: Testing AI Bias on Israel-Palestine
A 19-prompt, 6-model eval tested whether AI models push back evenly on false claims about Israel and Palestine. One model failed four times.
#6142: We Gave 4 AI Models the Same Toast Prompt
Four models, one buttered-toast physics question. Three invented studies that never existed — and the oldest model was the most fun.