Evals
7 episodes
#6135: How Many Runs Does Your AI Eval Actually Need?
Reruns shrink one kind of noise and leave another untouched — so where should your eval budget really go?
#6134: Cheap Tokens, Expensive Scripts
A model with worse list prices can cost less per script — if the cache hits. The bill hides in tokens, not rates.
#6133: One Rater Is Enough: Designing Daniel's Taste Test
When the ground truth is one person's taste, agreement with strangers is the wrong bar. Stability with yourself is the right one.
#6132: How to Run a Model Eval on a Budget
Daniel is teaching himself to run evals on a real budget. Successive halving, paired comparisons, and why three clean scripts prove almost nothing.
#6131: Judging the Judges: How AI Evaluates Creative Writing
LLM judges match human raters — but the human ceiling is low. What that means for evaluating creative writing.
#6130: Picking a Human Labeling Tool for LLM Benchmarks
Building a human review layer for LLM benchmarks? Here's what Argilla, Label Studio, Prodigy, and Potato actually cost in money and maintenance.
#6129: How to Build an AI Preference Dataset
Pairwise comparison, forced choice, or just "this one or that one" — the engineering behind collecting human preferences on AI outputs.