AI Evaluation
522 episodes · Page 14 of 22
#2191: Making Multi-Agent AI Actually Work
Research from Google DeepMind, Stanford, and Anthropic reveals most multi-agent systems waste tokens and amplify errors. Single agents with better ...
#2190: Simulating Extreme Decisions With LLMs
LLMs fail at the exact problem wargaming was built to solve—simulating irrational, extreme decision-makers. A new study reveals why.
#2189: Scaling Multi-Agent Systems: The 45% Threshold
A landmark Google DeepMind study reveals that adding more AI agents often degrades performance, wastes tokens, and amplifies errors—unless your sin...
#2188: Is Emergence Real or Just Bad Metrics?
The debate over whether AI models exhibit genuine emergent abilities or just appear to because of how we measure them—and why it matters for safety...
#2187: Why Claude Writes Like a Person (and Gemini Doesn't)
Claude produces prose that sounds human. Gemini reads like Wikipedia. The difference isn't capability—it's how they were trained to think about wri...
#2186: The AI Persona Fidelity Challenge
Advanced LLMs dominate benchmarks but fail at staying in character—especially when asked to play morally complex or antagonistic roles. What does t...
#2185: Taking AI Agents From Demo to Production
Sixty-two percent of companies are experimenting with AI agents, but only 23% are scaling them—and 40% of projects will be canceled by 2027. The ga...
#2181: When RAG Becomes an Agent
RAG in chatbots is simple retrieval. RAG in agents is a multi-step decision loop. Here's what actually changes.
#2178: How to Actually Evaluate AI Agents
Frontier models score 80% on one agent benchmark and 45% on another. The difference isn't the model—it's contamination, scaffolding, and how the te...
#2175: Let Your AI Argue With Itself
What happens when you let multiple AI personas debate each other instead of asking one model one question? A deep dive into synthetic perspective e...
#2173: Inside MiroFish's Agent Simulation Architecture
MiroFish generates thousands of AI agents with distinct personalities to predict social dynamics. But research reveals a critical flaw: LLM agents ...
#2172: Council of Models: How Karpathy Built AI Peer Review
Andrej Karpathy's llm-council uses anonymized peer review to make language models evaluate each other fairly—but can it really suppress model bias?
#2171: How IQT Labs Built a Wargaming LLM (Then Archived It)
A deep code review of Snowglobe, IQT Labs' open-source LLM wargaming system that ran real national security simulations before being archived. What...
#2165: Strip Your Agent to Bash
The frameworks matter less than you think. What separates a working agent from a failing one is the harness—the orchestration, memory, and tool des...
#2164: Why Bigger Context Windows Don't Fix Attention
Frontier models have million-token context windows, but attention degrades well before you hit the limit. New research reveals why bigger isn't bet...
#2162: When Knowledge Work Stops Being Safe
The knowledge economy promised safety from automation. Then AI arrived. Here's how we got here—and why the disruption this time is different.
#2160: Claude's Latency Profile and SLA Guarantees
Claude is measurably slower than competitors—and Anthropic's SLA promises are even thinner than the latency numbers suggest. What enterprises actua...
#2146: The AI Wargame's Flat Hierarchy Problem
AI wargames treat NGOs and nuclear powers as equals. That's a dangerous flaw for real-world policy planning.
#2144: AI Wargaming: One Model or Many?
Should geopolitical AI simulations use one model or many? We debate the pros and cons of a single-model approach.
#2143: Simulating Geopolitics Under Asymmetric Information
A two-stage AI pipeline predicted a 4% chance the Iran-Israel ceasefire would survive a month, using Monte Carlo simulations and an LLM council.
#2139: AI Wargame Memory: Beyond the Context Window
Why simply extending context windows fails in multi-agent simulations, and how layered memory architectures preserve strategic fidelity.
#2137: Wargaming's Methodology, Not Magic
Most AI wargames are just expensive role-play. Here's the professional methodology they're missing.
#2136: The Brutal Problem of AI Wargame Evaluation
Most AI wargame simulations skip evaluation entirely or rely on token expert reviews. This is the field's biggest credibility problem.
#2135: Is Your AI Wargame Signal or Noise?
Monte Carlo methods promise statistical rigor for AI wargaming, but the line between genuine insight and sampling noise is thinner than you think.