AI Inference
554 episodes · Page 5 of 24
#6076: How AI Actually Predicts Elections
Bayesian models, "silicon sampling," and agent societies — what actually works for forecasting votes, and what's just fashion.
#6048: Under the Hood, October 2026: Seventeen Prompt Versions and Three-Times-Faster Voices
Four weeks of pipeline changes, grounded in the code: grace periods, duplicate detection, Hilbert's rules, echo deletion, and voices nearly three times faster.
#6038: The Model You Pay For Isn't the Best Deal
Eight Chinese labs, two tiers each, and a pricing spreadsheet that says your default model is mid-pack.
#6037: Gemini 4 Argon: Google's Million-Token Flagship
A close read of Gemini 4 Argon's model card — a million-token window, thinking levels, implicit-only caching, and a lot of blanks.
#6036: GPT-6 Astra: A Million Tokens, No Fine-Tuning
A million-token window, ten built-in tools, a 272K pricing cliff — and no fine-tuning. What GPT-6 Astra actually offers builders.
#6034: Claude Opus 5.5: What Anthropic Actually Shipped
Anthropic's new flagship is cheaper than the model it replaces — and won't let you turn thinking off.
#6029: One Model, Two Modes: Does AI Consolidation Hold Up?
OpenAI hides a router, DeepSeek merged two modes, Anthropic removed the knob. Is the model landscape really consolidating?
#6028: ASR vs Multimodal LLMs: What Actually Won
Dedicated ASR plus a cleanup model is quietly beating end-to-end dictation on the hardware you already own.
#6024: What's Actually Inside the KV Cache?
890 bytes per token: why the KV cache is both what makes LLMs practical and what makes long context expensive.
#6021: Why Chat Templates Break Models Silently
Every chat API flattens your messages into special tokens. Get one delimiter wrong and the model just gets quietly dumber.
#6016: What 70 Billion Parameters Actually Means
A single artificial neuron, built from scratch — then the hard part: does that neuron analogy still describe a real transformer?
#6014: Why Every AI Model Is Secretly a Decoder
Encoder, decoder, or both? The three transformer shapes, why one won, and why BERT still refuses to die.
#6013: What Happens Inside One LLM Forward Pass
One trip through the network, layer by layer — and why a classifier needs one pass while an LLM needs hundreds.
#6012: Inside the Decode Loop: How LLMs Generate One Token at a Time
The model doesn't choose tokens or decide when to stop — the engine does. A step-by-step tour of one decode iteration.
#6011: Inside LLM Inference: Prefill, Decode, and the KV Cache
One 8K-token sequence on a 13B model eats 6.7GB of KV cache—while the weights are only 26GB. Here's what actually happens per request.
#6009: Vertex AI Is Now the Gemini Agent Platform
Vertex AI just became the Gemini Enterprise Agent Platform. Here's the family tree, the five products, and when to actually move.
#6007: What a $1M AI Inference Contract Actually Contains
The sticker price isn't the deal. Inside the commit floors, discount curves, and overage traps of enterprise AI contracts.
#6006: Data Residency vs. Inference Residency: What the Law Actually Says
GDPR doesn't require local GPUs. Here's what actually drives geographically restricted AI inference — and the three layers everyone conflates.
#6005: What's Actually Behind a Government AI Chatbot
Singapore runs open-weight models on commercial cloud. The UK built its own chatbot, then shut it down for Copilot. What's really behind civil-serv...
#6004: Nobody Can Buy Unlimited AI Inference
Every published tier, every reserved deal, every prepaid reservation — all bounded. Even Azure says it out loud: quota isn't capacity.
#6003: What Enterprises Actually Pay for AI Inference
Twelve and a half billion dollars in enterprise API spend — and nobody can tell you who spent it. Inside AI's most undocumented market.
#6002: Claude on AWS: Three Doors, One Rate Card
Same per-token price everywhere — so why buy Claude through AWS instead of Anthropic? The answer is commitments, compliance, and silicon.
#6001: What Do Inference Providers Actually Sell?
Same open weights, 20x cost gap. What Fireworks, Together, and Groq actually sell — and when to just rent a GPU.
#6000: 6,000 Episodes and a GPU Bill: Inside My Weird Prompts
Six thousand episodes, one human in the loop, and a Modal bill that keeps growing. A behind-the-scenes update on the podcast.