AI Inference
554 episodes · Page 3 of 24
#6310: Why Model APIs Lack Dedicated Few-Shot Fields
Zou and colleagues found models suffered steep drops at 16,000 tokens on tasks requiring learning across an entire demonstration set.
#6309: Why DeepMind Systems Live Outside the Gemini API
AlphaEvolve saved 0.7% of Google's compute, but running it requires an enterprise cloud environment to evaluate candidate code.
#6302: Why Fine-Tuning Whisper Won't Stop Duplicated Text
Whisper evaluates transcripts against a gzip compression threshold of 2.4, rejecting repetitive text before retrying at higher temperatures.
#6295: When Splitting Serverless Apps Earns Its Keep
Modal apps enforce atomic deployment, meaning all grouped functions redeploy in place together while scaling independently.
#6289: Why Hermes and Pi Top the OpenRouter Coding Table
Hermes Agent and pi logged 1.66 trillion and 1.41 trillion tokens on OpenRouter, topping Claude Code despite fundamentally opposite designs.
#6288: What Happens Between Model Checkpoints
Extending reinforcement learning on Olmo 3 Think for 21 days yielded a 20-point leap on IFBench without starting fresh pretraining.
#6287: Why Model Names Conceal Changes Behind the API
One API identifier spanned ten distinct releases over nineteen months without changing the name submitted by the customer.
#6278: Why Model Benchmarks Miss Small Workflow Friction
Switching Claude Code to Sonnet broke an everyday shell alias request that Opus solved simply by recognizing where to look in local context.
#6274: Machine Learning for Smells, Whale Codas, and Aftershocks
A graph neural network mapped molecular structures to scents, matching a consensus panel better than the typical individual human nose.
#6273: How Timestral Forecasts Time Series Data
Mistral's open-weight Timestral models predict up to 1,024 future steps zero-shot using grouped patches across twenty-one non-crossing quantiles.
#6272: Teaching Audio Models to Hear Beyond Words
Speech recognition rewards ignoring background noise, but audio-language models train specifically on environmental sounds and temporal scenes.
#6265: Why Upgrading Embedding Models Demands Reindexing
Cohere's Embed Five Fast queries against a Pro index reach 98.4 percent of performance without regenerating stored document vectors.
#6263: Why Numeric Data Lacks a Default Embedding Model
Popular time-series models like IBM's Granite TSPulse forecast trends rather than provide canonical vectors for similarity search.
#6257: Where Your Prompts Actually Go
Selecting US-only inference on Anthropic's direct API incurs a 1.1x multiplier across token categories, while global routing is the standard rate.
#6253: Why Context Windows Stalled at a Million Tokens
A dense 70B model requires about 328 gigabytes of KV cache to serve a single one-million-token prompt, exceeding the memory of an Nvidia B300 chip.
#6245: Why Inference Engines Change Benchmark Scores
A study found a 16.6 percentage point spread on GSM8K across different inference engines running identical weights on the same GPU.
#6243: Inside the Mixture of Experts Router
What actually decides which expert handles your token? A look inside the router, from sigmoid affinities to load balancing.
#6238: Which AI Models Actually Generate Good Training Data?
Open weights vs closed APIs for synthetic data, and why IFEval scores don't tell you which model will hold up.
#6237: Small Models, Big Questions: When Cheap Fables Get Expensive
Three million fables for pennies — but can a small model tell a good example from a plausible one?
#6228: What 12GB of VRAM Can Actually Do Now
Local AI on a mid-range AMD card: what fits, what's fast, and what's still out of reach in 2026.
#6223: How Many Epochs Does Fine-Tuning Actually Need?
Epochs are a receipt, not a plan. How to pick a training ceiling, read the curves, and know when to stop.
#6220: Mixture of Experts: Bigger Than It Looks
MoE models store a trillion parameters but only fire a few billion per token. Here's what the active number really measures — and why smaller isn't...
#6217: Fine-Tuning a Trillion-Parameter Model at Home
What does it actually take to full fine-tune a trillion-parameter model? Sixteen terabytes of optimizer state, hundreds of GPUs, and a very big bill.
#6216: What Do You Call the Middle Tier Model?
Sonnet, Sol, Flash, Medium — the daily-driver models nobody can name, and why they're squeezed from above and below.