AI Inference
554 episodes · Page 6 of 24
#5995: NVIDIA ASR: Parakeet, Canary, and Nemotron Explained
Whisper gives you one dial. NVIDIA gives you five. A practical map of the Parakeet, Canary, and Nemotron speech models.
#5991: Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cpp
Hosted batch APIs sell you half-price tokens. Here's how to get the same file-in, file-out workflow on your own Linux box.
#5981: Batch Inference: Four Doors, Half Price
Same tokens, half the price — and almost nobody using it properly. A tour of batch inference from OpenAI to OpenRouter.
#5975: Merging vs Distillation: Can Two Models Become One?
Merging and distillation get lumped together, but they're opposites. One composes existing capability; the other has to teach it.
#5974: One Model or Two? ASR Cleanup Explained
If speech models can punctuate and capitalize on their own, why does a separate 484MB cleanup model exist at all?
#5969: Ubuntu 26.04 LTS: Is Staying on LTS Still a Trade-Off?
Ubuntu 26.04 LTS shipped in April. Point releases, HWE kernels, and SRUs have quietly closed most of the freshness gap — but not everywhere.
#5964: Containers vs VMs for GPU Inference: What You Actually Pay
Containers cost 0.03%. Passthrough VMs cost 1%... or 36%. Same hardware, different systems — here's how to tell which you'll get.
#5963: Running One Model Across Mismatched GPUs
Tensor, pipeline, or offload? What actually happens when you run one model across several GPUs — including mismatched ones.
#5961: What Your GPU Is Actually Waiting On
Your GPU says 100% utilized. It's multiplying zeros. Here's what the counters actually measure during inference.
#5960: There's No Such Thing as Tokens Per Second
Why one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working.
#5959: What RTF Actually Tells You About Speech Models
Accuracy gets the headlines, but speed decides what ships. How RTF, latency percentiles, and memory shape real ASR deployments.
#5948: LangGraph, LangChain, and Deep Agents in 2026
Runtime, framework, harness — how three LangChain layers actually fit together, and what changed in 2026.
#5944: Keeping the Rollout Engine Fed: Inside RLHF Pipelines
Most of an RLHF run is spent waiting on generation, not gradients. Here's how TRL, OpenRLHF, and verl actually differ.
#5938: One Regex Rule Worked, One Was Impossible
A dictation app, two text-replacement rules, and the mathematical line between what regex can match and what it provably cannot.
#5936: Training a Model When You Don't Know the Labels Yet
Daniel wants to extract adapter specs from his shelf — but he can't name the labels until he's already annotating.
#5903: Video Tokens, Omni, and the Local VRAM Math
Video costs ~100 tokens per second. That single number explains why ChatGPT hides video upload and why local inference hits a KV cache wall.
#5900: Serving DeepSeek-V4.1-Flash on Your Own GPU
Five inference engines, one frontier MoE, and a shortlist that collapses depending on which GPU you already bought.
#5899: Same GPU, Same Weights, Different Answers
Identical hardware, identical weights, different engine — and answers that shift by up to 16.6 points.
#5898: ROCm 10.1: AMD's Software Stack Finally Grows Up
Three ROCm releases in six months — build system, agentic tooling, data movement. What actually changed for AMD GPU owners.
#5870: Homebox Fork's Two Backend Scripts Need a Real Job Runner
A Homebox fork has two incremental scripts — WebP conversion and AI image enrichment — and no scheduler built to run them safely.
#5860: Who Can Actually Train an LLM From Scratch?
Only a handful of organizations can truly train a model from random weights. Here's why nobody can count them.
#5859: Slow Labs, Fast Ships: What Release Cadence Actually Tells Us
Anthropic ships premium models every 22.5 days — the fastest lab in the industry. So much for the slow-and-careful story.
#5858: Routing AI Models: When 3 Rules Beat a Trained Router
A deep dive into AI model routing — and why a learned router may not beat three rules you wrote by hand.
#5857: Building a Scorecard for Our Own AI Writers
Daniel wants a real scorecard for the agent that writes this show — repetition, dialogue quality, and whether the retrieval harness actually works.