GPUS
228 episodes
#6217: Fine-Tuning a Trillion-Parameter Model at Home
What does it actually take to full fine-tune a trillion-parameter model? Sixteen terabytes of optimizer state, hundreds of GPUs, and a very big bill.
#6211: Attention Is Still Winning: The Hybrid Model Census
Mamba layers shipped — inside hybrids. A census of Qwen, Nemotron, Kimi, Granite, and why MiniMax went back to full attention.
#6205: Where Should Your Notebook Actually Run?
Colab's free GPU comes with a 12-hour wall and hidden secrets in your saved outputs. Here's where notebooks should actually live.
#6198: Benchmarking a TTS Model on Your Own GPU
Real-time factor, warm-up artifacts, and why an 8-bit quantized speech model might not run any faster.
#6106: なぜ日本は今も光ディスクを手放さないのか
新幹線を走らせる国が、いまだにブルーレイを買い続ける。最先端技術の国に残る、意外と合理的な理由を探ります。
#6070: The Papers That Moved AI: From AlexNet to This Year's Fights
The classics all said 'learn it from data and scale it up'. This year's most-discussed AI papers are auditing what scaling took for granted.
#6024: What's Actually Inside the KV Cache?
890 bytes per token: why the KV cache is both what makes LLMs practical and what makes long context expensive.
#6013: What Happens Inside One LLM Forward Pass
One trip through the network, layer by layer — and why a classifier needs one pass while an LLM needs hundreds.
#6012: Inside the Decode Loop: How LLMs Generate One Token at a Time
The model doesn't choose tokens or decide when to stop — the engine does. A step-by-step tour of one decode iteration.
#6011: Inside LLM Inference: Prefill, Decode, and the KV Cache
One 8K-token sequence on a 13B model eats 6.7GB of KV cache—while the weights are only 26GB. Here's what actually happens per request.
#6006: Data Residency vs. Inference Residency: What the Law Actually Says
GDPR doesn't require local GPUs. Here's what actually drives geographically restricted AI inference — and the three layers everyone conflates.
#6004: Nobody Can Buy Unlimited AI Inference
Every published tier, every reserved deal, every prepaid reservation — all bounded. Even Azure says it out loud: quota isn't capacity.
#6001: What Do Inference Providers Actually Sell?
Same open weights, 20x cost gap. What Fireworks, Together, and Groq actually sell — and when to just rent a GPU.
#5992: Ridiculous But Legal Ways to Get RAM for AI
Memory prices are up 485% on some kits. Here are ten ridiculous — but completely legal — ways to get RAM for AI without the markup.
#5991: Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cpp
Hosted batch APIs sell you half-price tokens. Here's how to get the same file-in, file-out workflow on your own Linux box.
#5981: Batch Inference: Four Doors, Half Price
Same tokens, half the price — and almost nobody using it properly. A tour of batch inference from OpenAI to OpenRouter.
#5969: Ubuntu 26.04 LTS: Is Staying on LTS Still a Trade-Off?
Ubuntu 26.04 LTS shipped in April. Point releases, HWE kernels, and SRUs have quietly closed most of the freshness gap — but not everywhere.
#5966: How Much of Your Linux Desktop Is Actually Misconfigured?
Most "Linux tweaks" are folklore. A few are real — and one toggle most people never open is worth up to 42%.
#5964: Containers vs VMs for GPU Inference: What You Actually Pay
Containers cost 0.03%. Passthrough VMs cost 1%... or 36%. Same hardware, different systems — here's how to tell which you'll get.
#5963: Running One Model Across Mismatched GPUs
Tensor, pipeline, or offload? What actually happens when you run one model across several GPUs — including mismatched ones.
#5961: What Your GPU Is Actually Waiting On
Your GPU says 100% utilized. It's multiplying zeros. Here's what the counters actually measure during inference.
#5960: There's No Such Thing as Tokens Per Second
Why one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working.
#5959: What RTF Actually Tells You About Speech Models
Accuracy gets the headlines, but speed decides what ships. How RTF, latency percentiles, and memory shape real ASR deployments.
#5955: Work Lights: Lumens, Lux, and the Specs That Matter
Lumens is the easiest spec to print and the least useful on its own. Here's what to read instead when buying a work light.