Local AI
394 episodes · Page 4 of 17
#6253: Why Context Windows Stalled at a Million Tokens
A dense 70B model requires about 328 gigabytes of KV cache to serve a single one-million-token prompt, exceeding the memory of an Nvidia B300 chip.
#6245: Why Inference Engines Change Benchmark Scores
A study found a 16.6 percentage point spread on GSM8K across different inference engines running identical weights on the same GPU.
#6244: When Your Fine-Tune Isn't Done Learning Yet
Falling loss, 90% word error rate, and a model that still says "Derry gym lag." What undertraining actually looks like — and when more epochs won't...
#6242: Training a Model on Your Own Taste
What does it actually take to train a small model on your editorial taste using pairwise preferences and DPO?
#6238: Which AI Models Actually Generate Good Training Data?
Open weights vs closed APIs for synthetic data, and why IFEval scores don't tell you which model will hold up.
#6237: Small Models, Big Questions: When Cheap Fables Get Expensive
Three million fables for pennies — but can a small model tell a good example from a plausible one?
#6229: Voice Design: Casting Ogres and Radio Voices
Text-described voices vs. reference audio, and which TTS knobs change who a voice is versus how it sounds.
#6228: What 12GB of VRAM Can Actually Do Now
Local AI on a mid-range AMD card: what fits, what's fast, and what's still out of reach in 2026.
#6223: How Many Epochs Does Fine-Tuning Actually Need?
Epochs are a receipt, not a plan. How to pick a training ceiling, read the curves, and know when to stop.
#6222: Why Fine-Tuning Runs Can't Always Resume
Eighty cents and twenty-five minutes per training run — so why does the checkpoint question still keep a tinkerer up at night?
#6220: Mixture of Experts: Bigger Than It Looks
MoE models store a trillion parameters but only fire a few billion per token. Here's what the active number really measures — and why smaller isn't...
#6219: Dictation Beyond the Keyboard
Pedals, push-to-talk, and Android gesture apps promise keyboard-free dictation — but can any of it become a dependable daily setup?
#6218: Training a LoRA to Write Like You
A Humanizer model stripped every trace of the author. The fix isn't a better Humanizer — it's a LoRA trained on your own writing.
#6217: Fine-Tuning a Trillion-Parameter Model at Home
What does it actually take to full fine-tune a trillion-parameter model? Sixteen terabytes of optimizer state, hundreds of GPUs, and a very big bill.
#6216: What Do You Call the Middle Tier Model?
Sonnet, Sol, Flash, Medium — the daily-driver models nobody can name, and why they're squeezed from above and below.
#6212: Woolly Sedate Snail and Other Model Failures
Most of Hugging Face's 2.96M models sit unread. But the dust has a personality — snails, pirates, and a matchbox that resigns.
#6211: Attention Is Still Winning: The Hybrid Model Census
Mamba layers shipped — inside hybrids. A census of Qwen, Nemotron, Kimi, Granite, and why MiniMax went back to full attention.
#6207: Two WERs, One Model: Inside ASR Eval Harnesses
Same model, same audio, two different word error rates. The gap lives in the normaliser — and that's just layer two of three.
#6205: Where Should Your Notebook Actually Run?
Colab's free GPU comes with a 12-hour wall and hidden secrets in your saved outputs. Here's where notebooks should actually live.
#6201: Inside Cloudflare's Clef: A Language Model That Can't Speak
Cloudflare's Clef reads text, images, and video but outputs only probabilities — no words. Here's what's actually under the hood.
#6198: Benchmarking a TTS Model on Your Own GPU
Real-time factor, warm-up artifacts, and why an 8-bit quantized speech model might not run any faster.
#6196: Fine-Tuning ASR: When Your Model Memorizes You
Your fine-tuned speech model aces your training recordings — but does it actually work on fresh audio?
#6194: Teaching a TTS Model His Own Accent
One man fine-tuned a text-to-speech model on his own voice to get Irish-accented English that can say Hebrew words mid-sentence.
#6192: What Actually Happens When You Train a Model
Data in, model out — except the "data" is pairs, the split is the whole game, and one leaked speaker can fake your score.