Local AI
394 episodes · Page 7 of 17
#5903: Video Tokens, Omni, and the Local VRAM Math
Video costs ~100 tokens per second. That single number explains why ChatGPT hides video upload and why local inference hits a KV cache wall.
#5900: Serving DeepSeek-V4.1-Flash on Your Own GPU
Five inference engines, one frontier MoE, and a shortlist that collapses depending on which GPU you already bought.
#5899: Same GPU, Same Weights, Different Answers
Identical hardware, identical weights, different engine — and answers that shift by up to 16.6 points.
#5898: ROCm 10.1: AMD's Software Stack Finally Grows Up
Three ROCm releases in six months — build system, agentic tooling, data movement. What actually changed for AMD GPU owners.
#5873: Abliteration: How $5 Undoes a Model's Guardrails
A rank-one weight edit costs under $5 and strips refusal behavior from open models. Then an automated tool beat human experts at it.
#5868: Fine-Tuning Parakeet for Um-Free, Hebrish Speech
Parakeet runs great on a phone — but fine-tuning it happens on a GPU. Here's the actual recipe for dropping ums and catching Hebrew.
#5859: Slow Labs, Fast Ships: What Release Cadence Actually Tells Us
Anthropic ships premium models every 22.5 days — the fastest lab in the industry. So much for the slow-and-careful story.
#5858: Routing AI Models: When 3 Rules Beat a Trained Router
A deep dive into AI model routing — and why a learned router may not beat three rules you wrote by hand.
#5853: Who Actually Competes With Hugging Face?
2.96 million model repos, and 85.6% have under 200 downloads. What ModelScope, Kaggle, and Ollama actually do differently.
#5830: Serving Your Own Fine-Tuned Model in the Cloud
You fine-tuned an open-weight model. Now how does anyone actually talk to it? Dedicated GPUs vs serverless inference, and the math that decides it.
#5814: Who Remembers Where You Left Off?
mpv remembers. MPRIS, GstPlay, and Media3 don't. There's no standard for playback history.
#5804: Text Classifiers: Local Models vs Always-On Endpoints
A specialist classifier costs $5.70 per million messages. A general LLM costs $142. So why is the obvious product so hard to find?
#5624: One Forward Pass: Building a Bounded Classifier
No generation, no parsing — just label scores in a single pass. How to build a small encoder classifier that actually works.
#5604: Fixing Typos Without Rewriting the Prompt
Three different jobs hide in one "clean up my prompt" pipeline — and only one of them should ever go to a generative model.
#5520: Keeping a Fine-Tune Alive Across Model Releases
Daniel wants to fine-tune DeepSeek Flash 4.1 on edited podcast scripts. The hard part isn't training — it's surviving the next release.
#5485: Fine-Tuning Parakeet for Hebrew and Your Own Jargon
NVIDIA's Parakeet beats Whisper on Android — but can you teach it Hebrew, or just your own jargon? Two answers, one much happier.
#5481: Small Models, Big Guardrails: PII Detection in ChatGPT
A tiny 600M parameter model is quietly scanning your ChatGPT tool calls for PII. Here's how that class of guardian model actually works.
#5469: Open Source's New Map: India, Croatia, Ethiopia
One in three new GitHub developers now comes from a country outside 2020's top ten. What that shift means for open source.
#5467: Small Models, Big Schemas: When JSON Constraints Backfire
Small models plus strict JSON schemas should be a safe bet. A 15,000-generation study found the opposite.
#5464: Your Keyboard's Hidden Data Problem
Your keyboard knows your email address, your phrases, your habits — and you can't take any of it with you.
#5460: Four Small Models, One Android Phone: Does It Actually Work?
A chained on-device dictation pipeline — VAD, ASR, cleanup — and why "it feels smooth" isn't the same as knowing it works.
#5457: Pigeon Alerts on a Repurposed i5: Frigate Without the TPU
Daniel wants a pigeon notification, not a custom model. Frigate already ships with bird detection — the real work is masks, zones, and filters.
#5456: Parakeet vs Whisper: Picking a Phone ASR Model
Why Whisper loses on Android, why Parakeet v2 beat v3, and how to benchmark speech-to-text without any tooling.
#5443: Heads vs Layers: How Model Merging Actually Works
Heads aren't the Lego bricks of model merging — layers are. Here's what heads really do and how frankenmerges get built.