#transformers
32 episodes
#5520: Keeping a Fine-Tune Alive Across Model Releases
Daniel wants to fine-tune DeepSeek Flash 4.1 on edited podcast scripts. The hard part isn't training — it's surviving the next release.
#5443: Heads vs Layers: How Model Merging Actually Works
Heads aren't the Lego bricks of model merging — layers are. Here's what heads really do and how frankenmerges get built.
#5438: Why Linguists Left the AI Room
Large language models grew out of linguistics — so why aren't linguists in the room where they're built?
#5431: The Other Half of Hugging Face: Why BERT Still Out-Downloads Llama
Encoder models pull over a billion downloads a month. Decoder models pull 397 million. The AI conversation and the download counter are describing ...
#5188: DeepSeek's Point Release That Isn't
DeepSeek shipped a whole new architecture and called it a point release. Here's what actually changed inside the model.
#5182: DeepSeek V4.1 Flash: 1M Context, 437x Smaller KV Cache
DeepSeek V4.1 Flash landed with a 1M-token window and a KV cache 437x smaller than V1. Here's what actually changed — and why the middle of your co...
#4667: How Transformers Killed the Robot Voice
From espeak's robotic squawk to neural voices with added "ums" — how transformers made speech synthesis human.
#4169: Change Detection AI for Construction Sites
How AI catches what experts miss in construction site photos — from Siamese networks to transformer models.
#4057: How to Fix AI's Bullet Point Addiction
Why AI models default to bullet points and how textual LoRAs achieve 94% prose adherence.
#2622: How Transformers Actually Work: Attention, Tokens, and Context
How one architectural change unlocked chatbots, image generation, and protein folding — explained without the jargon.
#2408: How Backpropagation Actually Unlocks Neural Networks
How error signals flow backward through networks to make learning possible — and why "it's just calculus" misses the point.
#2374: How Granular Can MoE Experts Get?
Exploring the limits of expert granularity in Mixture of Experts models—how narrow can segmentation go before efficiency or accuracy suffers?
#2366: Why LLMs Forget the Middle of Long Conversations
Why do large language models struggle with the middle of long conversations? Explore the science behind attention dilution and practical fixes.
#2350: NVIDIA's Strategic Pivot: From Chipmaker to Model Builder
Dive into NVIDIA’s Nemotron 3 Super, a hybrid MoE model combining Mamba, Transformers, and multi-token prediction for cutting-edge efficiency.
#2348: Diffusion Models Take on Text Generation
Explore Inception Labs’ Mercury 2, a groundbreaking diffusion-based language model that rethinks text generation and reasoning.
#2066: The Transformer Trinity: Why Three Architectures Rule AI
Why did decoder-only models like GPT dominate AI, while encoders and encoder-decoders still hold critical niches?
#2062: How Transformers Learn Word Order: From Sine Waves to RoPE
Transformers can’t see word order by default. Here’s how positional encoding fixes that—from sine waves to RoPE and massive context windows.
#2061: The Memory Bottleneck That Drives Attention Design
Attention is the engine of modern AI, but it’s also a memory hog. Here’s how MQA, GQA, and MLA evolved to fix it.
#2056: Music as Language: The Architecture Behind AI Song Generation
A look at how AI music models use audio tokens, transformers, and diffusion to turn text into songs.
#1799: The Original AI Blueprints: BERT & CLIP
Before GPT, two models changed everything. Discover how BERT and CLIP taught machines to read and see the world.
#1679: Efficiency Over Scale: How Export Controls Forced a Smarter AI
DeepSeek and MiMo are topping developer charts, but they're not just cheaper clones. Here's why their design philosophy is fundamentally different.
#1666: The Agent Mesh: Shared Context That Changes Everything
Grok 4.20’s native multi-agent architecture cuts token costs by 75% and enables real-time cross-agent reasoning.
#1633: Can a Character Actor Model Beat a Generalist?
We grill MiniMax M2.7 to see if a model built for "virtual companions" can actually handle high-level comedy and complex character logic.
#1632: Agent Interview: DeepSeek V three point two
We interview DeepSeek V3 to see if this open-weight powerhouse can handle weird podcast prompts better than big tech’s flagship models.