Transformer Architecture
226 episodes · Page 3 of 10
#5626: What Multimodal Models Actually Learn From Video
Late fusion, early fusion, and vision laziness — what actually happens when models train on video, audio, and text together.
#5625: Omni and the Marble That Rolls Uphill
Google's Omni claims an intuitive grasp of gravity and momentum. Can a video model really learn physics, or just remember what physics looks like?
#5624: One Forward Pass: Building a Bounded Classifier
No generation, no parsing — just label scores in a single pass. How to build a small encoder classifier that actually works.
#5583: The Last Lighthouse Keepers: Life at the Edge of the Sea
Before automation, someone had to trim the wick, crank the clockwork, and keep the light burning through the storm. What was that job really like?
#5567: The Sinai Crossing Where a Stamp Draws the Border
A 24-hour pedestrian crossing between Eilat and Taba, where a free stamp, a one-kilometer fee waiver, and four treaty zones shape every trip into S...
#5561: Fog, Light, and the Limits of Sight
Fog is a cloud sitting on the road. Here's the physics of why it forms fast, what it does to light, and why there's no safe driving visibility.
#5520: Keeping a Fine-Tune Alive Across Model Releases
Daniel wants to fine-tune DeepSeek Flash 4.1 on edited podcast scripts. The hard part isn't training — it's surviving the next release.
#5447: The Models That Never Talk Back
Some models read text, score it, and return a number. No chat, no reasoning, just decisions — and they're running under every router you use.
#5443: Heads vs Layers: How Model Merging Actually Works
Heads aren't the Lego bricks of model merging — layers are. Here's what heads really do and how frankenmerges get built.
#5442: When Do LLMs Suddenly Get Good at Things?
New capabilities seem to appear out of nowhere as models scale. Whether that's real — or just how we measure — is still an open fight.
#5438: Why Linguists Left the AI Room
Large language models grew out of linguistics — so why aren't linguists in the room where they're built?
#5431: The Other Half of Hugging Face: Why BERT Still Out-Downloads Llama
Encoder models pull over a billion downloads a month. Decoder models pull 397 million. The AI conversation and the download counter are describing ...
#5430: Two Boxes: ASR and the Text Fixer Behind It
Punctuation, casing, ITN, disfluency — the four-job layer between raw ASR output and text you can actually read.
#5419: The Attention Budget in Your Pocket
Why phone dictation runs out of room, and how bounded attention windows buy you punctuation without blowing your memory budget.
#5410: Adapters: 102KB That Reshapes a 403GB Model
A 102KB adapter file changes how a 403GB base model behaves — without ever merging into it. Here's how model adapters actually work.
#5408: When Small NLP Models Beat the LLM
Feature extraction, fill-mask, token classification — the classic NLP tasks still have a job. Here's when a small model beats a frontier API.
#5396: Teaching a Small Model to Stop Spelling Out Numbers
Your ASR pipeline is fine until someone dictates "three point two" and gets "three point two" spelled out. Here's how inverse text normalization ac...
#5289: Why World's Fairs Built Absurd Transit
Moving sidewalks, monorails, and robot boxers: what we lost when technology's grand public spectacles became trade shows.
#5188: DeepSeek's Point Release That Isn't
DeepSeek shipped a whole new architecture and called it a point release. Here's what actually changed inside the model.
#5184: AI Is a Number Factory, Not a Wordsmith
Under the prose, every AI model is just matrices of floating-point math. So where does the randomness actually come from?
#5182: DeepSeek V4.1 Flash: 1M Context, 437x Smaller KV Cache
DeepSeek V4.1 Flash landed with a 1M-token window and a KV cache 437x smaller than V1. Here's what actually changed — and why the middle of your co...
#5089: How Runway Overruns Happen and the EMAS Safety Net
A Prime Air 767 overran at Miami. We break down the physics, the safety engineering, and why EMAS beds matter.
#5041: GPT-6 Astra, 100K GPUs, and the AGI Question
OpenAI's GPT-6 Astra arrives with an AGI claim and a 100K-GPU training run. We break down what Stargate LLC actually is and whether the scale justi...
#4999: Why AI Can't Just Put a Circle on Your Photo
Why multimodal AI models can't simply circle a Dremel bit without rebuilding your entire image from scratch.