Transformer Architecture
226 episodes
#6517: Why GPU Wiring Dictates Model Parallelism
Crossing from NVLink inside a server to InfiniBand between servers drops available bandwidth by a factor of eighteen.
#6508: Why Speech Transducers Omit Common Symbols
A transducer-style speech model cannot learn to emit an at-sign through fine-tuning if the character is missing from its fixed output vocabulary.
#6503: Why Tokenizer Vocabulary Size Is a Hardware Choice
Inference-optimal vocabulary size shifts sixteen-fold depending on serving batch size, from 32,000 at batch one to 524,000 at batch sixty-four.
#6502: What a Cross-Entropy Loss of 1.8 Actually Means
A cross-entropy loss of 1.8 nats means a language model is about as uncertain as choosing uniformly among six equally likely tokens.
#6501: Splitting Frontier AI Across Thousands of GPUs
A 405-billion-parameter model requires 810 gigabytes just for weights, yet an H100 GPU holds only 80.
#6496: How Speech Models See Audio as an Image
Modern speech recognition models like Whisper never hear raw audio; they inspect an eighty-pixel-tall grayscale image of the sound.
#6494: Where Speech Models Actually Learn
Just three self-attention heads out of twenty in Whisper's decoder trigger over 75 percent of hallucinations on non-speech audio.
#6469: Why Switching Off One Attention Head Changes Everything
Masking just three decoder heads in Whisper drops hallucination on silent audio clips from 99.97% down to 24.1%.
#6465: Why Modern AI Models Cut Most Attention Layers
Frontier models like Kimi K3 now replace three out of four attention layers with delta-rule recurrence to curb massive KV caches.
#6444: Teaching Speech Models to Collapse Spelled Words
Fine-tuning formatting rules takes surprisingly little data: a Johns Hopkins study found truecasing accuracy jumps with only fifty documents.
#6433: Why Nvidia Ships Nemotron When Hardware Is the Business
Nvidia spent a trillion distillation tokens on Nemotron Super VL not to top benchmarks, but to hit 4,856 tokens per second on a single GB200.
#6384: Why LLM Output Tokens Cost More Than Input
Decode computes at just 1.4% utilization on an H100 because reading 140 gigabytes of weights takes 280 times longer than the arithmetic.
#6379: Porting Voice Models to Core ML and MLX
Core ML packages separate computation specs from raw weights, while MLX runs on CPU and GPU without touching the Neural Engine.
#6368: What Does It Take To Train An AI Model From Scratch?
nanochat trains a BPE tokenizer in Rust in roughly 34 seconds, yet it can quietly disadvantage languages like Korean by nearly half.
#6345: How vLLM Squeezes More Requests into GPU Memory
Early serving engines wasted up to 80% of KV cache memory on unspent reservations; vLLM cut that slack to under 4% using virtual memory paging.
#6311: How Transformers Process Images and Video
A Vision Transformer divides a standard image into a grid of 16-by-16 pixel patches rather than processing each individual pixel as a token.
#6288: What Happens Between Model Checkpoints
Extending reinforcement learning on Olmo 3 Think for 21 days yielded a 20-point leap on IFBench without starting fresh pretraining.
#6273: How Timestral Forecasts Time Series Data
Mistral's open-weight Timestral models predict up to 1,024 future steps zero-shot using grouped patches across twenty-one non-crossing quantiles.
#6253: Why Context Windows Stalled at a Million Tokens
A dense 70B model requires about 328 gigabytes of KV cache to serve a single one-million-token prompt, exceeding the memory of an Nvidia B300 chip.
#6245: Why Inference Engines Change Benchmark Scores
A study found a 16.6 percentage point spread on GSM8K across different inference engines running identical weights on the same GPU.
#6243: Inside the Mixture of Experts Router
What actually decides which expert handles your token? A look inside the router, from sigmoid affinities to load balancing.
#6220: Mixture of Experts: Bigger Than It Looks
MoE models store a trillion parameters but only fire a few billion per token. Here's what the active number really measures — and why smaller isn't...
#6211: Attention Is Still Winning: The Hybrid Model Census
Mamba layers shipped — inside hybrids. A census of Qwen, Nemotron, Kimi, Granite, and why MiniMax went back to full attention.
#6191: What Actually Happens During Fine-Tuning
Sixteen bytes per parameter in, two out — a walk through the forward pass, backward pass, and optimizer that turn a dataset into new weights.