AI Inference
554 episodes
#6517: Why GPU Wiring Dictates Model Parallelism
Crossing from NVLink inside a server to InfiniBand between servers drops available bandwidth by a factor of eighteen.
#6514: Why Even Solo Developers Face a GPU Shortage
H100 rental contracts jumped nearly 40 percent in five months as on-demand GPU capacity sold out across legacy and cutting-edge hardware.
#6513: Why Dual-Socket Servers Win on Dense AI Models
Populating a second EPYC socket sped up dense LLM inference by 183 percent, but delivered only 102 percent on sparse models like DeepSeek R1.
#6510: Voice Isolation Before Speech Recognition
Running voice isolation on clean speech without a competing talker increases transcription word error rates from 0.04 percent to nearly three percent.
#6508: Why Speech Transducers Omit Common Symbols
A transducer-style speech model cannot learn to emit an at-sign through fine-tuning if the character is missing from its fixed output vocabulary.
#6503: Why Tokenizer Vocabulary Size Is a Hardware Choice
Inference-optimal vocabulary size shifts sixteen-fold depending on serving batch size, from 32,000 at batch one to 524,000 at batch sixty-four.
#6502: What a Cross-Entropy Loss of 1.8 Actually Means
A cross-entropy loss of 1.8 nats means a language model is about as uncertain as choosing uniformly among six equally likely tokens.
#6501: Splitting Frontier AI Across Thousands of GPUs
A 405-billion-parameter model requires 810 gigabytes just for weights, yet an H100 GPU holds only 80.
#6499: Why Simple Rules Beat Value Models in Reasoning RL
DeepSeek R1-Zero boosted AIME accuracy from 15.6% to 77.9% using only two deterministic checks and no learned critic.
#6497: Why Speech Recognition Confidence Scores Lie
Cross-entropy loss rewards saturation, leaving speech recognition models to assign winning tokens over 0.9 probability even when they are guessing.
#6496: How Speech Models See Audio as an Image
Modern speech recognition models like Whisper never hear raw audio; they inspect an eighty-pixel-tall grayscale image of the sound.
#6484: Why Narrow Speech Tasks Still Need Large Models
NVIDIA experiments show speech models trained from scratch on target domains had nearly double the error rate of fine-tuned pretrained models.
#6468: Why Tiny Neural Gates Replaced Voice Activity Gates
Legacy WebRTC voice detection misses sixty-four percent of speech frames, while modern neural models compress full memory into a single megabyte.
#6465: Why Modern AI Models Cut Most Attention Layers
Frontier models like Kimi K3 now replace three out of four attention layers with delta-rule recurrence to curb massive KV caches.
#6464: Teaching Speech Recognizers to Stay Quiet
Masking just three self-attention heads in Whisper large-v3 cuts hallucinations on non-speech audio by over eighty percent.
#6463: Why Training Speech Recognition From Scratch Fails
Thirty-two hours of speech provides just three hundred thousand words of text, making from-scratch speech models struggle with basic spelling.
#6462: When Adding a Reranker Hurts Search
NVIDIA benchmarked a cross-encoder upgrade that scored three points below its baseline, while other models surged fifteen.
#6453: Who Actually Won the Open Weight Race
With over four billion downloads, the most downloaded open-weight model in history is an embedding tool, not a chatbot.
#6451: Which AI Models Actually Last
OpenAI kept the gpt-3.5-turbo alias alive for over three years, while Google Gemini Flash iterations often turn over in just a few months.
#6448: Why Fast Model Microtraining Needs an Evaluation Gate
A speech model fine-tuning run can cost twenty-three cents in twenty minutes, but unmonitored updates silently erase older vocabulary.
#6444: Teaching Speech Models to Collapse Spelled Words
Fine-tuning formatting rules takes surprisingly little data: a Johns Hopkins study found truecasing accuracy jumps with only fifty documents.
#6443: Why Nvidia Artificially Halves Consumer Tensor Math
A community tool flipped two registers to restore full-rate FP32 accumulation on GeForce cards without overclocking a single clock cycle.
#6433: Why Nvidia Ships Nemotron When Hardware Is the Business
Nvidia spent a trillion distillation tokens on Nemotron Super VL not to top benchmarks, but to hit 4,856 tokens per second on a single GB200.
#6422: Why Multi-Stage AI Pipelines Collapse in Production
AWS multi-container inference pipelines cap chains at 15 stages but officially forbid GPUs due to instance resource contention.