AI Inference
554 episodes · Page 2 of 24
#6421: Why Diarized Transcription Requires a Cascade
Open-source speech tools cannot transcribe and identify speakers in a single call, forcing pipelines to rely on timestamp-overlap math.
#6409: Why Speech Models Break at Chunk Seams
Shifting an audio transcription window by just one second can lose entire words or drop capitals mid-sentence without changing the model weights.
#6387: What Really Runs When You Compact an Agent's Context
Commercial compaction relies on main models and prompts, while purpose-built fine-tunes like Compactor-Qwen3.5-4B remain rare.
#6384: Why LLM Output Tokens Cost More Than Input
Decode computes at just 1.4% utilization on an H100 because reading 140 gigabytes of weights takes 280 times longer than the arithmetic.
#6383: Choosing Class Breaks for Skewed Price Data
In one dimension, k-means clustering avoids random restarts entirely and runs in polynomial time via dynamic programming.
#6382: Choosing AI Models on Cost and Performance
Model prices vary thirty-eight-fold per index point, meaning ratio formulas love cheap options regardless of actual quality.
#6380: Choosing the Right Voice Activity Detector
A compact model file can hide a massive runtime, while Cobra bundles its proprietary detector and engine into just 450 kilobytes.
#6379: Porting Voice Models to Core ML and MLX
Core ML packages separate computation specs from raw weights, while MLX runs on CPU and GPU without touching the Neural Engine.
#6378: Running Speaker Identification Locally on Phones
Exported voice embedding models like CAM++ run offline on mobile devices in packages as small as twenty-eight megabytes.
#6375: Why Modern AI Rests on Statistics
When polygenic risk scores derived from European data are applied globally, prediction accuracy drops to 25 percent in African populations.
#6369: The Most Popular GPUs - Today's Cloud Demand
RunPod charges about $1.34 an hour for serverless RTX 4090 access, while a community pod runs for roughly 34 cents.
#6368: What Does It Take To Train An AI Model From Scratch?
nanochat trains a BPE tokenizer in Rust in roughly 34 seconds, yet it can quietly disadvantage languages like Korean by nearly half.
#6366: Why Training Recipes Matter More Than Model Size
Altering learning-rate timing alone improved global word error rates by twelve absolute percentage points in speech recognition experiments.
#6355: Why Model Downgrades Are Rarely About Quantization
A caching bug and a shift to medium reasoning effort once reduced Anthropic internal benchmark scores by about three percent.
#6349: Splitting Model Layers Across the Internet
Splitting model layers over a wide-area network yields under two tokens per second until scheduling, speculation, and pipelining step in.
#6347: Splitting LLM Layers Across AMD and Nvidia GPUs
A two-node llama.cpp setup running a 70B model jumps from three tokens per second on gigabit Ethernet to six on 2.5-gigabit.
#6345: How vLLM Squeezes More Requests into GPU Memory
Early serving engines wasted up to 80% of KV cache memory on unspent reservations; vLLM cut that slack to under 4% using virtual memory paging.
#6343: Why Live Video Generation Still Struggles With Timing
A streaming video model can render fifty-eight frames a second while still delivering pictures that are noticeably late.
#6341: Why Fast Speech Detectors Still Struggle with Onsets
Aalto and Genelec experiments reveal human ears can detect sound timing shifts down to half a millisecond at certain frequencies.
#6317: Choosing Model Precision, Quantization, and Format
Quantizing Qwen 2.5 9B to Q4_K_M boosted generation speed 2.7 times over BF16, while prompt processing rates stayed roughly unchanged.
#6316: Why Protein Design Ignores Astronomical Counts
A hundred amino acids can form roughly 10 to the 130th possible sequences, far exceeding the number of atoms in the observable universe.
#6315: Inside China's Mixed-Language AI Models
Domestic Chinese models typically draw sixty to eighty percent of their training data from Chinese, leaving massive room for English and code.
#6314: Why Forgotten Runpod Workloads Keep Billing
Runpod stops pods at a zero balance, but attached volume disks double in cost when stopped and unattached pods vanish entirely.
#6312: Choosing Between Modal, RunPod, and Vast.ai
RunPod lists an H100 SXM at $3.99 an hour on Secure Cloud, but its Community marketplace drops it to $2.69.