More than eight hours. That's the human labour to transcribe one hour of speech. Not machine time — a person, sitting there, for most of a working day, per recorded hour.
And that's the number sitting behind Daniel's prompt, because when your personal speech fine-tune underperforms, the default instinct is just to record more. Read more text, dictate more notes, pile up hours. Daniel's question is whether the model itself can tell you which hours are worth recording — that's active learning, the loop flipped so the model points at its own weak spots — and, more to the point, whether the signals it uses are trustworthy and whether any of it actually saves you anything versus collecting at random.
His framing is nice: not every recording teaches the model the same amount. Which sounds obviously right, and the literature is not obviously on its side.
It really isn't.
So let's start with what the loop actually is, because the pitch and the mechanism are not the same thing.
The mechanism is concrete. You start with a small labelled seed set — for Daniel that's the audio he's already transcribed, the hour or so behind his Whisper fine-tune. You train. Then you run the checkpoint over your pool of unlabelled audio, and every utterance gets a score from something called an acquisition function. Highest-scelling batch gets selected, a human transcribes it, it goes into the training set, you retrain, and you go around again for however many rounds you've budgeted.
And the score is the whole game. The loop is plumbing.
Mostly. There's one structural detail that matters before we even get to signals: batch mode isn't optional. Retraining a large model after adding one labelled sample is impractical — one sample gives you basically nothing — so real active learning selects batches, a fixed number of utterances per round. And the moment you take the top-scoring batch, you inherit a problem: if the model is uncertain about a particular kind of audio, the top hundred uncertain utterances are often a hundred near-copies of the same thing.
Which is where diversity terms come in.
"focusing both on informativeness and diversity reduces the chance of choosing duplicate samples, which could hinder learning progress." That's the design principle the whole modern literature converges on.
One terminology trap before we go on, because it'll bite us later.
Say it.
The papers use "informativeness," "uncertainty," and "confidence" almost interchangeably, because the classic path-probability methods treat low probability as high informativeness. Low confidence equals high information. That conflation is where half the confusion in this literature starts — people read "confidence-based selection" and think it means selecting what the model is confident about.
It's the opposite, and the fact that both readings are grammatically available in the same sentence tells you a lot about the field. So — the loop has one job: score the unlabelled audio. Everything hangs on whether that score is any good. Let's go through the signals in order, weakest to strongest.
Confidence first. Give it its due before we bury it.
It gets a real amount of due, actually. Least-confidence selection from model posteriors is the oldest family in the business — confidence measures for HMM systems go back to the early two thousands, and the Amazon work in the Vosk maintainer's survey reportedly found that the specific confidence formula doesn't matter. Even simple ones work.
So the formula is not the lever.
Which is either reassuring or deflating depending on how much of your weekend you were planning to spend on it.
I was planning to spend it horizontally, so I'm fine.
Then the objection that actually bites: deep networks are overconfident at the softmax layer. The two-stage Bayesian paper puts it plainly — overconfidence "questions the reliability of using path probabilities for uncertainty assessment." On a modern CTC or attention model, the path probabilities are not a calibrated measure of anything. The model saying "I'm ninety-four percent sure" and the model saying "I'm seventy percent sure" are not two different epistemic states.
That's the theoretical objection. What's the concrete failure?
Microsoft's GERM paper has the most quotable one. At a one percent selection rate — you're only taking the very tippy-top of the uncertainty ranking — "most utterances selected by the confidence-based approach are noise and garbage utterances that have extremely low confidence but have little value to improving the performance of the overall system."
Silence, clipping, crosstalk.
Silence, clipping, crosstalk, half-sentences you abandoned mid-word. The lowest-confidence audio in your personal pool is disproportionately not speech. And GERM — global entropy reduction maximisation, a smarter selection criterion — beat confidence-based selection by about two percent relative.
Two percent.
Two percent relative. After replacing the whole selection principle.
So the smartest method in the room beats the naive one by a rounding error.
At that operating point, yes. Now, committee — disagreement between models — and this one is consistently the best-supported signal in the literature. Long et al. at Interspeech twenty-eighteen combined confidence with a word matching error rate computed across heterogeneous neural networks — different architectures disagreeing about the same audio — and that beat single-model confidence. And the Vosk survey states it flatly: "committee-based confidence wins over simple single-model confidence." Vote entropy. He lists that finding twice, for two separate papers, which for a survey writer is the equivalent of underlining it.
The catch for a solo practitioner is that "train several models" is a horrible instruction.
Here's the trick, and it's elegant: you don't have to. MC-dropout gives you a committee from one checkpoint. You leave the dropout on at inference, run the same audio through T stochastic forward passes, and each pass is a slightly different committee member. The variation ratio is one minus the fraction of passes that agree with the modal prediction — if all T passes say the same thing, ratio near zero, model is sure; if they scatter, ratio climbs, model doesn't know. The recent two-stage Bayesian pipeline does exactly this, using MC dropout to approximate Bayesian inference.
Same checkpoint, no retraining, and you get disagreement for free.
For the cost of T forward passes, which on a personal fine-tune is nothing.
That's the one I'd actually use.
I suspected you would. Now the branch that sounds best and performs worst: gradient and loss-based signals. Expected gradient length is theoretically lovely — how much would this sample move the parameters if we trained on it — it's uncorrelated with confidence, which is what you want in a second opinion, and it wins in experiments. Not significantly, but it wins.
It's the most honest sentence in the survey. The loss-prediction variant is where it gets messy. Luo et al. Train a second network to predict per-sample loss — the selector learns which audio will hurt the model — and their own abstract claims it outperforms random selection, least confidence, and estimated loss on English and Chinese tasks.
And the survey reads the same paper as...?
As not significantly better than random, with the flat observation that fifty percent random selection is much better than twenty percent loss-prediction selection.
Both claims are about the same results.
Same paper, same numbers. The survey is reading them more pessimistically and flagging what the method lacks — a diversity term. Loss prediction considers informativeness only, so it's chasing the same hard audio repeatedly. And this is the pattern the whole field has: the primary paper says it works, the practitioner reading says it barely beats random, and they're not disagreeing about arithmetic, they're disagreeing about what counts.
There's a vision-side version of this too, isn't there.
Mittal et al., "Parting with Illusions about Deep Active Learning" — the canonical methodological critique. Under realistic evaluation, deep active learning methods barely perform better than the random baseline, and they improve by a large margin only when you integrate them with semi-supervised learning.
The random baseline. The thing you were trying to beat by doing all this.
The control condition that keeps showing up in the winner's circle. Which leads us to the signal that makes all the others work — diversity, clustering by acoustic condition. The recent pipeline is two-stage precisely because uncertainty alone is insufficient. Stage one is unsupervised: extract x-vectors for every unlabelled utterance — speaker and environment embeddings — and cluster them with DBSCAN, isolating groups that share a speaker or similar acoustic conditions. Stage two does the Bayesian batch selection within that diverse pool. Earlier work did the same idea with i-vectors, K-means, and an entropy regulariser over cluster occupancy, and beat plain least confidence.
So the ranking, roughly: confidence at the bottom, committees in the middle, gradient and loss signals as an unresolved argument, and diversity as the thing you wrap around whichever uncertainty signal you picked.
That's the verdict the evidence supports, yes.
Daniel also asked about error clustering by word — the intuitive version every personal fine-tune owner eventually invents, "the model always gets these five words wrong, let me record those."
And the honest answer is that no paper in this literature does clean word-level error clustering as an acquisition function. The clustering that exists is by acoustic condition, not vocabulary item. The nearest relatives are Long et al.'s word matching error rate — used as a selection score, not a clustering method — and Adapt4Me, which highlights confused phonemes. Sub-word, not word.
Phoneme level.
So "cluster errors by word and record those" is a reasonable engineering heuristic, and it's not an established published method. Daniel invented it independently, which I'd argue is evidence it's the natural thing to invent.
Everything natural to invent is either unpublished or patented. Sometimes both.
Now — the signals rank from confidence at the bottom to diversity at the top, which takes us to the question Daniel actually asked: does any of this save you anything?
The savings numbers, and I want the complications served with them.
Best-case headline first. Long et al. report six to thirteen percent relative WER reduction against a four-thousand-hour baseline by selecting twelve hundred hours of informative utterances for human labelling. That's a thirty percent data increment on top of the baseline — twelve hundred fresh labelled hours — for a mid-single-digit to low-double-digit relative gain.
A real gain.
A real, replicated, respectable gain. It is not "a tenth of the data."
Lay out the spread.
It goes like this. At the optimistic end: seventeen percent of carefully selected samples matching full-dataset accuracy — that figure is cited by the Bayesian paper. Dossou's accent work: twenty-seven percent relative WER improvement with forty-five percent less data than baselines, on African-accented English across several domains. Then the CMU figure — a tenth of the dataset reaching full accuracy with a simple selection method — which the Vosk maintainer immediately flags with "I do not think the paper above applies to real-life data."
He didn't even let it stand for a sentence.
And at the pessimistic end: no method guarantees full accuracy with even thirty percent of the samples. They're all suboptimal.
Seventeen percent to "thirty percent isn't enough." That's not a range, that's a weather report.
The confound is that these aren't measuring the same thing. Inconsistent evaluation sets and different operating points — one paper selects a thousand lines from ten hours, another selects twelve hundred hours from four thousand — make apples-to-apples comparison close to impossible. And the Bayesian paper says the quiet part from the other direction: on heterogeneous test sets, performance often looks similar across strategies because test-set variability masks method differences. Even random sampling can look comparably good.
Wait — the test sets are so noisy that the methods are indistinguishable?
On heterogeneous sets, frequently. Your evaluation set has to be tight enough that the method difference can show through the noise, and a lot of these papers don't have that.
We did an episode on benchmarking custom ASR tools beyond the WER, and the whole argument there was that your evaluation set is the thing quietly deciding your results. This is that argument wearing active-learning clothes.
It's the same disease. And it's why the savings figures don't converge — not because the methods are chaotic, but because the rulers are different lengths.
Then the competitor. The thing active learning keeps losing to.
Semi-supervised learning. The Vosk survey's conclusion is blunt: "semi-supervised learning wins over active learning in all the works with a big margin." It's still beneficial to process a whole set at least once — pseudo-label everything with the model itself, train on the automatic transcripts — instead of forcing focus on a smaller subset. His suggested compromise is lovely: train on a subset, then fine-tune once on the whole set.
For Daniel, the semi-supervised version is almost free.
That's the thing. His unlabelled pool is his own dictated notes. Pseudo-labelling it costs machine time, no transcription labour at all. Active learning labelling costs the eight hours per hour. On pure cost, semi-supervised should walk away with it.
Except Long et al. argue the opposite.
They argue the opposite for their setting, and their reasoning is worth taking seriously. Automatic transcripts still contain errors, and those errors are very sensitive under discriminative training criteria. Worse: high-confidence pseudo-labelled data tends to have similar acoustic properties to the seed training data — the model is confident about what it already knows — so the semi-supervised gains degrade rapidly.
So pseudo-labelling teaches the model more of what it already has.
That's their claim. Fresh human-labelled data, selected where the model is actually wrong, buys more. "Our AL scheme can benefit much more from the fresh data than the SST in reducing the word error rate." Direct contradiction of the survey, both credible, and the disagreement maps onto something practical: it depends on whether your model's errors are transcription errors or coverage errors.
Split those two for me.
A transcription error is the model being miscalibrated on audio it fundamentally covers — it has the phonemes, the words, the accent, and still gets it wrong. A coverage error is the model having never seen that word or phoneme context in your voice at all. Semi-supervised helps with the first kind, badly, and barely touches the second — you can't pseudo-label what you've never covered, and your confidence estimates on uncovered territory are fiction anyway.
Which is the whole overconfidence problem again.
It loops right back around. And that split decides the single-speaker case, which is where Daniel's question actually lives.
Before we get there — and now it's time for One-Star Getaways with Clive.
This is One-Star Getaways, and I have just come back from Reykjavik, where I stayed at the Puffin's Sigh Guesthouse, one star, and I paid extra for the harbour view, which turned out to be a framed photograph of the harbour screwed to the wall above the kettle.
Breakfast was a single boiled egg and a note explaining that the chef was on a course, and the radiator made a noise like a man learning the trombone, so I have rated it one star and I recommend it warmly to anyone looking for a disappointing trip this winter.
A framed photograph of the harbour. I'm oddly tempted.
You are not booking a deliberately disappointing guesthouse in Iceland in November.
The trombone radiator alone... no. Fine. The single-speaker case.
So Adapt4Me — a demo paper from this year, the only system in this literature built for exactly Daniel's scenario: one user, their own voice, personalising a recogniser. Its diagnosis is the whole episode stated by researchers: "without human-in-the-loop guidance, users often expend effort recording redundant data — utterances that the model already transcribes well — while failing to gather the data that is most informative for further reducing recognition errors."
People record what they're already good at.
You dictate fluent paragraphs, because dictating fluent paragraphs is pleasant, and the model already transcribes your fluent paragraphs fine. What it doesn't have is the kitchen-table vocabulary, the half-swallowed consonant clusters, the rare words in your actual life.
In Daniel's case, presumably a scattering of Hebrew place names and whatever Ezra is currently obsessed with.
Whatever lives at the edges of the vocabulary in his actual voice. And Adapt4Me's answer is not a better acquisition function. It's a cheaper human step. Three stages: rapid profiling via greedy biphone coverage — it picks the phoneme pairs you haven't covered and gets you to record just those, bootstrapping personalisation in minutes of training data rather than hours. Then variational-inference LoRA for fast incremental updates — retraining in minutes, not overnight. Then continuous improvement through low-friction top-k corrections driven by visualised uncertainty: the interface shows you where the model is confused, you fix the top confusions, it updates.
The interface is the contribution.
The design claim is the contribution: "making epistemic uncertainty explicit reframes data efficiency as an interactive design feature rather than a purely algorithmic concern." And: personalisation "requires a shift from passive data collection to interactive curating."
Run the arithmetic on that, because it's the reframe I want to land. If the bottleneck is eight hours of transcription per hour of audio, a twenty percent better selection rule saves less than a five-times cheaper correction interface.
Order of magnitude, yes. Selection operates on which hours you pay the eight-fold price for. The correction loop operates on the price itself. And the correction loop also skips the transcription entirely — you're editing a draft, not producing one from silence, which is a different and much cheaper cognitive task.
So the answer to "how much does active learning save" might be "less than a good correction screen saves."
For a personal fine-tune, I think that's right. And there's a scale argument for why this case is different from everything we cited. The hard numbers — the six to thirteen percent, the seventeen percent, the forty-five percent — all come from multi-speaker corpora with hundreds to thousands of speakers. The dysarthric-speech world shows you the other end: TORGO is about three hours, UA-Speech about nine, BF-Sprache two and a half, versus Common Voice English at roughly thirty-eight hundred and LibriSpeech at a thousand.
Three hours versus thirty-eight hundred.
And the Adapt4Me point is that for non-normative speech, variability across individuals is so high that "the rich diversity of speech impairments cannot be addressed simply by collecting more data." Collecting more isn't even available — the corpus cannot be bought at any price.
A single speaker is that problem in miniature. You are a corpus of one, and you cannot go hire more of you.
Which means, at one to ten hours, your errors are dominated by coverage, not calibration. Words and phoneme contexts unseen in your voice. And the signal the evidence supports at that scale is the boring one: core-set and diversity selection. Record the phoneme contexts you haven't covered. It doesn't require the model to be well-calibrated, which is convenient, because it isn't.
So "record more" may be right after all.
The correction is record differently, not less. Same hours, pointed at the gaps.
The word you two keep using is "selection."
It is.
I've been doing this for years and nobody sent me a paper. I keep a notebook. Paper, bound, not a spreadsheet. Every word a transcription has ever got wrong on me goes in it. The timestamp, the audio, a column for what I actually said, a column for what came out.
How far back?
Further than the notebooks before it. I go back and re-record the entries, in order of how many times each one has appeared. The repeat offenders first. Most of them are fixed in one pass. Some take three or four, and those are the ones I keep an eye on.
There's a third column, isn't there.
There is. The ones where the machine was confident and wrong. Those get their own treatment — I re-record them whether or not they've repeated, because a machine that says it's sure and isn't is a different animal from one that admits it's guessing. Herman, you said that yourself twenty minutes ago, roughly, after a detour.
I did take the scenic route.
I re-record them in the car. Parked, engine off. Every one of my worst transcriptions happened in the car — the fan, the road, whatever — and I'm not letting the model learn my kitchen instead of the conditions that actually beat it. There's a spot I use. I won't say where. The acoustics are correct.
You have a dedicated recording spot for misheard words.
I have a parking spot with the right acoustics. A man can have that. It took a while to find — I tried four or five before it. My brother-in-law came along once, back when I was still driving lorries, and he said the whole exercise was mad, and then he misheard "Hilbert" as "Gilbert" on his own phone in my passenger seat, and he's been quiet about it since.
The car-cabin condition is real, for what it's worth — it's usually the noisiest sustained environment a personal model sees, and a model fine-tuned only on quiet-room audio degrades badly in it. Clustering your recordings by that condition is more or less what the x-vector stage is doing.
Then the field has caught up to a man with a notebook. I'm not asking for credit. I'm just noting the interval.
You've also pre-solved the confidence column, which took the literature a formal critique and a Bayesian pipeline.
It took me being irritated. The irritation is free and it doesn't need dropout. Though the engine-off bit — I did try recording with it idling, for the noise floor, and that was wrong. Too consistent. The real car has the fan and the indicator tick. The idle is a studio car. Engine off.
Which is a good place to land, because the thing the research cannot tell us is exactly the case we care about.
There's no head-to-head study. Nothing compares active learning against random collection for a single-speaker fine-tune at one to ten hours — Adapt4Me is a demonstration, not a measured result, and every hard number we quoted comes from corpora with hundreds to thousands of speakers.
Nobody has run the experiment on a corpus of one. The interesting frontier isn't a cleverer acquisition function anyway — it's a cheaper correction loop, and the field's own best system for this problem treats data efficiency as an interactive design feature, not an algorithm.
The misconception, before we go: people hear "active learning" and hear "record less."
The opposite of what the best case shows. Twelve hundred hours on top of four thousand for a real but modest gain — the win is in which hours, never in skipping them. And the second belief worth killing: that low-confidence audio is where the learning is. At low selection rates, confidence ranking hands you silence and crosstalk. It's diversity and coverage that pick the hours worth their eight-fold price.
If the next study is a single speaker, a notebook, and a parked car, we already know who's running it.
Producer credit to Hilbert Flumingtop, who has been at the desk the entire time and now we know has been at a notebook even longer.
For more along these lines, there's episode five, Fine-Tuning ASR For Maximal Usability; episode nine, Benchmarking Custom ASR Tools - Beyond The WER; and episode four, If Your Voice Ages, Does Your Fine-Tune Become Useless. This has been My Weird Prompts — no question too obscure, no rabbit hole too deep. If you enjoyed this one, the whole archive lives at my weird prompts dot com.
Including the episode on benchmarking custom ASR tools beyond the WER, if today's evaluation-set argument sounded familiar.
We'll be back soon. See you tomorrow.
