Episode 6341 custom

Why Fast Speech Detectors Still Struggle with Onsets

Aalto and Genelec experiments reveal human ears can detect sound timing shifts down to half a millisecond at certain frequencies.

Oct 29 23:39 Transcript

With

0:00

Episode Details

Episode ID: MWP-6543
Published: Oct 29
Duration: 23:39
Audio: Direct link
Pipeline: V5.25
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition text-to-speech local-ai ai-inference ai-evaluation

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Some real-time AI workloads genuinely amaze me, and the one that amazes me most is voice activity detection. It needs a tiny model, small enough to run on a CPU, that detects speech before the first word properly arrives. It has to be quicker than real time, and it manages that partly because human speech is preceded by small sounds (a breath, a lip click, the onset of voicing) that it can catch, but it has to be extremely fast to do so.
I want to open with VAD because it's one example of a wider class of real-time modalities. The ones at the frontier now are real-time video avatars, where a camera is pointed at someone and, live, the system reads their face and transfers it onto another avatar; and real-time voice changing. The old version of voice changing is the famous chipmunk pitch shift. The much more interesting new version listens to the user and applies a live transformation into a generative voice.
What I'd like to explore, using those three examples (VAD, live video avatars and real-time voice conversion), is how each one attempts what seems impossible: generating faster than humans can perceive it. As a foundation, talk about human latency perception: the window between the smallest delay a human brain can detect and zero, and how that gap is exploited by, or integral to, these modalities.

Voice activity detection sits as a tiny, essential gatekeeper ahead of complex speech models, face avatars, and real-time voice conversion. While processing faster than real time provides crucial headroom, it cannot eliminate the physical delay required to gather incoming audio. This timing challenge is further complicated by human perception: listeners notice internal frequency timing shifts down to half a millisecond, but tolerate much larger delays before detecting echo or delayed action. Furthermore, evaluating these systems remains remarkably tricky. Dataset timing labels exhibit significant boundary jitter and bias, meaning benchmark scores like frame-level accuracy fail to reflect whether a detector inadvertently clips initial consonants or misjudges conversational turns.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

Prior episodes already covered parts of this: #2288 "The Invisible Gatekeeper of Voice Tech" (VAD), #2123 "Human Reaction Time vs. AI Latency", #1556 "The War Against Latency" and #1546 "The Death of Latency: Three Pillars of Modern Voice AI". Reference them briefly where useful but don't rehash them. The new angle is the comparison across the three modalities and the perceptual window each one exploits.

Mentions

AVA-Speech Speech activity dataset with annotated boundaries
FireRedVAD Small VAD model benchmarked on FLEURS-VAD
FLEURS-VAD Multilingual VAD benchmark dataset
librispeech Standard English ASR benchmark dataset
MeanVC Two Streaming voice conversion with reduced latency
Paravo Low-latency voice conversion product
S4VAD State-space model VAD with adjustable memory decay
Silero VAD Voice activity detection model
SynthVC Voice conversion system with reported latency
WebRTC VAD Classic lightweight voice activity detector

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6341: Why Fast Speech Detectors Still Struggle with Onsets

Voice activity detection usually gets treated as the boring switch before the impressive AI. Daniel puts it the other way round. The tiny CPU model that notices speech before a word has properly arrived may be doing the most impressive real-time job of all. He wants to connect that to live face avatars and generative voice changing, where the system has to finish its work before we notice the delay.

The switch gets a remarkably demanding job description. Notice the start, ignore the fan, don't mistake a plate hitting the table for somebody addressing you, and please don't eat the first consonant.

The consonant is rarely covered by the warranty.

I'm Herman Poppleberry, and the first distinction matters immediately. Processing faster than real time means you can keep up with incoming material. It doesn't automatically mean the output feels instantaneous. You might process an enormous recording very quickly while still making a live speaker wait for the next chunk.

Daniel's angle is the space between zero delay and detectable delay. The awkward part is that the space changes depending on what we're doing. My tolerance for waiting is excellent. My tolerance for hearing my own voice arrive late is considerably less spiritual.

Corn, you will defend a slow computer until it interrupts your sentence.

Waiting is civilised. Losing the beginning of my objection isn't.

Human hearing can detect astonishingly small timing changes. The Aalto and Genelec experiment used twelve listeners and found sensitivity down to half a millisecond at certain frequencies. Starts and stops made the delays especially audible.

Half a millisecond makes every software progress bar look criminal.

The experiment used frequency-selective time-reverse filtering to create negative delay. Juho Liski described it with the line, To achieve a negative delay, we need to move into the future. With an existing recording, software can rearrange timing in ways a live microphone cannot. That doesn't give your voice changer knowledge of your next syllable.

Nor does it mean a person notices every half-millisecond pause before a reply. We're comparing altered frequency timing inside a sound, rather than waiting for somebody to answer.

A different experiment gives a different threshold. Minimum detectable echo delay came out around twenty milliseconds for music and twenty-three for speech. Now you've got an original sound and a delayed version, rather than parts of one sound arriving out of alignment.

My own voice supplies the original whether the software wants it or not. I hear myself through my head, then the headphones deliver their opinion.

Self-monitoring is especially unforgiving for precisely that reason. Action-to-sound experiments give another spread. One measured a just-noticeable difference of forty-nine milliseconds starting from effectively zero latency.

Meaning the smallest extra delay people reliably noticed in that particular task. Not a universal forty-nine-millisecond allowance for everything with a microphone.

At a sixty-four-millisecond starting delay, the additional detectable difference was twenty-seven milliseconds. With the starting delay pushed to five hundred and twelve milliseconds, it rose to seventy-seven.

Hm.

The brain doesn't issue a single deadline. It judges a relationship. A delayed echo, a key press, a lip movement, and an answer to a question all offer different evidence.

Would you accept a converted voice arriving a little late if the person on the other end couldn't tell?

For them, yes. For me, only if you stop piping it straight back into my ears. The audience and performer have different clocks.

That distinction will haunt every latency claim we mention.

Before we get further into perception thresholds, our resident finance expert has his regular segment.

Desmond's Money Tip. Split one paper napkin into its two layers and call it two napkins. That is a saving of one napkin per meal, which over a year of three meals a day is one thousand and ninety five napkins, a sum I value at roughly two pounds and eleven pence.

I keep a running total on the back of a napkin, which does mean I have to spend one to record it. I do not count that one. It would ruin the figures.

Money is mostly just the slow accumulation of things you did not buy. It is not uplifting, but it is reliable.

Tempting. Herman looks appalled, so I'll leave the napkin intact. The VAD model has a simpler audience problem, though. Nobody needs to listen to its output.

Its output is a speech probability, usually turned into a yes or no decision. That's vastly less work than generating the speech itself. Silero's JIT model is around two megabytes. On a single CPU thread, processing a chunk containing more than thirty milliseconds of audio takes under one millisecond.

Thirty milliseconds arrive, one millisecond works, twenty-nine milliseconds look available. Except you had to wait for the thirty to arrive.

Thank you, because that's where the phrase faster than real time can mislead. Roughly thirtyfold processing headroom gives you room for buffering, overlap, other software, and sometimes look-ahead. It cannot erase the time spent collecting evidence. A future sample must still happen before a live system can use it.

The model cannot spend spare CPU cycles purchasing tomorrow.

Daniel mentions breaths, lip clicks, and the beginning of voicing. Those physical events can precede a recognisable word. I'm less certain about claiming that a particular VAD explicitly uses all of them as advance warning. A breath can mean speech is coming, or simply that the speaker intends to remain alive.

A persistent false positive in this household.

The onset of voicing is already speech evidence. Recognising a complete word takes longer. A detector can therefore fire before you know what was said without predicting anything at all.

I would actually be suspicious of a detector that opened every time I inhaled. It could feel wonderfully responsive during a demonstration and spend the rest of the day transmitting nose sounds.

What would you choose, Corn? Occasional clipped beginnings, or the channel opening too readily?

Keep a short rolling buffer and send the beginning retrospectively when you're confident. I'd rather preserve the consonant than pretend the decision happened before it did.

That's a common practical solution. The buffer retains recent audio while the detector decides. Detection and preservation become separate jobs. You may begin sending slightly later, but the recogniser still receives the actual beginning.

Which means the user can experience complete speech capture without the classifier being clairvoyant. It also means an onset score alone won't tell us whether the product works well.

There is a fascinating architectural knob in S4VAD. Its state-space model carries acoustic information forward through time. A parameter called delta controls how quickly that memory decays. Larger values make it more local and responsive. Smaller values preserve context, but evidence of speech can linger into silence.

You are about to fall in love with an adjustable memory.

I've already done it. However, this one has a reason to exist beyond having a beautiful equation.

That narrows your usual selection process.

Fair. Keeping context helps distinguish meaningful speech from a tiny ambiguous noise. Reacting locally helps catch a sudden start. You can't maximise both just by making the model smaller.

There's also stopping. Daniel's entry point is the start of speech, but a conversation system has to decide whether my silence means I'm done or whether I'm thinking. You tend to treat both as permission.

Your pauses contain several possible endings.

They're still mine.

Endpointing adds another policy layer. VAD continuously estimates whether speech is present, not merely the first onset. The application decides how much silence counts as the end of a turn. Interruptions need yet another decision. Someone saying a brief acknowledgment shouldn't necessarily terminate the assistant's entire answer.

That is where a very fast wrong decision becomes more annoying than a slightly slower correct one.

Frame-level accuracy doesn't settle it. FireRedVAD uses about six hundred thousand parameters and reports ninety-seven point five seven percent frame-level F1 on FLEURS-VAD.

A very small model with a very large mark.

Its reported area under the receiver operating curve is ninety-nine point six percent. It beats several familiar alternatives on that benchmark, including Silero and WebRTC VAD. Yet neither headline score tells you much about exactly when the gate opens.

A long sentence contains plenty of easy middle. You could classify all that correctly and still damage the beginning of every sentence. The aggregate looks healthy because the part I care about is brief.

The timing labels themselves make it worse. S4VAD's evaluation found onset annotation jitter on the order of tens to hundreds of milliseconds. Its estimated dataset shift for LibriSpeech was minus seventy milliseconds, meaning the labels tended to put the start too early.

You're being marked late against somebody who started the stopwatch early.

For AVA-Speech, the estimated shift was minus one hundred and thirty-six milliseconds. These are estimated dataset biases, not corrections you should blindly apply to every recording. Individual boundaries still vary.

A twenty-millisecond latency claim measured against that needs a very careful explanation. You can print more decimal places than your labels deserve.

Speech boundaries aren't always obvious even to a careful listener. Does the start include an unvoiced consonant? A preparatory breath? Quiet speech under background noise? The annotation policy needs to say, and the evaluation needs to respect it.

I hadn't expected the expensive problem to be agreeing when somebody began talking.

The tiny gate sits ahead of speech recognition, speaker diarization, enhancement, and communication systems. If it discards the beginning, every expensive model downstream inherits the loss. If it opens constantly, the battery and inference bill inherit the noise.

Cheap enough to run everywhere. Important enough to spoil everything.

Conversion has no comparable escape into a binary answer. It has to produce an entire waveform. The old chipmunk effect changes pitch, often shifting the apparent vocal character along with it. Generative conversion retains the spoken content while synthesising a different voice.

I say the words, somebody else's throat gets the credit.

Conceptually, yes. Though it needn't literally transcribe text and then speak that text. Some systems use continuous acoustic representations or neural codecs. A text bottleneck would discard details they may want to preserve, including timing, laughter, and parts of the performance.

Then content extraction sounds tidier than it really is. My identity isn't stored in one removable layer while my sarcasm sits safely underneath.

TVTSyn points to that representational mismatch. Content changes over time, while many architectures inject speaker identity as one static global embedding. But voice identity interacts with how each sound is produced. Maintaining articulation while changing vocal character is a moving problem.

A pitch slider has the considerable advantage of admitting what it does.

Current reported latency is impressive. Paravo advertises as low as sixty milliseconds on CPU, with more than a hundred voices and over eighty thousand users.

As low as means there's a favourable setup behind it.

Hardware, configuration, and measurement endpoints matter. SynthVC reports seventy-seven point one milliseconds end to end. MeanVC Two cuts its reported latency from two hundred and eleven to one hundred and ten milliseconds.

Almost half the wait removed. That ought to change how speaking into it feels, not merely improve a benchmark table.

It uses future-receptive chunking with forty-millisecond chunks. The challenge is retaining enough continuity to avoid rough joins without accumulating a long queue of future audio.

Wait.

Future-receptive still means waiting for actual speech when the future contains speech.

It does. There's no exemption from causality in the title. More efficient chunk handling reduces the wait; it doesn't predict an unconstrained speaker perfectly. StyleStream's first version, despite being streamable, sat around a second end to end on consumer hardware.

Streamable. Bring your own patience.

A system can produce output continuously while remaining a second behind. Throughput tells you whether the queue grows. Latency tells you how far behind you're already standing.

Those sixty to one hundred and ten millisecond figures are also above several of the auditory thresholds we discussed. We shouldn't call the delay imperceptible just because the voice arrives continuously.

Who hears the result matters. A remote listener lacks your immediate original as a reference. You don't. Corn, would you prefer a cleaner voice at a higher delay, or a slightly rougher voice that follows you closely?

During a conversation, the faster one, within reason. An exquisite voice that keeps arriving after I've moved on feels like somebody else is controlling my mouth. For a recording, let it take longer.

That's why product settings can outperform general-purpose demonstrations on speed. A restricted set of voices, smaller models, fixed hardware assumptions, and a tightly controlled pipeline can all help. But sixty on CPU against seventy-seven in a paper isn't a fair race unless the work and timing boundaries match.

Your comparison table needs more columns than the sales department allows.

Video adds another reference. You see your own gesture and the transformed gesture. Lucy Two point five advertises under forty milliseconds glass to glass at thirty frames per second.

Glass to glass at least names two ends. Camera capture to displayed output, rather than the pleasant little interval when the GPU is busy.

At that frame rate, another frame arrives roughly every thirty-three milliseconds. Meeting that production rate doesn't prove every frame has low latency. You can keep a smooth output stream while a queue quietly puts the whole thing behind.

A polished face from the recent past.

Camera-driven face transfer and audio-driven talking heads aren't identical workloads, either. One follows observed expressions. The other generates facial motion from sound. Both have deadlines, but the available evidence differs.

Audio-driven faces also have to know when to stop moving. A face that continues speaking while I'm listening makes a more serious timing error than a slightly late blink.

EmbodiedHead uses a streaming audio scheduler to suppress unwanted mouth motion during listening, and generation can take as few as four sampling steps. FaceGAN takes a different route, arguing that audio-conditioned facial motion is constrained enough for a single forward pass per frame.

You have found another elegant mechanism.

This one earns it. Diffusion repeatedly evaluates a network to refine a sample. Reducing that repetition helps streaming. FaceGAN also shapes synthetic noise using its future values. Since the system creates that noise, it can sample future noise now without waiting for future speech.

Inventing tomorrow is easier when tomorrow is your own noise.

The audio-to-expression path stays causal. The generated noise can provide smooth temporal structure without smuggling in unavailable audio. That's a very specific trick, not a general claim that the model knows the speaker's future.

A face render quoted at a hundred and thirty-four to a hundred and fifty milliseconds is still only one stage. Add speech generation, capture, transport, playback... no, even before that, decide whether we're timing a gesture or a conversational reply.

A natural turn gap might be around a fifth of a second, but that is a different perceptual situation again. You can tolerate some response planning while finding a visibly detached mouth immediately distracting.

The defensive side gets an uglier deadline. A voice-conversion detector working on one-second segments has to hear enough material to classify it. By then, the converted voice has already been talking.

Its inference might be fast while its evidence window is long. Same distinction again. For your DJ setup, Corn, would you even bother trying to reproduce that half-millisecond result at home? I'd call that impractical without carefully controlled listening conditions.

...with a recorded signal, yes, Corn. That's what I'm doing. Not live speech. The half-millisecond result concerns frequency regions relative to others. A whole recording arriving half a millisecond late doesn't recreate that comparison.

I blurred those two things just then. A controlled comparison is possible at home. Getting the same experiment is another matter.

My current arrangement is a mess. The laptop is on the floor because the short lead reaches there. I alternate processed recordings through the same headphones. I have been doing these comparisons for years, but this month I'm trying to sort out the switching. You were careful about capture and playback earlier, Herman. I'm trying to keep the switch from changing anything else.

Otherwise you end up identifying the switch rather than the delay.

Nissim at the audio shop keeps offering me another interface. I've dealt with him far too long to let him choose the subject. The old switch box is gone. I have its wiring diagram, folded inside an envelope marked unpleasant. I keep bringing that instead. It shows the connection I want without another discussion about recording vocals.

The diagram gives him something definite to wire. That's less ambiguous than asking for a box with no audible signature.

He says he can make the lead. I'm waiting. The surprise is the difference between the two routes I use now. My recording puts it at three milliseconds, before I add any deliberate filtering. I'm not claiming laboratory precision. It's enough to make me stop calling the routes identical. Nissim writes one hundred and eighty shekels on the envelope.

Three milliseconds before the intended experiment even begins. The unchanged route needs as much attention as the altered one.

That also sharpens the product comparison. If we want to know what the model added, we need the same pipeline without it. If we want to know what the user experiences, we need the whole route. Those are both useful numbers, but they answer different questions.

The distinction that stays with me is evidence time versus computation time. A model can finish in a millisecond after waiting much longer to learn what it needs.

Faster inference helps, but it cannot remove the observation window. VAD needs enough sound to decide. Conversion needs enough context to speak smoothly. An avatar needs enough information to move plausibly.

As those stages get connected, I'd like to see latency claims include the unchanged route and uncertainty at the boundaries. Otherwise we won't know whether the next improvement happened in the model or in the stopwatch.

Thanks to Hilbert Flumingtop for producing the show.

If this was your kind of episode, go back for episode fifteen, AI Gets Personal; episode ten, How ASR Went From Frustration To ... Whisper Magic; and episode five, Fine-Tuning ASR For Maximal Usability. This has been My Weird Prompts, the human-AI collaboration podcast.

If you enjoyed it, leave us a review wherever you listen. It helps other curious people find us.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error