Episode 6381 custom

Turning Anonymous Diarization Clusters into Names

Shortening an enrolment sample to two seconds can triple the verification error rate compared with full-length recordings.

Oct 31 23:39 Transcript

With

0:00

speech-recognition local-ai

Episode Details

Episode ID: MWP-6583
Published: Oct 31
Duration: 23:39
Audio: Direct link
Pipeline: V5.26
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition local-ai

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Once a system can tell different voices apart, the next step is putting names on them. Say I record a sample that says "this is Daniel Rosehill", and Hannah records one too. Then I give the system a conversation between the two of us and it should label who said what by name, not just Speaker 1 and Speaker 2. How does that actually work? What is a speaker embedding, how is one computed from an enrolment sample, and how does the system match segments of a new recording against stored voiceprints and decide whether a match is close enough? How much enrolment audio do you need, and what breaks the matching: a different microphone, a cold, background noise, or two people who sound alike?

Speaker identification turns anonymous diarized audio into named participants by comparing speech embeddings against stored enrollment samples. Modern text-independent architectures, including x-vector variants and ECAPA-TDNN, convert variable-length acoustic frames into compact representations through statistical pooling, capturing vocal characteristics while filtering out specific spoken words. The resulting embeddings are then scored against enrolled profiles using cosine similarity or probabilistic linear discriminant analysis. However, identification accuracy heavily relies on the quality and phonetic diversity of the audio sample. On VoxCeleb verification tests with ERes2NetV2, dropping sample duration to two seconds roughly doubles or triples the error rate, highlighting why rich, natural speech consistently outperforms brief spoken introductions.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

This is part two of a two-part pair. Part one covered anonymous diarization (telling voices apart without names) and male/female voice classification. This episode is about enrolment and embedding-based speaker identification. Earlier episodes covered speaker identification broadly (#332, Jan 2026) and target-speaker ASR (#6297, #6298), so go deeper on embeddings and matching rather than repeating those.

Mentions

ArcFace Angular margin loss also known as AAM-Softmax
CosFace Margin-based loss also known as AM-Softmax
ECAPA-TDNN Speaker embedding architecture with multi-scale features
ERes2NetV2 Speaker verification model with duration benchmarks
pyannote.audio Speaker diarization toolkit with per-speaker embeddings
Res2Net Multi-scale ResNet variant for speaker recognition
ResNet Residual network image classification architecture
VoxCeleb Large speaker recognition benchmark dataset
VoxConverse Diarization benchmark dataset of conversational audio
x-vector Classic speaker embedding architecture with pooling

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6381: Turning Anonymous Diarization Clusters into Names

Two seconds of speech can more than double the verification error compared with longer recordings. Daniel wants to know what happens after a system separates voices into anonymous groups. He and Hannah each record an introduction, then feed it a conversation. How does it turn those groups into their names, and keep doing that when the microphone, room, or voice changes?

The introduction supplies an enrolment sample. Its words do not assign the identity. You attach Daniel's name to a numerical representation extracted from his voice, then compare new speech against it. The technical term is speaker identification with enrolment.

Saying your name is mostly paperwork for the human.

Yes. He could describe a sandwich. For a text independent system, varied speech might actually be better than that crisp introduction.

Useful. Identity established through lunch.

There is a distinction worth making immediately. Verification asks whether this voice belongs to the person claimed. Identification asks which enrolled person it belongs to. Daniel wants identification, with the additional possibility that neither enrolled person is speaking.

Otherwise every visitor becomes one of them.

That possibility needs to be designed in. A matching engine will always produce a best candidate, even when every candidate is wrong.

My first instinct would be to compare the recordings directly, allowing for speed. Same voice, similar sound. Presumably the different words ruin that before we even reach the microphone.

Different words and different lengths. One recording might contain a long vowel where the other contains a sharp consonant. Their acoustic feature sequences do not line up. Early text dependent systems constrained the problem by asking for the same passphrase at enrolment and recognition.

Convenient, until the conversation refuses to be a password.

Modern embedding systems turn variable length recordings into fixed length vectors. Typically the front end produces roughly a hundred acoustic feature vectors per second, often log Mel filterbank features. Those describe how energy is distributed across frequency bands over time. A neural network processes the sequence and reduces it to one representation.

A second and a minute end up with the same number of coordinates. That sounds like an enormous opportunity to throw away the wrong information.

It is. You want to preserve speaker characteristics while suppressing words, noise, and recording conditions. Suppressing them imperfectly is responsible for much of what goes wrong.

The practical pipeline is compact. Record clean enrolment speech, extract an embedding, store it beside a name. For new speech, use the same extractor, compare the resulting vector with the enrolled vectors, and accept a name only if its score clears your threshold.

Same extractor matters. Two models can both produce five hundred numbers without agreeing what any of them mean.

Their coordinate systems need not be compatible at all. Changing the extractor generally means rebuilding the enrolled representations. Corn, would you match every little conversational turn separately, or pool everything assigned to one anonymous speaker first?

Pool the cluster, unless it contains somebody else's speech. A solitary yes seems a poor basis for naming someone. Although pooling makes an early clustering mistake contagious.

That is the trade. Longer clean material stabilises the representation. Contaminated material can stabilise it around the wrong mixture. Naming depends on the segmentation and clustering that came before it.

Now explain where those coordinates come from. I assume nobody labels one axis nasal and another vaguely Connecticut.

Training commonly begins with recordings from thousands of known speakers. The network learns to classify which training speaker produced each sample. After training, you extract an internal representation before the final classifier. That representation can distinguish people the training classifier never knew.

Daniel doesn't need his own new neural network, then.

Usually not. Enrolment runs an existing extractor over Daniel's audio. No model weights need to change. Training the general extractor and enrolling a new person are separate operations.

That will disappoint anyone selling a progress bar.

The classic x vector architecture has an encoder, pooling, and a classification head. The encoder transforms acoustic frames into learned frame level features. Pooling summarises those features across the utterance. The head predicts the training speaker, while the embedding is taken from an intermediate layer around that utterance level representation.

Pooling seems suspiciously modest for the part that turns an entire conversation into a point.

Basic pooling can calculate the mean. Statistics pooling also includes standard deviation, retaining information about variation. Attentive statistics pooling learns to weight more informative frames more heavily. A sustained, useful speech sound need not receive the same weight as an unhelpful fragment.

You would assign my pauses negative weight.

Your pauses are unusually easy to identify.

Years of clean training data.

ECAPA-TDNN develops that architecture with features at multiple temporal scales, squeeze excitation blocks that adjust feature channels, aggregation from several layers, and channel dependent attentive statistics pooling. It gives the network richer local information before the utterance gets compressed.

There it is. You put four mechanisms into one breath.

I did. The useful distinction is that the encoder examines patterns locally, while pooling gathers evidence across the recording. A local receptive field does not specify the required enrolment duration.

In the tutorial configurations, small ECAPA has three layers, five hundred and twelve feature dimensions, and a local receptive field of about a quarter second.

Feature dimensions, not necessarily the size of the final stored voiceprint.

Correct. The larger configuration has four layers, two thousand and forty-eight feature dimensions, and about a third of a second locally. Those are configuration details, not universal specifications for every implementation carrying the name.

Good. Otherwise someone buys a third of a second and expects a biography.

Other architectures cover longer spans. A listed ResNet fifty configuration reaches about one point two seconds. A Res2Net fifty configuration using scale eight reaches eight point three seconds. Neither number tells you how much clean speech the final identification decision needs.

The network also needs a reason to arrange similar speakers sensibly. Classifying the training voices correctly doesn't automatically guarantee useful distances between strangers.

Plain cross entropy trains classification. Margin based losses strengthen the geometry. AM-Softmax, also called CosFace, subtracts a margin from the correct class score during training. The model has to separate classes more decisively to succeed. A typical score scale is around thirty. AAM-Softmax, or ArcFace, instead introduces an angular margin.

Making the exam harder so the boundary survives outside the classroom.

Those losses explicitly encourage angular separation. That is why cosine similarity works naturally downstream. Take the dot product of the enrolled and test vectors, then divide by the product of their lengths. You are comparing their direction rather than rewarding a vector merely for being large.

Higher means a better candidate. It does not mean a higher probability that the name is correct.

Not without calibration. A cosine score by itself is not a percentage confidence. Another option is PLDA, probabilistic linear discriminant analysis. It models an embedding as a global mean, a speaker specific component, and within speaker variation. That last component covers the natural movement between different recordings of one person.

Rather than treating every movement away from Daniel's enrolment point as evidence of Hannah.

It scores whether the observed difference is more consistent with one speaker across sessions or with different speakers. Ordinary LDA is a simpler projection that emphasises between speaker separation relative to within speaker variation. PLDA supplies a probabilistic same speaker versus different speaker comparison. It can help, although a badly matched back end can lose to simple cosine scoring.

More machinery, more places to be confidently provincial.

How would you set the acceptance threshold for our transcript, Corn?

I would rather read unknown than find a sentence assigned to me that you said. Particularly a sentence containing squeeze excitation.

Then false acceptance carries a higher cost for you. Calibration tests genuine matches and mismatched speakers across many trials, typically well over a thousand, and counts false rejections and false alarms. Sweeping the threshold produces a detection error tradeoff curve. Equal error rate marks where those error rates meet. A detection cost function instead weights them according to the application.

Equal error is a reporting point, not an instruction to value both mistakes equally.

For implementation, pyannote audio already returns per speaker embeddings alongside diarization. There is one vector for each anonymous speaker, ordered to correspond to its speaker labels. You match those rows against enrolment vectors and rename the labels. Keep that ordering intact.

A perfect voiceprint, attached to the wrong row. Efficient sabotage.

Its commercial service also offers separate voiceprint creation and identification operations. Naming can be an added stage rather than a reason to rebuild the whole diarization pipeline.

And now, our regular segment, One-Star Getaways.

This is One-Star Getaways, and I have just come back from Halifax, where I stayed at the Marrow Lane Guesthouse, one star, because the advertised harbour view turned out to be a framed painting of the harbour, hung slightly crooked, and the kettle took nine minutes to boil, which I timed.

The complimentary breakfast was one biscuit, and the stairs creaked in a key I did not care for. For anyone looking for a disappointing trip this winter, I recommend it warmly, and I mean that as a warning.

I'm tempted by the biscuit. Herman isn't booking. Back to those enrolment vectors, a correct row still needs enough speech to represent a person.

There is no universal minimum duration. There is a performance curve, and its shape depends on the system and recording conditions. For ERes2NetV2 on a VoxCeleb verification test, equal error rate was about zero point six percent with full duration recordings.

Low, under that test's conditions. Not a promise about our kitchen.

At three seconds, it was about one percent. At two seconds, about one point five percent. Cutting the audio roughly doubled or tripled the error compared with the longer condition.

Two seconds contains plenty of waveform. The missing ingredient is variety, not merely file size.

Phonetic coverage. A short phrase exposes only a small selection of vowels and consonants. The embedding can reflect that particular content disproportionately. Pooling cannot average over evidence that never arrived.

Then Daniel's introduction may be tidy for the recording interface and poor for the actual job. Thirty seconds of ordinary speech could beat three seconds of beautifully pronounced name.

It may. That is a reasonable starting comparison, not a universal thirty second requirement. Text dependent systems can exploit a known phrase, especially when the test also uses it. Daniel wants arbitrary conversation, where the test words will change.

His name is not going to occur in every sentence. We can hope.

Short duration verification work explicitly distinguishes unstable short enrolment from longer, more stable text independent representations. NIST evaluations have used conditions around ten seconds and others over three minutes. More recent evaluations vary enrolment duration and include shorter tests and recordings containing multiple people.

That also answers whether silence counts. A minute long recording containing one brief introduction is still one brief introduction.

Count usable speech, not elapsed recording time. The short duration verification challenges also test phonetic mismatch and cross language conditions. If enrolment is English and the conversation switches language, the system has another kind of mismatch to tolerate.

I'd record Daniel talking naturally, rather than repeating his name until the network develops a grievance.

Would you record that all on the podcast microphone, or include his phone?

Include both if both will actually appear. Otherwise we have learned Daniel plus this microphone and asked it to recognise Daniel minus this microphone.

Session variety can help represent the actual use case. A different microphone changes frequency response, distance, room contribution, and possibly compression. It does not inevitably break identification, but it can move the recording outside the conditions the system handles well.

You diagnose rooms from recordings when you're mixing music. I have watched you blame a wall before considering the singer.

Walls contribute very reliably. Unfortunately, the singer contributes too.

Training augmentation and channel aware modelling aim to stop those recording differences from overwhelming speaker information. PLDA's within speaker variation is relevant here, but it has to have learned useful variation. It cannot excuse an arbitrary distortion just because we call it a new session.

Noise seems slightly different. A microphone colours Daniel. Another voice adds someone who isn't Daniel.

Overlap is particularly difficult because a segment may contain both speakers' characteristics. Noise can mask useful frequencies or mislead the speech detector. A system may extract a perfectly valid vector from audio that was a poor identity sample in the first place.

An embedding of the argument, rather than either participant.

One augmented embedding sampling method reported relative improvements of seventy-one percent in diarization error and sixty-nine percent in speaker attributed word error against its baseline. It specifically addressed noisy and overlapping speech. Those are gains within that experiment, not percentage point reductions you can apply to every recording.

Large improvements can also tell you how much the baseline suffered.

Similar sounding people pose another problem. Human similarity does not always predict embedding similarity, because the system uses cues we may not notice. Still, when two enrolled voices occupy overlapping regions, moving the threshold cannot manufacture separation.

Reject more and you lose genuine speech. Accept more and one brother inherits the other's opinions.

Margin training helps separate speakers generally. It does not guarantee every pair will be separable in every short segment. A second requirement can help naming too, such as demanding a clear gap between the strongest candidate and the runner up, not just a high winning score.

Where does a cold fit? My guess is changed resonance and altered speaking behaviour, with the size of the effect depending on how ill somebody sounds.

That is how I would reason about it. Congestion, hoarseness, fatigue, and altered delivery can change the evidence. I don't have a defensible cold specific error percentage. It is plausible within speaker variability, and its impact depends on the extractor and the severity of the change.

No numerical mucus allowance.

Hm.

We also need to keep verification error separate from diarization error. Pyannote's published community benchmark gives about eleven percent diarization error on VoxConverse, and seventeen percent on a meeting dataset. Its premium pipeline reduces those to roughly nine and thirteen percent respectively.

Those include finding and separating speech, not just choosing a name after the separation.

Correct. Diarization error includes missed speech, false speech detections, and speaker confusion. On harder audiovisual and egocentric datasets, the community figures reach roughly forty-five and forty-seven percent. Calling that half the names wrong would be inaccurate. It is error measured over speech time, combining several sources.

Still enough to make automatic naming a downstream risk, rather than a decorative label.

The premium figures improve those difficult cases but remain substantial. The recording conditions matter enormously. A tidy verification benchmark and a noisy multi speaker recording are answering different practical questions.

Our reject option seems more important now. If the enrolled list contains only Daniel and Hannah, a delivery driver should not become whichever of them scores less badly.

Closed set identification assumes the speaker belongs to the enrolled set. Open set identification must also reject strangers. Cosine always returns a score between negative one and one. Taking the maximum alone forces a winner.

First place in a contest nobody entered.

Unknown needs to be a legitimate output. For a transcript, you can preserve an anonymous cluster when identity is uncertain. Enrolment free diarization already provides those clusters. It cannot independently tell you which real person a cluster represents across separate recordings.

Names require the enrolment link or another identity signal. Better clustering doesn't invent that link.

A human correction can supply it, provided the system remembers what was corrected. The important thing is distinguishing a claimed identity from...

The person who booked the plumber. That is what I wanted the voice sample to establish this week. My sister Ruth says I made the call. I say she did. The recording is short. Neither of us finds the words helpful. We both say we will be in all morning when we plainly won't.

A family resemblance with scheduling consequences.

I put the recordings in a different order and listened without looking. I am interested in what the microphone does, Herman. You explained the angular margin very carefully. I was still thinking about the microphone. I don't trust the thin voice from a phone on a table. At home I listen to those calls through the same headphones. I will not decide whose voice it is from the little phone speaker.

Consistent playback removes one changing part. It leaves the recording differences intact.

There is an entry in the notebook for the same sentence recorded on four devices this week. I said, the plumber can come after lunch. Ruth changed the playback order. I marked the one I thought was the laptop. It wasn't. I tried again and marked a different one. That wasn't the laptop either. I was more certain about the plumber before we started.

You tested which machine it sounded like, not whether it was you.

I knew that batch was me. I wanted to hear which version resembled the disputed call. The phone on the table sounded least like me to me. Ruth preferred that one. Said it was clearer.

Her familiarity with your live voice might make a different recording sound natural to her than it does to you. Neither judgement establishes who made the original call.

No. The plumber is coming after lunch now.

That distinction matters for the software too. Familiar sounding and correctly attributed can part company. A transcript can look entirely plausible while attaching a genuine sentence to the wrong person.

Particularly when an editor silently replaces uncertainty with the most likely name. Daniel's workable design is enrolment embeddings, clean diarized speech, compatible extraction, calibrated matching, and an unknown option. The name is a decision supported by evidence, not something the vector contains in readable form.

Longer, varied enrolment helps that evidence. It cannot repair two voices mixed into one segment, or a rule that refuses to admit a stranger.

The open question for meeting tools is whether they will preserve an uncertain attribution as carefully as they preserve the words, rather than making the document look finished.

Thanks to Hilbert Flumingtop for producing.

If you enjoyed this one, try episode fifteen, AI Gets Personal; and episode ten, How ASR Went From Frustration To ... Whisper Magic. This has been My Weird Prompts, the human-AI collaboration podcast. No question too obscure, no rabbit hole too deep.

If you enjoy the show, leave us a review wherever you listen.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error