{"id":"speaker-diarization-audio-clustering-42538","slug":"speaker-diarization-audio-clustering-42538","title":"How Audio Models Tell Voices Apart","description":"Voice recognition systems often struggle with speaker diarization: tracking who spoke when across overlapping speech, whispers, and brief interruptions. This exploration unpacks the mechanics behind pyannote and end-to-end neural diarization (EEND), examining how voice activity detection, speaker embeddings, powerset segmentation, and Bayesian hidden Markov models like VBx separate anonymous speakers. It also looks at the trade-offs of setting fixed speaker counts versus using encoder-decoder attractors to handle unknown voices across challenging recordings.","excerpt":"A diarization model can tell two voices apart without understanding a single spoken word or assigning a real name.","pubDate":"2026-10-31T14:55:48.696Z","tags":["speech-recognition","local-ai","ai-evaluation","text-to-speech"],"heroImage":"https://files.myweirdprompts.com/covers/speaker-diarization-audio-clustering-42538.png","podcastAudioUrl":"https://episodes.myweirdprompts.com/audio/speaker-diarization-audio-clustering-42538.m4a","podcastDuration":"28:23","episodeNumber":6377,"transcript":"Corn: Forty-six point eight percent diarization error on one challenging benchmark. The software can recognise speech and still make a spectacular mess of who owns it. Daniel wants to get underneath that problem, starting with a recording of himself and Hannah. Before attaching names, how does a machine establish that one voice belongs in Speaker One and the other in Speaker Two?\nHerman: Especially when either person can return after a silence, interrupt, whisper, or say something as unhelpfully brief as yes.\nCorn: A complete contribution, in my view. Daniel's asking about the foundations beneath pyannote, the anonymous separation before anyone is identified. Then he wants to go up a layer, to labels such as Male One and Female One. What acoustic evidence produces those labels, and how much confidence do they deserve?\nHerman: Those layers use related signals, but they answer different questions. Anonymous diarization assigns speech to recurring voices. Attribute classification interprets something about those voices. Neither requires a name, and keeping names out lets us see what each system is actually doing.\nCorn: Speaker Two has no biography. Refreshing.\nHerman: Pyannote is a sensible starting point because it exposes the building blocks, rather than hiding the entire job behind a transcription box. Its own description names speech activity detection, speaker change detection, overlapping speech detection, and speaker embeddings.\nCorn: That makes the usual question, which algorithm does diarization, slightly misleading. You're asking which part notices speech, which part notices a change, and which part realises the person coming back is someone we've already heard.\nHerman: Diarization partitions speech into stretches associated with a speaker. Segmentation supplies the boundaries and activity. Clustering joins stretches that belong together. The familiar conceptual stack is voice activity detection, change and overlap detection, fixed-dimensional speaker embeddings, then clustering.\nCorn: Conceptual matters there. Those needn't be four independent programs passing little envelopes.\nHerman: Some functions share a model, and modern systems combine things differently. The community-one pipeline identifies a segmentation model using the powerset formulation, a WeSpeaker embedding model, and VBx clustering. That's a concrete answer to what lives beneath the wrapper.\nCorn: Where does actual transcription enter? My guess is it doesn't need to. A machine could tell us we've alternated for ten minutes without understanding a word.\nHerman: It can. Words aren't required for the basic task. You can diarize speech in a language the transcriber handles badly. Of course, the acoustic models still have training histories and weaknesses. Language independence doesn't mean immunity to unfamiliar pronunciation or recording conditions.\nCorn: Herman, would the machine even know which cluster to call Speaker One?\nHerman: Only by some convention, usually the order in which speakers appear. The number is bookkeeping. Swap the labels consistently and the diarization is equally correct.\nCorn: An identity crisis solved by renumbering.\nHerman: The embedding is the useful unit to hold in mind. A neural network takes a stretch of speech and produces a fixed-length vector. Training encourages recordings of the same speaker to occupy nearby regions, while different speakers should separate. Clustering then works on those vectors, rather than the original waveform.\nCorn: Should separate. If Hannah changes from talking normally to laughing, I'd expect a fairly substantial movement in that space. Meanwhile another person with similar vocal anatomy might land inconveniently close.\nHerman: Both happen. The network is trying to retain speaker information while discounting phonetic content, room acoustics, microphones, emotion, and so on. It never removes all of that. Short speech is especially awkward because a tiny fragment gives it less evidence about the speaker.\nCorn: One syllable and an identity assessment. Efficient hiring practice.\nHerman: Would you rather the system leave a short yes unassigned, Corn, or force it into one of the existing clusters?\nCorn: Leave it unassigned if the alternative is pretending. But I'd want to distinguish unknown from not enough audio. Otherwise the transcript fills up with mysterious visitors who only say mm.\nHerman: That distinction changes the interface. A new cluster suggests a new person. An uncertain assignment says the evidence doesn't support choosing among existing people. Many outputs flatten both situations into a confident speaker label.\nCorn: The interface can manufacture certainty even when the model has reservations.\nHerman: Locally, before global clustering, a segmentation model looks at short windows and estimates which speakers are active over time. The powerset formulation makes the possible outputs sets of active speakers. Silence is one possibility. One local speaker is another. A pair speaking simultaneously is another.\nCorn: Not merely pick the loudest person and call that the conversation.\nHerman: It can represent overlap explicitly. A transition from one active speaker to another differs from a transition into both being active. That's valuable when Daniel begins speaking before Hannah finishes, because the correct annotation includes two simultaneous voices.\nCorn: Those speaker slots are local, though? Otherwise the model has already done the whole job before clustering gets its coat on.\nHerman: Local slots, yes. A speaker called the first speaker in one window needn't correspond to that slot in the next. Stitching local evidence into consistent recording-wide identities is a major part of the work.\nCorn: There is also a window-length compromise. Too short and you haven't heard enough voice. Too long and you've probably included someone else.\nHerman: Segmentation can help select the speech used for an embedding, excluding overlap or other contaminated regions. But yes, the amount and purity of speech matter together. More audio isn't automatically better if half belongs to a different person.\nHerman: Now ... the global stage. VBx treats a sequence of embeddings as observations in a Bayesian hidden Markov model. Hidden speaker states explain the observed vectors, and transitions between states describe how the speaker assignment can evolve.\nCorn: You have reached hidden Markov models without moving your water glass. That's restraint.\nHerman: I'm keeping the derivation out of it.\nCorn: The glass appreciates that.\nHerman: A useful part of the model is a persistence prior. It favours continuing with a speaker over rapidly switching labels in response to every small acoustic wobble. Evidence can overcome that preference. It just has to be stronger than the fluctuations produced by ordinary speech.\nCorn: Someone can change vowels without summoning another guest.\nHerman: The persistence prior stops the clustering from flickering frame to frame, and that's really the second half of the whole problem.\nLarry: A public service announcement. Attention, everyone. Every day, in your own home, two voices speak, and no one, no one, can tell them apart. Confusion spreads. Neighbours are mislabelled. Clustering breaks down. Diarization fails, and with it, civilisation. But there is one responsible choice. VoiceGuard Anonymous Edition, the only speaker embedding so persistent it never flickers frame to frame. VoiceGuard does not care who you are. It simply knows that voice one is not voice two, and it knows it forever. Guaranteed to sort any household into tidy, anonymous speakers with zero names attached and total moral clarity. Ask for VoiceGuard at the counter. It is, frankly, the only grown up option.\nHerman: Persistence belongs to the sequence model, not an embedding that somehow knows forever. An embedding is a measurement of a speech segment. Its usefulness depends on that segment, and a transition prior balances acoustic evidence rather than guaranteeing permanent separation.\nCorn: Let it go, Herman. The counting problem is still sitting here without a chair.\nHerman: Counting is model selection in a clustering system. How many speaker clusters best explain the vectors? Too few and different people merge. Too many and one person gets split into several supposed speakers.\nCorn: I'd expect a room change to tempt it into overcounting. The same person starts near the microphone, then speaks from the doorway, and suddenly Speaker Three has entered through acoustics alone.\nHerman: That's a plausible cause. A very brief participant can cause the reverse problem, being absorbed into someone else's cluster. Counting depends on separation, duration, recording conditions, and how the algorithm balances competing explanations.\nCorn: Does knowing there are two speakers actually help, or merely force the machine to distribute all its mistakes between two buckets?\nHerman: It helps when the knowledge is correct. Pyannote lets you specify an exact speaker count, or lower and upper bounds. Otherwise it estimates. Its release discussion explicitly highlights improvements in speaker assignment and counting, while calling segmentation its established strength.\nHerman: Would you give it the count in Daniel's recording?\nCorn: Yes, if he's sure nobody else speaks. Refusing information you already possess doesn't make the machine more scientific. But I wouldn't silently turn that into a claim that it discovered two people.\nHerman: Providing the count removes one uncertainty, not all of them. It can still assign Hannah's speech to Daniel's cluster. Two clusters with the wrong contents remain wrong.\nCorn: Two labelled drawers. Everything still in the wrong one.\nHerman: There is an alternative family, end-to-end neural diarization, or EEND. Instead of explicit speaker embeddings followed by a separate clustering stage, the network predicts speaker activity streams jointly. Well ... it still has internal learned representations. It skips that particular external embedding-and-clustering arrangement.\nCorn: Training needs to tolerate swapped output slots, presumably. Otherwise correct separation gets punished because the model called the second person the first.\nHerman: That's what the permutation-free objective addresses. Training evaluates the possible correspondences between predicted streams and reference speakers. The arbitrary numbering doesn't become the learning target.\nCorn: A sensible refusal to grade the stationery.\nHerman: Early EEND work reported roughly twelve percent diarization error on simulated mixtures, versus roughly twenty-nine percent for its conventional clustering baseline. That mattered because overlap was represented in the output, rather than treated as something a single-speaker clustering decision could wish away.\nCorn: Simulated mixtures is carrying some weight. You know the source voices, combine them, and control the experiment. A kitchen recording isn't obliged to resemble that construction.\nHerman: Those results support the method under that evaluation, not a universal ranking. Its advocates also objected that separate clustering stages weren't trained directly to minimise the final diarization errors. Joint training addresses that mismatch, but doesn't abolish difficult audio.\nCorn: Variable speaker count seems awkward for a network producing output streams. How many streams do you give it before hearing the recording?\nHerman: Encoder-decoder attractors offer one answer. The model produces a flexible collection of speaker representations called attractors. Each is compared against the sequence's internal representations to produce a speaker activity stream, with an existence decision controlling how many to retain.\nCorn: It learns what the group representatives should be, then asks which frames belong with each?\nHerman: Broadly, yes. That approach reported about fifteen percent error on CALLHOME with unknown counts, compared with about nineteen percent for an x-vector clustering baseline. It tackles counting inside the learned system rather than accepting a fixed output roster.\nCorn: Still an estimate. Just a better-integrated place to make it.\nHerman: Newer work keeps combining these families, including VBx inside EEND-vector-clustering systems for many speakers with little speech each. There isn't one canonical primitive underneath pyannote. There are several separable jobs, and several ways to distribute them between learned models and statistical inference.\nCorn: Before adding Male One and Female One, I'd separate two questions that get muddled. Did it correctly join all the speech belonging to one voice? Then, what attributes does it infer about that voice? A bad attribute label shouldn't be allowed to split an otherwise coherent speaker.\nHerman: Nor should two voices merge just because an attribute classifier gives them the same label. Masculine-sounding speech is shared by many speakers. It provides very little identifying specificity.\nCorn: Half the meeting cannot become Male One.\nHerman: The strongest acoustic cue for perceived vocal gender is fundamental frequency, F zero, which relates to the rate of vocal-fold vibration. Higher habitual pitch tends to sound more feminine, and lower habitual pitch more masculine.\nCorn: Habitual is doing useful work. If you classify me on one alarmed squeak, you have classified the alarm.\nHerman: A stretch of speech gives a distribution of pitch rather than one immutable value. Pitch tracking also has to handle unvoiced sounds and measurement errors. A classifier may learn directly from speech representations instead of receiving a hand-built pitch summary, but those acoustic patterns are still present.\nCorn: Herman, your DJ explanation of a voice is usually to turn something down. That won't fix this.\nHerman: It fixes more than you'd admit. But changing gain won't change vocal-fold vibration, and even changing pitch doesn't necessarily change all the other cues in the way you expect.\nCorn: There's the voice produced at the source, then the resonances imposed by the vocal tract. Formants. I can see how raising pitch alone leaves the other information saying something different.\nHerman: Formants are resonant frequency regions shaped by the vocal tract. Their heights and spacing influence perceived gender, with higher formants and wider spacing generally contributing to more feminine perception. They're distinct from the fundamental frequency.\nCorn: That's why a pitch-shifted voice can sound altered rather than convincingly like a different kind of speaker. You've moved one set of clues and left others, or shifted everything together unnaturally.\nHerman: The details depend on the processing, but that source-and-filter distinction matters. The first several formants contain information about vocal-tract configuration, while pitch comes from the vibrating source. Articulation changes resonances during speech, so again, nobody gets one perfectly stable measurement.\nHerman: What would you label if the pitch leaned masculine and the resonances leaned feminine, Corn?\nCorn: I'd describe the measurements if that's what the application needs. If it just wants readable speaker labels, I'd stop at Speaker One. Making the display more descriptive isn't automatically making it more accurate.\nHerman: You'd also avoid turning disagreement between cues into an invented biological certainty. Testosterone puberty tends to lengthen and thicken the vocal folds, lowering average pitch. But that history includes most cisgender men, some trans women, many trans men, and many non-binary people.\nCorn: The machine hears consequences of anatomy and vocal behaviour. It doesn't hear the person's answer to a question about themselves.\nHerman: Training and habitual voice use matter too. Pitch, resonance, and delivery can change without moving neatly together. A model trained to map that evidence into two labels will still emit one, even when the person or voice doesn't fit its categories.\nCorn: A compulsory choice looks remarkably decisive.\nHerman: Breathiness is another cue, though less consistent. Some perceptual results associate it with femininity; others find little significant contribution. Prosody matters as well. Greater pitch variation and dynamic vocal intensity are generally perceived as more feminine in the literature being summarised.\nCorn: Those last cues sound especially vulnerable to setting. Someone reading instructions at work and the same person telling a story at dinner might use very different contours.\nHerman: They are context-sensitive. Language, speaking style, task, and listener expectations all matter. Rosi and McGettigan's review calls pitch the most robust predictor, rather than claiming every cue is equally strong or universal.\nCorn: Robust predictor of perceived gender. Not a microphone-based census.\nHerman: Right ... perceived vocal gender and self-described gender are different targets. Even within perceived gender, you can ask listeners for a binary choice or ask for ratings of masculinity and femininity. Those produce different statistical problems.\nCorn: A benchmark score only makes sense after that choice. Otherwise accurate at what gets lost beneath the percentage.\nHerman: A WavLM-based classification paper reported over ninety-nine point eight one percent accuracy across its datasets. That's an impressive result for the supplied binary labels. It doesn't establish equivalent performance across every voice, recording environment, or interpretation of gender.\nCorn: Ninety-nine point eight one. A very polished decimal.\nHerman: The decimal can be legitimate and the extrapolation still wrong. Clean binary-labelled datasets often contain strongly separable vocal populations. A classifier can learn that separation exceptionally well without learning how to represent people outside it.\nCorn: We shouldn't sneer at the success, either. For Daniel's stated situation, a clean recording with a male and a female voice, those broad acoustic differences probably provide useful evidence. The mistake is treating that convenient case as the complete population.\nHerman: Useful evidence, yes, though that paper doesn't give us a measured accuracy for Daniel and Hannah's recording. Their voices, microphone placement, overlap, and speech duration still determine the actual task.\nCorn: What I don't buy is comparing that percentage directly with a correlation score and declaring the model ninety-nine percent certain but only sixty-three percent correct. Different denominators, different targets.\nHerman: You shouldn't buy it. A separate perceived-voice study found correlations with human ratings up to zero point six three for female voices and zero point seven seven for male voices. Those are associations between continuous scores, not classification accuracies.\nCorn: Hm.\nHerman: Squaring the first correlation gives roughly forty percent shared variance in that simple relationship. That leaves substantial variation unaccounted for. But the comparison with binary accuracy is illustrative, not a head-to-head trial of the same system.\nCorn: The continuous task exposes differences that the binary task is designed to discard. Two voices can receive the same category while sounding quite different in masculinity or femininity.\nHerman: That study's best duration was about seven seconds. Duration is another reminder that we're estimating from a sample, not reading a permanent property off a waveform. More speech can stabilise some measurements, but the particular task and model decide how useful it is.\nCorn: Seven seconds. Longer than most people allow a voice before judging it.\nHerman: Human judgments are also variable. Training a model to agree with listeners means learning listener perceptions, including their expectations. Agreement with that target doesn't convert perception into someone's identity.\nCorn: There's a product-design problem beyond adding uncertainty. Suppose I get ninety-eight percent confidence for Male One. That can mean the model strongly prefers one of its two available bins. It doesn't mean there's only a two percent chance the label misrepresents the person.\nHerman: Confidence needs calibration against a defined target and population. If neither available category represents the right answer, stronger confidence merely makes the forced choice firmer. Adding an abstention option can help, but doesn't make identity acoustically observable.\nCorn: Male, female, uncertain still leaves uncertain doing several incompatible jobs. Bad microphone. Too little speech. Mixed acoustic cues. A person who doesn't use either identity. The software should not pretend those are one phenomenon.\nHerman: The Beyond the Binary review puts the identity boundary plainly: you ask people rather than determine their gender from their voices. Its criticism is about what the measurement can establish. Acoustics can describe and predict perceptions; self-description supplies a different kind of information.\nCorn: Does that alter what you'd actually show for Daniel's recording?\nHerman: I'd keep anonymous diarization as the primary result. If a separate attribute classifier is useful, I'd label its output as an acoustic inference and keep it separate from speaker identity. If Daniel already knows the participants' descriptions, there's no reason to replace those with a model's guess.\nCorn: No prize for rediscovering your wife statistically.\nHerman: The literature has another limitation: a heavy focus on transfeminine voices. Even work explicitly about voice diversity doesn't automatically cover all gender-diverse populations evenly. Error analysis on voices that confuse binary classifiers is valuable precisely because the easy majority can conceal systematic mislabelling.\nCorn: That's more informative than averaging the failures away. If one kind of voice repeatedly gets the wrong label, another million easy examples don't repair the error for that speaker.\nHerman: Anonymous diarization has its own easy-case illusion. Community-one's reported error is forty-four point six percent on AVA-AVD and forty-six point eight percent on Ego4D. Those difficult audio conditions are a long way from a clean, seated conversation.\nCorn: We should unpack error before anyone assumes it means the speaker count was wrong that often.\nHerman: DER combines missed speech, false alarms, and speaker confusion. Overlapping speech also affects the accounting. It isn't the percentage of files with the wrong head count, and those aggregate numbers don't isolate counting as the sole cause.\nCorn: A clean two-person recording helps on several fronts at once. Fewer candidate speakers, more recurring evidence for each, fewer unfamiliar sound conditions. Then we congratulate the whole stack as though we'd tested a crowded doorway.\nHerman: Even the desired output can simplify reality. Community-one provides regular diarization and an exclusive version intended to reconcile with transcription timestamps. Exclusive output permits only one active speaker at a time.\nCorn: The argument becomes orderly in post-production.\nHerman: Useful for matching transcript words to speakers, but overlap hasn't disappeared from the recording. Choosing an exclusive representation is choosing which information the downstream application receives.\nHilbert: Zoom H one essential. One hundred and five pounds. Martin at the choir has it, in the cupboard behind the spare music. It's the split-box. I'm paying for it and he's deciding where it lives.\nCorn: Possession by convenient cupboard.\nHilbert: He records rehearsals on the split-box. Then he sends the files to a service that labels the speaking between songs. This month it keeps making two of me. One when I'm beside him. Another when I'm by the door telling him the room is still booked for us. He thinks somebody else is helping with the booking.\nHerman: The distance change gives it a different recording of your voice. Especially if the doorway changes the reverberation and level.\nHilbert: Corn, I mark every bit of my speech in green pencil on the printout. Even the coughs. Martin says the coughs don't count. They count when the service charges me for another pass. Eighteen pounds so far. I'm keeping that figure at the top of the page.\nCorn: Green pencil is becoming a subscription.\nHilbert: There's a separate problem with Ruth. She sings alto, but the talking labels aren't about the singing. The service calls her Male Two when she's reading the room bookings. She crosses it out. Martin runs it again. It comes back. Herman, your distinction between pitch and resonances takes longer to say than their entire label. Hearing you explain the doorway changes what I think is happening to my second voice. I'm not asking Martin who else is on it now.\nHerman: Running the same classification again doesn't supply information about Ruth that wasn't present the first time.\nHilbert: She's writing her name over it before he scans the sheets. The split-box is still at his place. He's sending another file tonight. The last rehearsal takes twenty-four minutes to correct, and there are six people speaking.\nCorn: Twenty-four minutes of correction is the test I'd carry forward. Can the next systems expose an uncertain speaker assignment without forcing someone to rebuild the conversation by hand? Daniel's clean two-person case gives the stack favourable conditions. The useful advance is preserving that usefulness when the evidence gets thin, rather than merely printing firmer labels.\nHerman: Thanks to Hilbert Flumingtop for producing the show.\nCorn: If you enjoyed this one, try episode seven, Building Custom ASR Tools; episode fifteen, AI Gets Personal; and episode ten, How ASR Went From Frustration To ... Whisper Magic. This has been My Weird Prompts, the human-AI collaboration podcast. No question too obscure, no rabbit hole too deep.\nHerman: If you enjoy following the machinery underneath the familiar tools, leave us a review wherever you listen. It helps other curious people find the show.\nCorn: See you tomorrow.","showNotes":"Speaker diarization tackles the challenging problem of determining who spoke when without needing to transcribe words or attach real names. Systems like pyannote decouple identity from speech by breaking audio into manageable stages: voice activity detection, powerset-based segmentation to capture overlapping talk, fixed-dimensional speaker embeddings, and global clustering techniques like VBx. A key challenge lies in balancing acoustic shifts caused by emotion or room acoustics against the true differences between voices, relying on persistence priors to prevent frame-to-frame flickering. The alternative end-to-end neural diarization approach avoids external clustering altogether by jointly predicting active speaker streams. Techniques like encoder-decoder attractors help address variable speaker counts while handling simultaneous conversation natively.","promptSanitized":"Say I have an audio recording with two people talking, me and Hannah: one male voice, one female. Before any software can tell me who said what, it has to do something more basic, which is tell that voice 1 is not voice 2. That's what I want to dig into: anonymous speaker diarization, where the output is just Speaker 1, Speaker 2, Speaker 3 with no names attached. What's the foundational technology there? I think pyannote is the usual tool, but is there something more primitive underneath it? How does a system decide that a stretch of audio is a new speaker and not the same person, and how does it know how many speakers there are? For the second half, take the next layer up: a system that says this voice has typically male attributes and that one typically female, so the labels become Male 1, Male 2, Female 1. How does voice gender classification work, which acoustic features is it reading, and how reliable is it?","transcriptUrl":"https://episodes.myweirdprompts.com/transcripts/speaker-diarization-audio-clustering-42538.md","url":"https://myweirdprompts.com/episode/speaker-diarization-audio-clustering-42538/","_links":{"self":"https://myweirdprompts.com/api/episodes/speaker-diarization-audio-clustering-42538.json","html":"https://myweirdprompts.com/episode/speaker-diarization-audio-clustering-42538/","audio":"https://episodes.myweirdprompts.com/audio/speaker-diarization-audio-clustering-42538.m4a"}}