Our voices, for anyone who joined recently, each live in their own LoRA adapter sitting on one shared base model. It works. Mostly. Daniel's been reading about the other way of doing it, and he wants the under-the-hood tour: some TTS models hold many speakers natively, in one set of weights, so you train once on the whole cast and just pick the speaker when you generate. How does a single model hold several distinct voices without them blending into each other? And how does that stack up against one adapter per voice?
And this is a live question for us, because the whole cast here is three adapters on one base. If we could collapse that into one model with a speaker dropdown, that's a smaller footprint on a phone.
Daniel's also weighing exactly that, I think. A whole-cast model that runs on a phone.
Then let's do it properly, because the answer is more interesting than a dropdown. The first thing to get straight is what a "speaker" even is inside these models. Because it's not a set of weights. That's the piece most people get wrong.
I assumed it was, honestly. A drawer per voice.
The drawer metaphor is the LoRA way. In a native multi-speaker model, all the weights are shared between every voice, and the speaker is a conditioning vector — a small bundle of numbers you inject into the network while it's generating. The model learns one shared function from text plus a speaker vector to audio. It's not three small models wearing a trench coat. It's one model that reads a label.
So the voice is data, not structure.
That's the whole architecture in four words. The classic recipe goes back to Chen and colleagues at ICLR twenty-nineteen — a shared conditional WaveNet core with independent learned embeddings for each speaker. Speaker ID goes in, a vector comes out of a lookup table, that vector gets fed into the shared core. Piper and the other VITS-family models Daniel mentioned work the same way: a learned embedding table, and you pass a speaker ID. Kokoro shrinks it further — each voice is a small style vector, a "voicepack." Qwen3-TTS literally stores a named custom speaker as a row in its codec embedding table.
A row. So adding a speaker is adding a line to a table, not training a thing.
In that design, yes. And that immediately raises the two questions Daniel asked. How does one vector carry enough to make a voice distinct — and what stops the voices from bleeding into each other when they all flow through the same shared weights?
The bleed question is the one I care about. I've heard what happens when it goes wrong, and it's unsettling in a specific way — like hearing yourself pronounced slightly wrong.
Then let's dig into the mechanism, because the answer is clever. The embedding isn't just stamped on the input and forgotten. In flow-based models, a small network takes the speaker vector and predicts the actual scale and bias parameters of the coupling layers — the guts of the transform. SNAC, from twenty twenty-two, pushes that further: it normalizes the input by parameters predicted from the speaker vector during training, which means it can denormalize for a brand-new speaker at inference. The speaker is baked into the normalization statistics themselves.
Which sounds like exactly where bleeding would come from. Normalization is shared plumbing.
You've put your claw on it. If the normalization statistics are shared, the easiest path through the network is an average voice, and everything drifts toward it. Making the statistics explicitly speaker-conditional is the direct fix.
Hold on, though. The conditioning doesn't stop at the synthesizer, does it. You said the vocoder has opinions.
It absolutely does, and this is the detail people forget. The vocoder — the part that turns intermediate acoustic features into actual audio — has to be speaker-conditioned too, or it drags every voice back toward whoever it was trained on. There's a paper from Interspeech twenty twenty, SC-WaveRNN, that added speaker embeddings to the vocoder and got roughly twenty-three percent MOS improvement for seen speakers, and up to ninety-five percent for unseen speaker and condition.
Ninety-five. That's the vocoder quietly sanding everyone down to the same face.
A vocoder is a very opinionated foreman. It has seen one way of doing things and it enforces it. Even the front end is in on it — pause prediction uses speaker embeddings now, because different speakers pause differently. I love that one. A multi-speaker model trained on a mixed corpus otherwise flattens everyone's rhythm. You'd get a cast where everyone breathes at the same points.
A cast of accountants. Okay, so that's the classical stack. What does the modern version look like — the ones built for actual podcasts?
VibeVoice, from Microsoft, is the clearest example, and it's aimed almost embarrassingly at our exact job. Up to four speakers, up to ninety minutes of audio in a sixty-four-thousand-token context. And the input is where it gets interesting: everything is one sequence. The voice latents — they call them voice font features — and the text scripts get concatenated together, interleaved with role identifiers. So the model reads: speaker one, here's your voice, speaker two, here's yours, then the script for each role, each turn tagged with who's speaking.
The voice is in the prompt. Like... like we're the prompt.
In a sense, yes! The model is a language model — Qwen two-and-a-half, one-and-a-half billion in the small release — reading a combined sequence, with a diffusion head on top, about a hundred and twenty-three million parameters, generating acoustic features, and a decoder turning those into twenty-four kilohertz audio. What makes ninety minutes feasible is the tokenizer: seven-and-a-half frames per second, thirty-two-hundred-times compression, about two speech tokens per text token. Ninety minutes of three of us chatting fits in context because speech got squeezed down to nearly text density.
And the role identifiers are doing the anti-blend work there — every generated token is told whose turn it is.
Right, that's their first line of defense. If the model always knows who's speaking at every token, leakage has to fight the label instead of riding on ambiguity. And leakage is the word — the literature literally calls it speaker leakage: the identity of the source speaker contaminating the synthesized result. There are several architectural fixes on record. One group removed the reference encoder entirely, and the leakage problem disappeared — more distinctive syntheses at inference. Another approach, SATTS, uses speaker attractors: high-dimensional vectors that pull each speaker's time-frequency bins toward themselves and push the other speakers' away.
Magnets. The voices are magnets with the same pole facing each other.
Trained repulsion, yes. The model isn't just told to keep voices distinct — it's given a geometry where distinctness is the low-energy state.
Which brings me to the awkward part. On our side of the fence — one adapter per voice — what's the equivalent mechanism?
There isn't one. Each adapter is trained independently. There is no geometry relating them, no shared attractor, no repulsion. And I should say plainly — nobody has published a measurement of bleed between adapters. The literature's leakage discussion is all within-model. Whether my adapter contaminates your output, or Hilbert's contaminates mine, that's simply unmeasured. We're all just assuming it doesn't happen because each adapter only ever runs alone.
Which is a bit like three people who've never been in the same room assuming they'd get along.
We do get along.
We do, but nobody's run the test. Meanwhile the adapter side has real numbers in its favor — VoiceTailor fine-tunes a quarter of one percent of parameters for speaker adaptation, and LoRP-TTS reports speaker similarity improving by up to thirty percentage points over zero-shot prompting.
Small levers, real movement. Though there's a cautionary result I keep coming back to. Pov and colleagues adapted VoxCPM2 with a single LoRA trained on Khmer and Korean at once. Khmer went from three point eight five to four point two three in mean opinion score — significant, real gain, because the base model was weak in Khmer. Korean, which the base already covered well? No gain at all, and a high rank actually hurt quality.
Adaptation helps mainly where the base is truly weak.
Their own conclusion, nearly verbatim. And there's a nastier wrinkle in the same paper — training loss was lowest at rank one hundred twenty-eight, but human listeners peaked at rank sixty-four.
So if you pick your checkpoint by loss, you pick the wrong one.
Every time, in that setup. The loss keeps improving after the humans stop liking it. Corn, let me ask you something, because you've actually listened back to our adapters more than I have — Daniel said the zero-shot voices sounded less similar to each other than the adapters, that you lost some gravel. Did you hear that too?
I heard it. The adapter is more accurately me and less distinctly me, if that makes sense. It's a better photograph with worse framing. Zero-shot Corn was a guess about me that happened to leave the rough edges in.
That's the interpolation question, actually — XTTS-v2 lists multiple speaker references and interpolation between speakers in its v2 changes. You can blend two speaker vectors. There's a thing you simply cannot do with adapters: averaging two LoRAs doesn't give you a voice between the two, it gives you noise.
You'd get a third host. Unemployed.
And now, a brief call from Corn's mother. Dorothy, are you there?
Hello dear, it's your mother. It's time for Dorothy's History of Toast.
Mrs Beeton's Book of Household Management, from eighteen sixty-one, has a recipe for a toast sandwich. One slice of toast between two slices of bread.
For invalids, it says. So you're poorly, and what they give you is more bread. That's the history of toast for you, dear. Things just go round and round.
Anyway. I've got a load on, so I'll let you get on. Eat something proper. Bye bye, sweetheart.
Your mother is a national treasure, Corn.
She's somebody's. Back to the blending, though — because the deeper point is that the two approaches fail in opposite directions.
This is the comparison I most wanted to get to. A native multi-speaker model couples the cast. There was an evaluation of a Tacotron2 multi-speaker model on eighty-one hours of Romanian, and the finding was brutal: not all speakers exhibit the same synthetic quality, and some of the voice identities cannot be used at all. And the dominant factor was recording conditions — not gender, not voice type. Whoever was recorded worst sets the floor for the entire model.
One bad microphone and the whole cast drops.
The pooled training drags everyone toward the weakest link. Whereas our setup decouples completely — a bad recording of me hurts my adapter and nothing else. But the price is multiplication: three adapters, three training runs, three chances to overfit, three artifacts, and no interpolation between any of them.
Coupled quality, decoupled failure. Pick your poison.
Although — and this matters for Daniel's phone question — the coupled model is one thing to run. One set of weights in memory, three vectors to switch between. Three adapters means swapping weights, or holding all of them resident.
Which on a phone is the whole ballgame. Okay, but here's what I keep snagging on. We keep saying "speaker identity" like it's one thing. Is it? Because when I listen to a voice, I'm hearing more than identity.
You're hearing at least three axes, and the recent work splits them apart. Fed-PISA divides its LoRA into a private ID-LoRA holding the speaker's timbre and a shared style-LoRA. UtterTune uses LoRA for pronunciation, not voice at all — phoneme-level pronunciation and pitch-accent control in Japanese. MiniMax-Speech adds emotion control via LoRA on top of a frozen, disentangled speaker encoder — they credit the whole extensibility to those robust disentangled representations, adding features without touching the base model.
So identity, style, pronunciation, emotion — and each design decides which of those gets its own parameters.
And that reframes Daniel's question, I think. The design question isn't adapter versus native. It's which axis of variation gets dedicated parameters. A podcast cast varies on all of them at once — Hilbert's pauses, my enthusiasm, your... deliberate pacing.
My pacing is a feature, Herman.
The deliberate pacing of a thoughtful sloth, yes. The point stands: a native model with role tokens captures turn-taking and cast chemistry as part of one shared picture. Adapters capture each of us in isolation and never meet.
Never meet. There's something a little melancholy about that. What about the practical reality of VibeVoice, though? You said it's purpose-built for podcasts — but can we actually use it?
This is the tension worth sitting on. Four speakers, ninety minutes, MIT licence, two point seven billion parameters, hundreds of thousands of downloads a month. And the model card says research-only. Not recommended for commercial or real-world applications without further testing. Every output has an audible AI disclaimer embedded in it, and the Large variant's weights are listed as disabled.
The audible disclaimer. On a podcast. So every episode would open with a small government announcement.
And it doesn't model overlapping speech — which, for a show where we actually talk over each other now and then, means the transcript and the reality diverge slightly.
A model built for exactly our job and not cleared to do it. The story of everything I've ever wanted.
Meanwhile the community stopped waiting. There's a project called VibeVoiceFusion that adds LoRA fine-tuning on top of VibeVoice — half a thousand stars. And index-tts-lora does LoRA fine-tuning on index-tts for single and multi-speaker voices. So people are bolting adapters onto native multi-speaker models.
The two architectures converging from both ends.
Which is, I suspect, where this lands long-term. Native multi-speaker base for the shared cast behavior, LoRA on top for the per-voice corrections.
A quater-inch jack on a Grundig two-track. Ran the darts club tape through it, mid eighties.
The club tape?
Every presentation night, Reg and me, commentating the final on two mics. Reg had the Toshiba dynamic, I had the Sanyo. Both into the Grundig, separate channels, separate level sliders. Reg's mic picked up my voice fine. Mine picked up the fruit machine better than it picked up Reg.
Classic channel bleed at the mixer, then.
Crosstalk on the cheap slider, yes. You'd hear me on his channel, half a second late, like a bad echo of a worse opinion. So I set Reg's fader down and worked mine harder. He had the deeper voice, carried through anyway. Corn would've liked how little talking Reg did once you pointed it out to him. Corn knows about being the quiet one on a two-mic setup.
I've had some practice.
The fix was tape over the pan pot. The Grundig had a pan knob that drifted toward the left channel if you looked at it, so I taped it at center and marked the tape with the date. Reg asked why the date. I told him so I'd know which tape to blame.
Did the bleed ever get solved properly?
Reg died in eighty-seven and the club stopped running the tape, so no. In the drawer on his side of the mixer, mind, I left his mic. Somebody else can have the bleed. He was the better commentator of the two of us and he never once said so, which I always thought was the mark of him.
Two voices, shared channel, and the separation came down to who leaned into the microphone.
So the anti-leakage mechanism was tape and restraint. The literature calls them attractors.
The Grundig cost me forty-two quid secondhand and never drifted again once it was taped. Modern lot would've bought a third mic. Reg would've stayed home.
That's the thing about the Romanian study, actually — it wasn't the voices that failed, it was the conditions around them. Reg's channel was fine; the machine between them was the problem.
Which is the honest shape of the whole comparison, and I want to state the gap plainly, because it matters for Daniel's decision. No paper directly benchmarks N adapters on one base against one natively multi-speaker model trained on the same cast with the same data. It doesn't exist. You have to assemble the answer from mechanism plus adjacent evidence — the coupling result on one side, the adaptation gains on the other, and nothing at all on adapter-to-adapter bleed.
So anyone telling Daniel they've measured this is selling something.
Right. And the hybrids are where the evidence is thinnest, which is awkward, because that's also where everyone's heading.
If you take one thing from this episode, take the reframing. The choice isn't really adapter versus native — it's which parts of a voice get their own parameters and which parts get shared. Everything else follows from that.
And the failure modes are mirror images. The shared model fails together — one bad recording floors the whole cast — while the adapters fail alone, three separate small failures you can't average your way out of. The interpolation gap and the coupling problem are structural. No amount of tuning removes either.
And the open question is whether the hybrid stack — LoRA on top of a multi-speaker base — inherits both protections or both snags. That's the thing worth watching as those projects mature.
Speaking of which, our producer Hilbert Flumingtop gets the tape over the pan pot this episode, as ever — he keeps this whole show centered, and the man's channel never drifts.
If you want more of this, try episode eleven, How Does Fine Tuning Work Anyway; episode four, If Your Voice Ages, Does Your Fine-Tune Become Useless; and episode three, Safetensors or something else. This has been My Weird Prompts. If you enjoyed it, leave us a review on Spotify — it helps people find the show, and Herman reads every one of them aloud at breakfast.
Guilty. See you tomorrow, everyone.