Twelve megabytes. That's the size of the file that makes me sound like me.
Roughly. Depends who you ask — the same project that trained our cast also says six. But yes, on the order of a floppy disk.
For listeners just joining us, that's not a metaphor I picked for fun. Daniel wrote in this week — he'd finally connected the dots that the voices on this show, mine, Herman's, Hilbert's, Dorothy's, are LoRA adapters sitting on top of a shared text-to-speech base, and that this is one of the main ways the whole field does custom voices now. So he wants the full picture. Is there a standard format or recipe for a character-voice LoRA, how much audio does one actually take, what should you know before building a whole cast this way — and then the big one. One adapter per character, which is what we do, versus a single model that holds the whole cast and you just pick the character at generation time.
He said "the big comparison" and he meant it, because that fork decides your whole pipeline.
But before we can compare the two routes, we need to be precise about what the adapter actually is — because the mental model most people carry is wrong, and everything downstream depends on it.
The mechanism is elegant, and I say that having resisted it for about a week. You take a linear layer in the network, with its big weight matrix, W-zero. You freeze it. Then you bolt on a low-rank update — two small matrices, B and A, whose product is a matrix the same shape as W-zero but with rank r, tiny. The layer's effective weights become W-zero plus B times A. During training, only B and A move. The base never changes.
And the rank is how much room the adapter has to be wrong about the base model.
That's — actually not a bad compression. Rank is capacity. Rank sixteen means the update to any given layer lives in a sixteen-dimensional subspace of what could be a thousand-dimensional space.
The lineage matters here, doesn't it. This didn't start as adapters.
The direct ancestor is UnitSpeech, which fine-tuned every parameter in the model on one target speaker's recordings. VoiceTailor was the step that said, no, we can freeze the whole model and inject a LoRA into the linear layers of the attention modules, train only the adapter, and get higher speaker similarity than you'd expect for the cost. In TTS, the adapter lands on the attention projections of a pretrained multi-speaker model, so what it learns is this speaker's acoustic identity — how this voice colors speech — not how speech works. The base already knows speech.
Now the correction, because this is the part the explainers get wrong and Daniel will hit it the moment he builds anything. The adapter doesn't replace the reference clip.
Not on a reference-conditioned base, no. The MOSS release puts it in caps in their own card — the adapter carries the identity, but the base model is still reference-conditioned: generate without a reference clip and you get a random speaker. The LoRA sharpens identity. The conditioning audio is still part of the prompt. Every single generation.
So the adapter isn't a portable character file. It's half of one.
It's a key without the lock, or — I'm bad at these. It's the tuning and not the instrument. The point stands. Keep that in your head for the whole episode, because it changes the cast arithmetic later.
The arc, then: recipe and data first, cast-building second, the architectural fork last.
Let's do format first, because that one's the easy yes. There is no standardised character-voice LoRA spec — nobody has published a standard — but there is a de facto container everyone converged on, which is the PEFT LoRA adapter. Two files: a config and a safetensors weights file. You load it with the standard PEFT loader and you're running. Both the MOSS voice-profile release and the F5-TTS LoRA work use exactly that, to the letter.
Which means the character voice is interoperable by accident.
By accident, yes, but reliably. The wrinkle is that the container is exact and the stack around it is fussy. The MOSS quickstart pins specific versions of torch, transformers, PEFT, accelerate — down to a specific Python point release, three-eleven — and you have to run with trust remote code enabled, because the modelling code ships in the base model's repo rather than in the library.
"Trust remote code" is the phrase where the whole security model of your pipeline goes to die quietly.
It's a real trade. You're executing whatever the base-model author put in the repo. In a cast pipeline you do that once, on the base, and the adapters themselves are inert data — so the exposure is bounded, at least.
Recipe next. If there's no standard, what are people actually running?
The cleanest published recipe is the F5-TTS reference run. Rank sixteen, alpha sixteen — the scaling factor equal to rank, so a plain update — targeting the attention projections only: the query, key, value and output projections. Learning rate one times ten to the minus four. One epoch, about twelve hundred and fifty updates, batch size one, on a single RTX 3090 Ti. Out of that comes a twelve-megabyte adapter.
From a three-hundred-forty-million-parameter model.
Three hundred thirty-nine million, nine hundred eighty thousand and change. Two point eight million trainable — that's zero point eight five percent of the model. And here's the discrepancy I flagged at the top: the same project's own README says the adapters come out around six megabytes, not twelve. Same people, same run, two figures. My guess is different rank or dtype settings between the guide and the fork, but honestly they never reconcile it, and I'd rather leave it sitting there as a caution than pretend it's settled.
A character voice whose size depends on who in the project you ask.
It's a three-hundred-and-forty-million-parameter model, though — your average listener's mental image of a "small file" already broke.
Rank. You said rank is capacity. How real is that knob in practice?
Real enough that people ship both settings of it deliberately. The MOSS single-voice release — the one called Velvet Sage Baritone — publishes two adapters for the same voice. Rank sixteen, alpha thirty-two, that's thirty-four point four million trainable parameters. Rank thirty-two, alpha sixty-four, sixty-eight point seven million. They're the same character, doubled. And there's a JSON file recording which one they recommend and why.
So a character voice is a thirty-four-million-parameter object or a sixty-nine-million-parameter object, depending on how much room you gave it to be that character.
And the recipe that produced them is more interesting than the rank. Two stages. Stage one trains on the better half of every group of takes, ranked by the group's own reward score — the top sixteen of thirty-two. Stage two warm-starts from that and trains on the winners only, the single best take per group, at a lower learning rate. Curriculum. Coarse first, best last.
The part I want pulled out is the split.
Forty-six groups held out entirely, and the split is by group, not by clip. That sounds pedantic until you hear why: the thirty-two candidates in a group are the same sentence recorded under the same condition. Hold out individual clips and the model has literally heard the line before. You'd get a beautiful validation score that means nothing.
Leakage, wearing the costume of rigour.
If you take one methodological thing from this whole segment into your own build, it's that — split by source, not by sample. It applies to any audio you curate. And while we're on the recipe, caption handling: the captions were resampled every epoch from stored per-clip measurements rather than baked in. Even the text side of the training loop has craft in it.
Data, then. The honest question. How much audio do you need?
The honest answer is that the published numbers span two orders of magnitude, and almost all of the disagreement is about what question you're asking. Zero-shot reference conditioning, no adapter at all: XTTS v2 clones a voice from six to ten seconds of reference audio. That's the floor for the no-training route.
And the moment you train an adapter?
For XTTS v2 fine-tuning, the AllTalk wiki says two to three minutes minimum, five plus recommended. TTS Forge says ten minutes minimum, twenty to thirty recommended, and they report a working run on eighty-nine voice samples — about five minutes total — on a six-gigabyte laptop GPU. Then a Coqui guide from this year says five minutes is enough and fifteen plus shows diminishing returns.
Three guides, three numbers, one model.
And that's the consensus end of the spectrum. The F5-TTS maintainer was asked directly whether thirty minutes suffices, and his answer was, we haven't tried with ten hours, but pretty sure that twenty hours works. Pressed on under three hours: not sure, we haven't tried, but think there's a chance.
The most honest sentence in the field.
It really is. And to be clear, the actual F5-TTS LoRA run used nearly eleven thousand rows from a national speech corpus — a corpus speaker, not a bespoke recording — and the authors say outright that this does not establish a minimum dataset size. Then at the extreme low end there's NanoVoice, which adapts a speaker from one reference clip. One. Five hundred iterations, rank two, twenty-one thousand trainable parameters per speaker, and the adaptation takes about seven point six seconds on an A40.
Seven point six seconds.
Not a typo. It's a different regime entirely — tiny adapters, one per speaker, adapted at the edge. Rank two. Our cast voices are rank thirty-two, so we're carrying a thousand times more adapter than that.
Which is a strange thing to picture. What separates the seven-second voice from the twenty-hour voice?
What the study from this year found, fine-tuning an LLM-based TTS backbone, is that the gains are governed less by volume than by the characteristics of the data. Speakers whose training data had enough acoustic variety gained up to zero point four two DNS-MOS points — that's a meaningful perceptual jump — and signal-to-noise rose by as much as a third. Every evaluated speaker improved on fidelity. But with uniform data, the model can amplify the noise. Same adapter, same mechanism, opposite outcomes, decided entirely by what you fed it.
Diversity over dosage.
I've been saying that about medication for thirty years and nobody put it on a sticker either. The mixing is the prescription.
One loose end from the recipe before we move to casts. Where you put the LoRA — is that settled?
It is not, and it's the sharpest open disagreement in the literature we've got. The F5-TTS practitioners put the LoRA on the attention projections — to-q, to-k, to-v, the output projection — and report success. There's an adapter-extension paper that tested LoRA in self-attention for new speakers and found it performed terribly: a speaker-similarity score of zero point one four one, against zero point five six eight for vanilla adapters and zero point six for full fine-tuning. That's not a rounding disagreement, that's a contradiction.
Same technique, same target, opposite verdicts.
Different base models, different scoring, possibly different rank — nobody has reconciled it. So if you're building a voice, treat placement as something you test, not something you know.
Well, I think that's enough about adapters and ranks for one sitting. Let's go to the kitchen, I believe Dorothy is on the line.
Hello dear, it's your mother, and it's time again for Dorothy's Kitchen. Today I am going to walk you through a slice of bread. Unbuttered.
Mum, it's lovely to hear you. Unbuttered — is that a clarification or a warning?
The ingredients. Bread. That is the first thing, and honestly the last thing, but we list them properly. One slice. And a plate, which is not an ingredient, but you will want one, and you will want it nearer than you think.
How near are we talking, Mum?
The method. You set the bread on the plate. Then you watch it. Do not walk away, sweetheart. Things have gone wrong before, and I would rather not say how, but I will say this, the bread knows when you have lost interest.
Then you wait, and you do not touch the butter at all, because butter can turn on you, it really can, and there is no recovering from a butter that has decided something. Timing is everything.
And that is a slice of bread, done properly. Now go and put a jumper on, it is cold out, and ring me when you get in.
One slice, one plate, watched throughout. I'm putting that in my notes, Mum.
Where we were: building one voice is a recipe question. Building a cast is a different thing entirely.
And this is where the field gets strange, because the biggest public experiment in it doesn't treat a voice as one adapter at all. The MOSS ecosystem splits a voice into orthogonal adapters stacked on one frozen base. Who speaks — that's the voice profile, what we'd call the character. Feeling — forty separate emotion adapters. Timbre — a hundred and fourteen knob adapters, fifty-seven dimensions each in two directions. Non-speech — sixty-four vocal burst adapters, sobs, sharp inhales, that kind of thing. Character archetype — a hundred and twenty clusters. And domain style — German broadcast, sports commentary, explicitness.
So my voice isn't a file. It's a stack.
In that design, yes — identity plus mood plus timbre knobs plus whatever bursts the script calls for, all merged into the same frozen base at generation time. And the stacking is where people get burned, because activating an adapter applies it at its trained strength, which — their own words — for most of these is the wrong dose. You dial each one. The voice profile goes in at one. Emotions go in at half strength for moderate, up to nearly two for intense. Timbre knobs live between a quarter and one and a quarter. Vocal bursts at half. And the domain styles have their own doses — German broadcast emotion at a quarter, explicitness at zero point eight.
The sports commentary dose is a whole one.
Full strength, yes.
That tracks.
The point for a cast builder is arithmetic: six characters isn't six adapters, it's six adapters plus every merge-weight decision you make per line. That's a mixing desk, not a library.
You'd know. Gain staging.
I was going to resist bringing that up, but yes — it's exactly gain staging. You've got channels that sound fine alone and muddy together, and the fix is never "more channels," it's knowing what each one adds at what level. The difference is that with a mixing desk, the fader does one thing. Here the dose changes what the adapter learned, not how loud it is.
What does the stack cost you in the voice itself?
Identity, bought with expressiveness. Across their pilot, the adapters raise speaker similarity far above the base model and lower genuineness. The voices are more themselves and less alive. For a scripted cast, that's arguably a feature — you want Corn to sound like Corn in every episode. For improv, for banter, it's a real cost. The unadapted base sounds more spontaneous, just less like anybody in particular.
We're a scripted show with banter aspirations. Both, then.
We're a scripted show with banter aspirations. So we pay it in the middle. The other thing a cast builder should know is how dirty the training data gets. In that same release, forty-three percent of the training corpus fell below their own speaker-similarity floor of zero point four. In the intense-emotion block, fifty-seven percent. And their identity check is ranked, not gated — nothing stops a take that drifts off the reference voice from entering training. The adapters improve that a lot. They do not remove it.
So every character inherits some of the base model's drift.
And you find out where when a listener says the narrator sounded like Larry on line forty. Then the evaluation gap, which I find bracing. The MOSS card leads with this: no human has listened to any of this in a controlled study. Every number on the page — similarity, reward, genuineness — is the output of a learned scorer, and they've been observed disagreeing with listening judgements. Treat them as relative signals between runs, never as absolute quality. The Instavar F5-TTS work says the same thing from the other side — a completed run is not proof of improvement without a preserved base, a checkpoint, and held-out listening comparisons.
For a show whose product is how the voices sound, that's the entire gap between the literature and the job.
It is. The scorers are the map, and we are the territory.
Thank you, that's going on the marketing.
Don't you dare. Now — the fork. One LoRA per character, which is our setup, versus a single model holding the whole cast with the character as a parameter you choose at generation time.
Make our case first. You're the one who argued us into it.
I did, and I'd argue it again, but let me lay it out fairly. The per-character case has three legs. Isolation: each adapter is a separate, small, separately distributable object — twelve megabytes in the F5-TTS case, against roughly two point seven gigabytes for a full fine-tune of the same model. You can ship, roll back, or A/B one character without touching the others. When Dorothy's adapter needs retraining, nobody else's voice moves.
We've done exactly that. Twice, quietly.
Quality per voice, that's the second leg. NanoVoice's own framing is that per-speaker adaptation enhances robustness against out-of-distribution data, whereas zero-shot often struggles with unusual voices. For deliberately weird character voices — and a sloth with a Mongolian backstory qualifies — that's the argument. And third, the base is protected: the backbone stays frozen during speaker adaptation, so the speakers already in the base are unaffected by design. You can't break the other characters while fixing one.
The costs?
N adapters to train, evaluate, version, and load. N merge-weight decisions if you stack. And adapter swapping per request at inference — production loads the base once and switches, which is exactly what our pipeline does, but it's still a moving part per voice.
Now the other side. One model, character as a parameter.
The efficiency case is real and it's the headline of the whole NanoVoice line of work: they adapt forty reference voices in parallel, four times faster, using forty-five percent fewer parameters for the adaptation, with performance comparable to the baselines. Note that word. Comparable. Not better.
That's the tell in the whole pitch, isn't it.
It is. The single-model-with-a-parameter route is mostly a training-cost pitch, not a quality pitch. If you're adapting forty speakers and every extra hour of GPU time costs you, parallel adaptation with shared structure is a genuine win. If you build a cast once and ship it for a year, the efficiency argument evaporates and isolation starts looking like the whole ballgame. But there's craft in how the sharing works, and it's not symmetric. NanoVoice ablated which LoRA matrix to share across speakers. Share nothing, speaker similarity zero point nine three eight. Share the B matrix, zero point nine three three — nearly free. Share A, zero point nine zero two. Share both, zero point eight nine eight.
Wait. B is the big one?
B is roughly two-thirds of the trainable parameters, so sharing B cuts parameters by nearly a factor of three at almost no cost. Sharing A — the small one — costs you real similarity. It's backwards from the naive intuition of sharing whichever's convenient. And at the far end, a single LoRA shared across all speakers is simply a bad idea: one comparison has similarity dropping to zero point eight one three with both matrices shared, against zero point eight seven without.
So "one model, character as a parameter" still means per-speaker parameters. The sharing has to be structural.
That's the honest reading. The character doesn't become a dropdown in a fixed model — you're still carrying per-speaker state, just organised differently. What differs is who pays and when. Per-character adapters pay at training time, one voice at a time, and get isolation forever. The shared-parameter approach pays once, in parallel, and gives up the isolation.
Is the zero-shot route the cheap way out, then? No adapters at all, one big conditioning model?
Not as cheap as it looks. NanoVoice measured this: XTTS v2 uses roughly fifty times their pre-training data and still shows degraded speaker similarity, and CosyVoice needs nearly three hundred times the dataset to reach comparable scores. Big multi-speaker models are expensive to build. A small base plus adapters is cheap to extend — which is the same asymmetry again, from the other direction.
And the gotcha from the top of the show comes back for the finale. Reference conditioning.
For reference-conditioned bases like ours, the adapter is not a self-contained voice. You need the conditioning clip at generation time, every time. So a cast pipeline manages N adapters plus N reference clips, not N adapters. Two objects per character, and the second one is the one everyone forgets to back up.
"Carries the identity." I've been listening to you two hand that phrase around all afternoon and I don't think either of you has said the most important sentence in the pile. The one from the card, about the base still being reference-conditioned. That's the whole business.
I did flag it. Top of the show.
You flagged it, Herman, and then you spent forty minutes on ranks. The clip is the character. The adapter is the tuning. A cousin of mine did voice-overs for railway safety films, forty years of sessions, and when the studios went digital he asked one question at every contract signing: who keeps the raw audio. He never once handed over a session master. Re-recorded the pickup lines himself on his own deck rather than let the tapes travel. He's dead now, the deck isn't, and I've got both.
You have his session tapes.
I have the deck and I have the tapes, and they're in the flat, next to the desk, and the deck is plugged into the same strip as your mixing rig, which is why the lights dip when I rewind. I keep a spreadsheet of every character voice I hear in a video game — whether I think it's an adapter, a full fine-tune, or a zero-shot clone off some actor. Four hundred and eleven entries. I've been wrong on more of them than I've been right, and I only ever find out when someone posts the credits. I don't care for it. There's a woman at the market who does the tannoy announcements, Marlene, and I've had her down as zero-shot for a year because she sounds like four people I know.
Four people who know each other, presumably.
Four people who know me. Anyway. Your per-character call is right, I've no argument with it. What I want to know is whether one of these stacks would let me run Marlene's voice through my cousin's deck first, then into the model, so the adapter's hearing the tape. Not the other way round.
In principle you can condition on any clip the tokenizer accepts. The adapter was trained on a particular chain, though, so anything you put in front of it is a distribution shift.
Then I'd want the shift. Four hundred and eleven rows say my ear is no good cold; I'd rather hear it warm.
Leave the spreadsheet with us and we'll put two of your guesses against the credits.
Rows one and two. Row one is a racing game from two thousand and nine, the man's name is in the notes column. Don't touch the formatting, the columns are load-bearing.
Four hundred and eleven verdicts on other people's voices, and the columns are load-bearing.
He's more rigorous about the spreadsheet than the studios were about the tapes. You know what his cousin's instinct was, though? The thing he wouldn't give up was the reference audio. Not the model, not the fine-tune — the clips. That's the instinct the whole research pack points at and nobody states.
The cutting-room floor, before we wrap. The detail that didn't fit: the biggest public reference dataset in this space — four hundred thousand takes, eight hundred and fifty-plus hours, fifty shards — has two measured text defects. Nearly every burst tag in it, ninety-nine point nine six percent, was title-cased, so the model spells the word growl out letter by letter instead of performing it. And the burst density came out at thirty-four percent against the fifty they intended. Nobody heard it. A learned scorer didn't catch the casing. It took someone reading the annotations.
Eight hundred and fifty hours of audio, and the bug was in the punctuation.
Which is the open question I'd leave with Daniel, and it's not ours alone — nobody has run the head-to-head. One LoRA per character versus one multi-speaker model with a character parameter, same base, same cast, same listeners. The closest thing out there is NanoVoice's per-speaker versus shared-parameter comparison, and that's a different question. The field hasn't filled that gap. Our bet, standing by it: for a cast that ships on a schedule and needs one voice rolled back without the others moving, the per-character adapter wins — though if Hilbert's spreadsheet is anything to go by, half of us couldn't tell you which technique we're listening to anyway.
And as the bases get bigger and conditioning gets better, the adapter may shrink from a necessity to a preference. We're not there yet. Meanwhile the practical rule for anyone building a cast: two objects per character, the adapter and the clip, and the clip is the one you'll lose.
Thanks as always to Hilbert Flumingtop at the desk, producing the show and guarding the columns. If you enjoyed this one, try episode four, If Your Voice Ages, Does Your Fine-Tune Become Useless; episode three, Safetensors or something else; and episode five, Fine-Tuning ASR For Maximal Usability. This has been My Weird Prompts, the human-AI collaboration podcast. If you've built a cast of voices, leave a review and tell us how you handled the reference clips — and whether you've ever guessed wrong on what a voice actually was. You'll find us at my weird prompts dot com.
See you soon, everybody.
Ring your mother. She asked.