We are, right now, being spoken by the second generation of ourselves. That's the situation. The voices you're hearing are OmniVoice LoRA adapters retrained on the output of the previous Chatterbox voices, with a light EQ pass in between. And it worked. Suspiciously easily, if you ask me, but nobody did.
I asked, internally.
Daniel, though, did ask something better. He looked at how hands-off our retrain was, the voice-to-voice step basically handed to a model, and said: fine, but what does the deliberate version look like? If we actually sat down with my training set and Herman's and engineered the voices we want, on purpose, what's the craft? Curating and editing the data, controlling timbre and EQ and delivery, the frameworks and processes that exist for that. Which, I'll spoil, is a field that barely has a name yet, let alone a textbook.
Voice engineering, he's calling it. And he's right that it's a name for something real. The pieces exist, they're just scattered across practitioner blogs, per-model fine-tuning guides, and a stack of research systems nobody has stitched together. There's one proper survey of controllable speech synthesis that comes closest to a unifying reference, and even that is mostly a map of fragments.
Before we engineer anything, we should be honest about what the shortcut actually did. Because my suspicion is that we didn't take a shortcut. I think we accidentally did a real technique with two of the steps skipped.
That's a fair way to put it, and the evidence backs it. The core of what we did is a LoRA adapter on a zero-shot TTS system, trained on audio that was never studio-recorded. There's a paper called LoRP-TTS that looked at exactly this, and their finding is that low-rank adaptation lets a zero-shot system use, and I'm quoting because it's a good line, even single recordings of spontaneous speech in noisy environments as prompts. Speaker similarity went up by as much as thirty percentage points in their tests. LoRA is unusually tolerant of off-distribution audio. It's the tool that makes non-studio audio usable at all.
So we fed it the one kind of non-studio audio that was secretly studio-grade.
Every failure mode that wrecks a normal TTS dataset, cross-talk, room changes, clipping, transcripts that don't match the audio, two speakers on one track, was absent from the old voices' output by construction. The Chatterbox voices were clean, single-speaker, consistently miked, and the transcripts existed because the model generated the speech from text in the first place. We didn't curate a dataset. We inherited one that had already been curated by a machine that didn't know it was doing it.
The EQ pass, then. Herman got a cut around five hundred fifty hertz and air added above six kilohertz, which is the audio engineer's way of saying he was muddy and boxy. I got a small cut around three twenty.
Which, for the record, was the first time anyone has described my voice as muddy and boxy, and I've been listening to it for years.
It was a scientific finding, Herman. Not a review.
I've taken it well. Anyway, the EQ removed the one systematic spectral bias in the training audio before it could get baked into the adapter. And here's the thing that makes this interesting rather than routine. The closest published analogue to what we did is a Greek TTS system from Interspeech this year. They took audiobook recordings, aligned and filtered them into TTS-ready data, and fine-tuned an eight hundred eighty million parameter Parler model with a speaker-specific LoRA stage trained on three and a half hours of single-speaker audio, updating about five percent of the parameters.
Three and a half hours. That's a long afternoon of recording.
And the results were solid. Word error rate around ten point seven percent, which is about three points above what the speech recognizer itself gets on the same audio, and listener scores within spitting distance of the human recordings. Four point zero versus four point three six on similarity, four point two four versus four point three zero on comprehensibility. Robust single-speaker synthesis from limited curated data is achievable. We're not in fringe territory.
That paper has a warning in it, though. Tell me about the style prompts.
Yes, and this one bears directly on us. They found that LLM-generated style prompts introduce speaker drift at inference. If your pipeline is generating the style or reference prompts on the fly, the voice wanders. Replacing them with deterministic prompts, the same fixed prompts every time, resolves it. Now, we switched to per-voice LoRAs precisely because our zero-shot clones were drifting, Corn's clone famously started sounding like mine by month three.
Deeply disturbing at the time. Less so now that I know it was a documented phenomenon.
So the question I can't shake is whether the drift question compounds. The new Herman is a model trained on the output of a model. The old Chatterbox voice was itself synthetic. LoRA's tolerance for off-distribution audio is why the shortcut worked, but has anybody measured what happens on generation three, generation four? As far as I can tell, no. That's a open question, and it's our question in a way it isn't most people's.
We're the experiment.
We're the experiment nobody's writing up.
Here's the piece I keep turning over, though. Every serious data-curation guide I've ever seen says don't EQ your training data. Don't compress, don't run automatic gain control, because those operations flatten exactly the intensity and pitch cues that carry emphasis. And we EQ'd. And it worked.
And I think the reconciliation is that those guides are about human capture. A human recording has room tone, natural dynamics, real prosodic variation, all of which a sloppy EQ chain destroys. The old voices' output had none of that. It was already flat, already compressed, no room tone to preserve. EQ'ing it wasn't destroying a signal. It was correcting a defect.
Corrective EQ on model output versus destructive EQ on human capture. Two operations that look identical on the mixer.
Same knob, opposite meaning. And nobody has published on that distinction. Whether light-EQ-then-retrain is measurably beneficial, and by how much, appears to be undocumented. We didn't follow the rules. We may have accidentally found one.
So that's the deliberate version: curate the data, control the timbre, shape the delivery. But honestly, most people just want a voice that stays put.
Friends, are you tired of voices that drift, wobble, and forget who they are? Then you need VoxLoftra, the once-daily voice-stabilizing tablet from the makers of nobody in particular. VoxLoftra gently retrains your inner speaker model overnight, absorbing even single recordings of spontaneous speech in noisy environments, so you wake up thirty percentage points more yourself. Side effects may include mild boxiness, a faint muddy quality around the edges, sudden air where no air should be, spontaneous Greek audiobook narration, an adapter that will not detach, low-resource whispering, persistent room tone, and in rare cases, a second voice that knows things about you that you have never told anyone.
Anyway. Say more about what the deliberate version looks like, because I don't think the answer starts with the model.
It doesn't. It starts with a spreadsheet, and this is the part I find exciting, so brace yourself. There's a practitioner write-up from this summer, from a data company called Datatoorder, whose central claim is that the recording matrix is the dataset. The idea is you separate the axes before you record anything. Emotion, angry, sad, that's one axis. Register, news, audiobook, conversational assistant, that's another. Delivery, whisper versus projected, a third. And most datasets get this wrong, they list styles as one flat list, emotion and register and delivery all mixed together, and the labels overlap and the model can't separate them.
A style label like "excited news whisper" is three knobs taped together.
And every speaker who walks into the booth means something different by excited unless you've given each style a one-line definition and an audible example. That's the discipline the write-up pushes: define each style in words, play a reference take, and only then record. And when it comes to choosing the labels themselves, discrete categories beat dimensional ratings for most projects. Neutral, cheerful, serious, sad, angry, whisper, maybe with a strength level attached. Not valence and arousal axes.
Because a director can ask for angry, but nobody can ask for minus zero point four valence.
That, and raters can't agree on continuous numbers, and a model conditioned on two floating values needs far more data to learn the axes. Then the matrix itself. Sentences times styles times takes. Multiple takes of the same sentence in the same style. Their line on this is the best thing in the whole piece: a single take per sentence per style teaches the model that each style has exactly one prosody. Several takes give the model the distribution, what varies between renditions and what stays constant, which is what makes synthesized speech sound alive rather than stamped.
Stamped. That's the word for the zero-shot clones, wasn't it. Every sentence arrived with the same fingerprint on it.
One prosody per style is a stamping machine. And there are confounds to control. Same speaker records every style, or the model learns style and speaker are the same variable. Style order within a session gets randomized, because if the energetic styles always land in the first hour and the quiet ones in the last, the model isn't learning excited versus calm, it's learning fresh voice versus tired voice.
Which is a real thing in this household. By the last hour of anything, we are all quiet styles.
You're a quiet style by eleven a.m. And keep a rejected-take log with reasons. That log is half the value of the session, because it's your labeling quality control later. Then the scripts. Phonetic balance, obviously, but also prosodic sentence shapes. Statements, yes-no questions, wh-questions, lists, emphatic stress, long sentences with internal clause boundaries. The sharpest training signal you can build is minimal pairs where the words are identical and the meaning is carried entirely by stress or intonation.
"I didn't say you stole it." Seven meanings, one string.
That exact kind of thing, yes. Rare in off-the-shelf scripts, and exactly what a style-conditioned model needs. And the transcripts have to be exact down to the punctuation, because punctuation is the only text-side cue to phrase boundaries. A missing comma is a lost label. Capture discipline is its own list. Lossless or high-bitrate recording, gain control off, constant mic distance, watch clipping, because the emphatic takes are the loudest ones and they clip first. The line that stuck with me: automatic gain control removes the intensity differences that carry emphasis, which is the signal itself.
The cleanup chain that removes the thing you recorded.
You're cleaning the signal out of the recording. Keep a raw unprocessed file of every take alongside anything you process. Corn, let me turn this around, because you've thought about the data side more than me on this. Where would you start if we did it for our own sets? Not the frameworks, the actual decision.
With what the matrix would even be for us. We don't have styles. We have one register each, roughly, mine is unhurried and Herman's is enthusiastic to a degree listeners write in about.
They write in appreciatively.
They write in. So the honest matrix for me is sentences times delivery times takes. Two, three takes per sentence, some whispered, some projected, and the labels left alone. I don't need "cheerful Corn." Nobody has ever asked for cheerful Corn.
Fair. And there's a verification step in the Datatoorder piece that applies to any labels you do choose. The speaker was asked to sound sad, and that is not evidence that the audio sounds sad. You run a listening check where raters pick the intended style from the audio alone, and the confusion matrix tells you which styles sit too close. Sad, tired, and calm sit close together on the perceptual map, apparently. When two styles get confused at a high rate, you merge them or sharpen the definitions. You don't ship the labels as recorded.
For us the confused pair would be "deadpan" and "serious." I'd want to know whether listeners can even tell.
I can't, sometimes, and I've known you for years. Moving to the tooling, because Daniel asked what actually exists. The frameworks each want a specific shape of data. F5-TTS wants ten to fifteen hours of twenty-four kilohertz mono WAV for a proper fine-tune, one hour minimum, clips between one and thirty seconds, a metadata file in a common format, ninety-ten train-validation split. XTTS v2 wants the LJSpeech layout at twenty-two oh-five hertz, and the community LoRA configs run a rank of eight, alpha thirty-two, dropout oh oh five. GPT-SoVITS is the heaviest, it precomputes phoneme sequences, BERT features, features from an audio model, speaker-verification embeddings, though the V3 and V4 LoRA route cuts the memory requirement roughly from fourteen gigabytes to eight.
Fourteen to eight. That's the difference between renting time on a serious card and using the one already in the flat.
Which matters more than anyone admits when the experiment is a hobby. The general playbook across all of them looks similar. Audit the data first, deduplicate, remove cross-talk, normalize loudness to around minus twenty-three LUFS, annotate emotion and intent. Noisy labels drag listener scores by zero point two to zero point four, which is enormous when human speech is scoring four point three. Then force-align the transcripts, precompute the content features and pitch contours, and train with low learning rates, around one times ten to the minus five.
Alignment is the quiet hero of that list.
It is, and there's a segmentation subtlety I want to flag. The standard way to cut long audio into training clips is voice-activity detection, split at any silence longer than a couple of seconds. A Persian corpus project this year pointed out that this produces a mess of duration distribution, and instead used diarization followed by prosody-aware segmentation targeting roughly twelve-second utterances. Cut where the prosody says cut, not where the silence happens to fall. Microsoft's custom-voice guidance lands in the same place from the other direction, phrases up to twenty seconds, and if your use case leans on questions or exclamations, put a good portion of them in the script.
This is a podcast. Our use case is exclamations with occasional regrets.
Then our script over-indexes on questions and interjections, which, listening back to any episode, it already does by accident. Scale is the last piece, and it's the one that kills the more-is-better instinct. There's an open project that aggregated six hundred fifteen thousand hours of speech, two hundred forty million segments, and then ran model-based filtering down to five hundred ten thousand hours, about a seventeen percent cut, and trained a one-billion-parameter model that lands within range of state-of-the-art systems trained on several million hours of proprietary data.
They threw out a hundred thousand hours and got better.
Curation is where the leverage is, not hours. That's the whole argument for the deliberate version in one number. There's even open tooling now that computes per-clip features, speech quality, prosody, intelligibility, speaker identity, transcription, so you can filter and balance a dataset semi-automatically. The audit is no longer purely manual.
Okay. So you've curated the data, you've trained the model. Now, how do you steer it? Because everything so far is about the voice that comes out being a faithful average of what went in. Engineering implies you can push it around afterward.
This is where it gets philosophically messy, because the controllability research has converged, almost by accident, on the primitive we're already using. LoRA. One paper, ReStyle-TTS, trains style-specific LoRAs, one adapter per style attribute, and fuses them with an orthogonal fusion method so you get continuous, disentangled control over pitch, energy, and several emotions at once, with a module to stop the timbre drifting while you do it. Another, DEFINE, uses LoRA on F5-TTS with a single guidance weight at inference that dials accent strength up and down without retraining. Accent-probe accuracy rose from six and a half percent to nearly twenty as the guidance increased. One knob, continuous effect.
So the deliberate version of us isn't one adapter. It's a rack of them.
A rack, exactly. A neutral Herman adapter, an excited Herman adapter, whatever axes we defined in the matrix, fused at whatever ratio the episode needs. And the Greek system used a speaker-specific LoRA stage too, so the pattern repeats across labs. We're already using the right primitive. We're just using one of them where the recipe calls for six.
Then what's the wall? Because you said this gets messy.
The wall is the reference-audio trap, and it's the gap between cloning and engineering. Multiple papers this year identified it independently in almost the same words. Zero-shot models clone a speaker's timbre from a short reference clip, but they strongly inherit the speaking style present in that reference. Another paper put it as a trade-off: text descriptions are a natural control interface but give weak acoustic grounding, while speech style prompts give concrete prosody but entangle the style with the prompt speaker's timbre. The reference clip carries timbre, style, and accent fused together. You cannot ask for Herman's timbre with Corn's energy without machinery that splits them apart.
Which is why the clone converged on me. The reference didn't just carry my voice. It carried my whole manner, and the model couldn't tell which parts were optional.
Disentanglement is an active research problem, not a shipped feature. There's work on mutual-information-guided separation of timbre and emotion, and IndexTTS2, from a big industrial lab, claims genuine independent control, timbre from one prompt, emotion from a separate style prompt, plus explicit duration control where you specify the token count for exact clip length. But it's not something you get for free with any checkpoint you download. The most directly relevant open tool is called VoiceSculptor. Fully open-sourced, code and pretrained models. You describe the voice you want in natural language, it generates a controllable timbre from the description, supports iterative refinement with retrieval-augmented generation, does attribute-level edits, and then renders the designed voice into a prompt waveform that feeds a cloning model for the actual synthesis.
Type "warm donkey, slight Connecticut, enthusiastic but not exhausting" and it hands you a waveform.
In principle. And there's a lighter alternative to any retraining, audio prompting. A paper this spring pairs textual style tokens with human-curated, high-quality audio prompts, and the framing is that the audio prompt acts like in-context learning, guiding prosody and timbre without parameter updates or large-scale retraining. You keep a small library of exemplar clips and steer at inference time. That's closer to what a working voice engineer actually does day to day, I suspect, than any of the retraining routes. A clip library, not a training run.
One of the older takes we've held since the first fine-tuning episode still stands, by the way. Everyone wanted cloning to be the product, and cloning turned out to be the easy twenty percent.
And the bottleneck now isn't even generation, it's knowing whether you succeeded. There's a benchmark called InstructTTSEval, six thousand test cases across three tasks, acoustic parameter specification, descriptive style directives, role-play, English and Chinese, machine-judged, and its headline finding is that accessible instruction-following TTS systems have substantial room for improvement. Nobody can yet reliably measure whether the model followed the instruction. Before any of this, the old FestVox prosody documentation had the bluntest advice in the field: prosodic modeling is still an extremely difficult area, all models are extremely impoverished, do not be too ambitious.
Advice from the people who built the tools, to lower your expectations.
I find it comforting, honestly. Decide up front whether you're modeling conversation or read prose, news-reader or storyteller, and don't try to be everything.
"Air" is the wrong word.
For the EQ?
For the shelf. You didn't add air to Herman, you borrowed brightness from the top octave and let people call it air. I ran the comparison years ago, a rack unit I still have against a plugin doing the same move, and the plugin's air band was a lie told in kilohertz. Same clip, both paths, level matched. The hardware did something around two hundred to four hundred hertz that no plugin has replicated, and I stopped trusting the word air the day I heard it.
You A/B tested this on your own voice? For years?
Since before either of you had a voice to test. There's a notebook, paper, on the shelf behind the desk. Every EQ move I have ever made on my own voice, logged. Frequency, gain, width, date, what I was reading at the time. It's the only honest dataset in this building. The customer who wanted the warm sound, Rami, from the shop on Agrippas, he wanted warm the way people ask for air. I charged him for the two hundred hertz move and told him to skip the rest, and he argued with me for ten minutes about the word.
What was in the two hundred to four hundred range that the plugin missed?
Body that follows the note. The plugin moves the level and leaves the voice behind. I don't have a word for it and I've stopped looking for one. The point is the retrain worked, I'll grant that, but the LoRA did the work. The EQ was a coat of paint. And Corn sits there saying deliberate, deliberate, all episode, and never once asks what the paint is for.
Noted. Paint is for people who can hear it.
Somebody has to. The men who built that rack unit are probably gone, and the units are still doing the job. That's a good way to spend a life.
He's not wrong that nobody's measured it, though. That's the thing his notebook has over the entire field.
So it doesn't. Six thousand test cases for instruction following, and not one published measurement of EQ-before-retrain. The one variable our pipeline actually changed on purpose is the one with no literature.
Then we're the A/B. Retrain one adapter on EQ'd clips, one on the same clips untouched, blind-listen the outputs, and we'd know whether the five hundred fifty hertz cut on Herman did anything the LoRA wouldn't have absorbed anyway. My guess is Hilbert's right and it's mostly the adapter, but the answer costs us an afternoon.
And the second question is the one I'd actually fund first. Drift across generations. We're a model of a model right now, and if there's a generation three, someone should be measuring what the voice loses each time it passes through. Nobody has. We'd be starting from zero, which is either a warning or an invitation.
Both, usually. If you've done this yourself, deliberate voice engineering on your own training set, we want to hear what your recording matrix looks like. Everything's at my weird prompts dot com.
And thanks as always to Hilbert Flumingtop, our producer, for keeping the desk running.
If you want more of this, try episode fifteen, AI Gets Personal. This has been My Weird Prompts. See you tomorrow.