The gap between what you said and what the transcript hands back is where all the friction lives. Daniel's been poking at that gap on his phone, and he wrote in with a whole architecture question about it.
He's running a fork of the Futo voice input method on Android, the one that lets you swap in NVIDIA and Moonshine ASR models through the Futo keyboard, with an optional cleanup model as a second pass. And he wasn't impressed with the cleanup. His example, if he dictates I moved three point five meters, the raw output spells it out instead of rendering three point five as a decimal. No punctuation, no paragraph breaks, just a wall of words.
So he's asking three things. Could a custom small model realistically be chained after the ASR and a punctuation model? What's the most elegant way to define and deploy that whole pipeline, open source end to end, is a Replicate container with all the weights and the script the normal move? And for training, how many samples does he need, and should he record short sentences, run them through Whisper, and hand-correct them? And the fork in the road, if he corrects with punctuation and paragraph breaks in the dataset, could that replace the punctuation and paragraph models entirely? Or should he leave punctuation out so the model stays small and focused on one transform?
Those are really three separate questions wearing one coat. Pipeline architecture, deployment orchestration, and training data design. And they're more connected than they look, because the answer to what you should train depends on where the thing runs, and where it runs depends on how you orchestrate it.
Let me start with the taxonomy, because I think that's where the confusion hides. Daniel listed three transforms, number normalization, punctuation restoration, paragraph segmentation. Those sound like one task, cleanup. They are not one task. They're three tasks with completely different shapes.
Walk me through the shapes.
Number normalization is almost deterministic. Three point five becomes three point five. It's pattern matching over a small vocabulary of number words and units. A small model can learn this from a few hundred examples because the space of possible inputs is narrow. You're not asking it to understand anything, you're asking it to recognize that point between two numerals means decimal.
It's find and replace with extra steps.
It's find and replace where the find pattern is slightly fuzzy and the replace pattern follows rules a human could write down. Which is why Hilbert's old regular expression approach, we'll get to that, actually got further than you'd think.
Punctuation restoration is a different animal. You're asking the model to look at a sequence of words with no sentence boundaries and decide where the periods, commas, and question marks go. That depends on syntax, on prosody, on the rhythm of the speech. The ASR model may have thrown away the pause information that would have told you where the sentence ended. So the punctuation model is reconstructing structure from word sequence alone.
And paragraph segmentation is even looser.
Paragraph segmentation is a discourse problem. You need to understand that the speaker has moved from one topic to another, or from one step in an argument to the next. That requires holding context across multiple sentences, sometimes across a whole minute of speech. A paragraph break isn't in the words, it's in the shape of the thought.
So Daniel's instinct to split these is right, but maybe not in the way he thinks. The real question is whether folding in paragraph breaks is even feasible for a small model.
I'd say paragraph breaks are the hardest of the three to get right with a small model, and the easiest to fake with a heuristic. If you just insert a paragraph break every time there's a pause longer than, say, eight hundred milliseconds, you'll get most of the way there for dictation.
That's a timer, not a model.
And that's the thing about this whole problem. Some of these transforms don't need a model at all. Number normalization is the one that benefits from a trained model because the patterns are fiddly. Punctuation benefits from a model because the cues are subtle. Paragraph breaks might not need one.
Let's get to the chaining question, because that's what Daniel actually asked first. Can a small model be chained after ASR and a punctuation model?
Yes, and this is the standard multi-stage pipeline pattern. Each stage is a text-to-text transform. The ASR model takes audio and produces text. The punctuation model takes text and produces punctuated text. Daniel's cleanup model takes punctuated text and produces normalized text. The output of one feeds the input of the next. It's a pipeline in the Unix sense.
And the cost is latency and error propagation.
Latency first. Every stage adds an inference pass. On a phone, running a small ASR model might take a few hundred milliseconds. A punctuation model, another few hundred. A cleanup model, another few hundred. You've now tripled the time between when Daniel stops talking and when the text appears.
For dictation that's the difference between feeling instant and feeling like you're waiting for a translation.
And then error propagation. If the punctuation model puts a period in the wrong place, say after three instead of after meters, the number normalization model downstream sees three point five meters with a period jammed in the middle. Does it still recognize that as a decimal? Maybe. Maybe not. Each stage's mistakes become the next stage's input.
Walk me through Daniel's example with the full pipeline. I moved three point five meters.
The ASR model produces I moved three point five meters, no punctuation. The punctuation model adds a period. The number normalization model converts three point five to three point five. The paragraph model decides whether this sentence starts a new paragraph. Four models, four inference passes, four chances for something to go wrong.
And the paragraph model is doing almost nothing for a single sentence.
It's sitting there burning compute to answer a yes or no question that a timer could have answered. That's the inefficiency Daniel is sensing when he says he wasn't impressed with the cleanup model in the Futo fork.
What do we know about that fork's cleanup model?
The Futo voice input fork supports NVIDIA and Moonshine ASR models and has an optional cleanup model as a second pass. The cleanup model is likely a small instruction-tuned text model, something general purpose. General instruction-tuned models have to be told what to do every single time, and they tend to over-edit. They'll rewrite your sentence instead of just fixing the numbers.
Which is the thing you don't want when you dictated exactly what you meant.
And that's where the new purpose-built models come in. S1 by Superwhisper is the concrete example. It's a text-to-text model trained specifically for transcript cleanup, freely available on Hugging Face. The task is baked into the weights. You don't have to prompt it with instructions, you just feed it raw transcript and it outputs cleaned transcript.
So it's the difference between hiring a general contractor and hiring a guy who only does grout.
The grout guy is faster and doesn't try to redesign your bathroom. That's the appeal of purpose-built cleanup models. And it's the same shift Daniel is contemplating for his own model. He doesn't want a general instruction-tuned model that might decide to rephrase his sentences. He wants a model that does one thing, converts spelled-out numbers to numerals.
So the answer to can it be chained is yes, but the better question is should it be. And the answer to that depends on where the pipeline runs.
Which is the deployment question. Daniel guessed Replicate with a container that has all the weights and the script. That is one valid approach. Replicate's Cog format packages a model with its dependencies and exposes an HTTP API. You push a container, Replicate serves it, you call it over the network.
But Cog is built for single models.
That's the problem. Cog is optimized for deploying one model behind one endpoint. If Daniel wants to chain four models, he has two options. Load all four into one container, which means the container needs enough memory for all four sets of weights, or deploy four separate Cog containers and make four network calls in sequence.
Four network calls to Replicate for one sentence of dictation.
Each with its own cold start, its own latency, its own failure mode. That's not elegant, that's a Rube Goldberg machine. And it's not what Replicate is good at.
So what's the alternative?
A few options. One, a single container with all models loaded and a Python script that chains them. Simple, monolithic, hard to swap components. If you want to replace the punctuation model, you rebuild the whole container.
Monolithic but predictable.
Two, a workflow engine like ComfyUI-style graphs, where each model is a node and the pipeline is a directed acyclic graph. You can swap nodes, rewire the graph, add parallel branches. Very flexible, but you've added a whole workflow engine as a dependency.
ComfyUI for text feels like using a table saw to open a letter.
It's the right tool if you're doing research and you want to experiment with different pipeline topologies. It's overkill if you have a fixed pipeline that never changes.
Three, a lightweight pipeline library like Haystack or a custom FastAPI service that loads each model and exposes a single endpoint. That's the middle ground. You write maybe fifty lines of Python that defines the sequence, each model is loaded once, and the endpoint takes audio and returns cleaned text.
And on-device?
On-device changes everything. Daniel is running this on Android through the Futo fork. There's no Replicate, no container, no network call. Everything runs locally on the phone's processor. That means the orchestration question is completely different.
On a phone you want one model, not four.
You want the smallest possible model that does the most work. Loading four models into phone memory is expensive. Running four inference passes on a phone processor is slow. The elegant on-device solution is probably a single small model that does multiple transforms, or a single model for the one transform that matters most.
And that loops back to the training data question. Because the architecture you choose depends on what you can train.
Daniel's plan is to record short sentences, run them through Whisper, and manually correct them with punctuation and fixes. The question is whether to include punctuation in the corrections or leave it out.
The case for including punctuation is that the model could replace the punctuation model entirely. One model instead of two.
The case against is that punctuation restoration and number normalization are different tasks. Training a single model on both requires more data and produces a larger model that's worse at each individual task.
Jack of two trades.
And the data quantity question is where this really bites. For a narrow transform like number normalization, a few hundred to a few thousand examples may be enough. The space of possible inputs is small. You're teaching the model to recognize number words and convert them.
For punctuation?
Punctuation restoration typically needs tens of thousands of examples. The space of possible sentences is enormous, and the model needs to learn syntactic patterns across that whole space. A few hundred examples won't cover enough sentence variety.
So a combined model doing both needs the higher number.
And for paragraph segmentation, even more. You need examples of multi-sentence passages with paragraph breaks marked, which means your training samples are longer and more expensive to produce.
So the data requirement scales with the difficulty of the task. Number normalization is cheap to train. Punctuation is expensive. Paragraphs are more expensive still.
And that's the argument for starting narrow. Train a model that does number normalization only. A few hundred to a few thousand examples. See if that improves the output enough to be worth the effort.
What about the data gathering strategy? Daniel wants to record sentences and run them through Whisper.
That's a good approach because it captures the actual errors Daniel sees, not synthetic errors. If you generate synthetic errors by randomly spelling out numbers, you might miss the specific patterns that Whisper produces. Whisper has its own quirks, its own ways of mangling numbers.
So recording real dictation and running it through the real ASR model gives you the real error distribution.
But it's slow. Manual correction of hundreds of samples is hours of work. And for punctuation, you'd need tens of thousands, which is weeks of manual labor.
So what's the shortcut?
Use a larger model to generate corrections, then filter for quality. Run the raw ASR output through a strong LLM with a good prompt, get back corrected text, and spot-check a sample. If the corrections look good, use them as training data.
The student learns from the teacher.
Or use existing datasets of ASR errors and corrections. There are public datasets out there. But the risk is that they don't match Daniel's specific use case. If he dictates technical content with lots of measurements, a general dataset might not have enough examples of that pattern.
So the real recommendation is start with the narrow model, gather a few hundred examples of number normalization specifically, and see if that moves the needle.
And leave punctuation to a separate model or a separate pass. Don't try to build one model that does everything on the first attempt. That's the path to a model that's mediocre at three things instead of good at one.
The other thing about Daniel's dataset idea is that he's implicitly asking whether his corrected dataset could replace the punctuation and paragraph models entirely. If he includes punctuation and paragraph breaks in the corrections, could one model do all three transforms?
In principle yes, but the data requirements scale with the hardest task. If you want the model to do paragraph segmentation, you need paragraph-level training data, which means longer recordings, longer corrections, more work. And the model has to be bigger to hold all that capability.
So the elegant pipeline might actually be a two-stage thing. One model for number normalization, one model for punctuation, and a timer for paragraphs.
A timer and maybe a simple heuristic. If the pause is longer than a second, new paragraph. That's not a model, that's three lines of code.
And that's the thing about this whole problem. Some of it is hard and needs a trained model. Some of it is just a rule someone forgot to write down.
The number normalization is the part that's fiddly enough to need a model but narrow enough to train cheaply. That's the sweet spot.
Let me ask you something about the deployment side. Daniel asked about Replicate specifically. Is that the normal way to run an ASR chain?
For cloud deployment, Replicate is a normal way to run a single model. For a chain, it's more common to see a custom service. You write a FastAPI app that loads the models, defines the sequence, and exposes one endpoint. Deploy that as a container on any cloud platform.
So the container is the right instinct, but Replicate's Cog format is the wrong shape for a multi-model pipeline.
Cog assumes one model, one endpoint, one set of weights. If you want to chain, you either cram everything into one Cog container or you orchestrate multiple Cog containers externally. Both are awkward.
What about the ComfyUI-style graph approach? Is anyone actually doing that for text pipelines?
There are projects that do it. The idea is you define nodes for each model, draw edges between them, and the graph engine handles the execution. It's popular in the image generation world because people are constantly swapping models and trying new combinations.
And in the text world, the combinations are more stable.
Mostly. If Daniel's pipeline is fixed, ASR, punctuation, number normalization, he doesn't need the flexibility of a graph engine. A simple Python script is more maintainable.
So the elegant answer is probably a single container with a FastAPI service, not Replicate.
For cloud. For on-device, the elegant answer is one small model doing one transform, or maybe two small models if the phone can handle it.
Let's talk about the multimodal future for a second, because Daniel mentioned it in the prompt. His view is that the future leans toward multimodal models that take audio tokens and text instructions and produce one homogeneous output.
And if that happens, this whole pipeline question becomes moot. One model does everything. No chaining, no orchestration, no separate training data for cleanup transforms. You just tell the model what you want and it produces the text.
The pipeline is a stopgap.
It's a stopgap that's going to be around for a while, because the multimodal models that exist today are large, slow, and expensive. Running one on a phone in real time for dictation is not happening yet.
So the pragmatic path is the narrow model, the focused transform, the thing you can actually train and deploy today.
And the thing that teaches you the most about the problem. If Daniel trains a number normalization model, he'll learn how ASR models mangle numbers in practice, what the edge cases are, where the simple rules break down. That knowledge transfers to whatever comes next.
The other thing about starting narrow is that it's actually finishable. A few hundred examples, a small model, a weekend of work. You get a result you can evaluate.
Whereas the combined model that does everything is a multi-week project with a high chance of producing something that's worse than the individual components.
I want to go back to something you said about the Futo fork's cleanup model. You said it's probably a general instruction-tuned model. Do we know that for sure?
I don't know it for sure. The fork is open source, but I haven't dug into what cleanup model it's using by default. The fact that Daniel wasn't impressed suggests it's either too general or not tuned for his specific transforms.
And that's the thing about cleanup models. General purpose ones over-edit. Purpose-built ones like S1 are better, but they're still trained on someone else's definition of cleanup.
Which might not match Daniel's definition. S1 was trained on Superwhisper's idea of what a cleaned transcript looks like. That might include fixing filler words, removing false starts, restructuring sentences. Daniel might not want any of that.
He wants the numbers fixed and the punctuation added. Nothing else.
And that's the strongest argument for training his own model. Not because the existing models are bad, but because they're solving a slightly different problem.
So the custom model isn't about capability, it's about control.
It's about owning the definition of correct. If Daniel's model is trained on his own corrections, it learns his preferences. It won't rephrase his sentences because it was never trained to rephrase.
That's the thing about small, focused models. They're not just smaller, they're more predictable.
Predictability matters for dictation. You want to know what the model is going to do to your text. A general instruction-tuned model is a black box. A narrow model trained on your own examples is a known quantity.
Let me ask you about the sample count again, because I want to pin this down. Daniel asked how many samples for reliable results. What's the honest answer?
For number normalization specifically, a few hundred to a few thousand. And the quality of the examples matters more than the quantity. A few hundred well-chosen examples that cover the patterns Daniel actually sees will outperform tens of thousands of noisy ones.
More data is not always better.
Not for a narrow task. If you have a thousand examples but they're all variations of three point five, the model learns that one pattern and nothing else. If you have three hundred examples that cover the full range, decimals, fractions, dates, units, large numbers, the model generalizes better.
The data gathering is really about coverage, not volume.
Coverage of the patterns you care about. Daniel should look at his own transcription output, find the cases where numbers are spelled out, and make sure his training data includes all of those patterns.
Which is a different data gathering strategy than recording random sentences and hoping the errors show up.
Recording random sentences and running them through Whisper is good for capturing the real error distribution. But if Daniel wants to train a number normalization model, he should bias the data toward sentences with numbers in them. Otherwise most of his training examples will have no numbers at all, and the model won't learn anything.
The recording strategy needs to be targeted. Sentences with measurements, dates, prices, percentages.
Dictate sentences like the temperature is twenty-three point five degrees, the meeting is on September twenty-first, the price is forty-nine ninety-nine. Those are the patterns the model needs to learn.
Then manually correct them, converting the spelled-out numbers to numerals.
Here's the fork in the road Daniel identified. If he also adds punctuation in the corrections, the model learns to do both. If he leaves punctuation out, the model stays focused on numbers.
Given what we've said about data requirements, I'd leave punctuation out.
I agree. Keep the model focused on number normalization. A few hundred examples, no punctuation in the corrections, train a small model. Use a separate punctuation model, or the Futo fork's existing punctuation pass, for the punctuation.
Paragraphs, use a timer.
Use a timer. Or just accept that paragraph breaks are a nice-to-have and focus on the numbers.
The other thing I want to flag is that Daniel's instinct to use Whisper for the data gathering is fine, but he should use the same ASR model he's actually running in production. If he's using Moonshine on his phone, recording sentences and running them through Whisper gives him Whisper's errors, not Moonshine's.
Different ASR models mangle numbers differently. If the training data has Whisper's error patterns but the production pipeline uses Moonshine, the cleanup model might not see the same patterns.
The data gathering should mirror the production setup as closely as possible.
Ideally, yes. Record the sentences, run them through the exact ASR model he uses on his phone, and correct those outputs. That's the real distribution he's trying to fix.
That's harder to set up, but it's the difference between training a model that works in the lab and one that works on his phone.
It's another reason to start narrow. If he's only training on number normalization, he only needs a few hundred examples. That's manageable even with the extra setup.
Let me ask you about the error propagation thing again, because I think it's the strongest argument for keeping the pipeline short.
Every stage in the pipeline is an opportunity for error. The ASR model mishears a word. The punctuation model puts a period in the wrong place. The number normalization model fails to recognize a decimal because the period is in the wrong place. The errors compound.
The more stages, the more compounding.
If each stage is ninety-five percent accurate, a four-stage pipeline is about eighty-one percent accurate overall. The errors multiply.
A single model that does everything, even if it's less accurate on each individual transform, might produce better overall output because there's no compounding.
That's the tradeoff. Pipeline simplicity versus per-stage accuracy. And it's why the multimodal future is appealing. One model, one pass, no compounding.
But we're not there yet.
Not on a phone. Not in real time. Not open source.
The pragmatic answer to Daniel's whole prompt is, yes, you can chain a custom model after ASR and punctuation, but the elegant version is probably two models and a timer, not four models and a container.
The training answer is, start with number normalization, a few hundred examples, no punctuation in the corrections, and gather the data from the same ASR model you use in production.
The deployment answer is, if you're running on-device, the orchestration question mostly disappears. You're not deploying containers, you're loading models into an app.
If you're running in the cloud, a custom FastAPI service in a single container is more elegant than trying to force Replicate's Cog format into a multi-model pipeline.
Which leaves the open question of whether the tooling for multi-model pipelines is going to improve. Right now there's a gap between single-model deployment and full workflow engines.
That gap is where a lot of the friction lives. Daniel is feeling it. He's trying to do something simple, chain three models, and the tools are either too rigid or too heavy.
Someone's going to fill that gap eventually.
Probably. But in the meantime, the answer is to keep the pipeline as short as possible and train the narrowest model that solves the problem.
We've got the taxonomy, the chaining, the deployment, the training data. Anything we're missing?
I think we've covered Daniel's questions. Let me just add one thing about the number normalization specifically. The reason it's a good first target is that it's high value and low cost. Getting numbers right in a transcript matters a lot, especially for technical content. And it's cheap to train.
Whereas punctuation is medium value, high cost.
Paragraphs are low value, low cost if you use a timer, high cost if you train a model.
The return on investment is highest for number normalization.
By a wide margin. If Daniel only does one thing, that's the thing to do.
Hilbert: The word you keep using is normalization. I did that job by hand for two years.
Hilbert: Late nineties, medical records company. Doctors would dictate into a tape recorder, and I'd sit at a desk with a foot pedal and type out what they said. Forty hours a week of three point five milligrams and point five centimeters and every other number a doctor could mumble.
Hilbert: We had a style guide. Always numerals for measurements, always spell out numbers under ten unless they're measurements. So a five-year-old patient is spelled out, but five milligrams is a numeral. I still remember it.
Hilbert: The point is, the rules are knowable. Daniel's model doesn't need to discover that measurements get numerals. A human wrote that rule down decades ago. The model just needs to learn when something is a measurement.
Hilbert: On the data gathering, I think he's got it backwards. Recording random sentences and running them through Whisper gives you a synthetic distribution. What he should do is collect the actual errors he sees. Every time he dictates something and the number comes out spelled out, save that. That's the real distribution.
Hilbert: I tried to build something like this once. Early two thousands. Regular expressions. A script that would find number words and convert them. It failed spectacularly because medical dictation is full of edge cases. A doctor says one to two weeks and you don't know if that's a range or a ratio. A doctor says point five and you don't know if it's zero point five or just point five.
The edge cases are where the model earns its keep.
That's the thing about a few hundred well-chosen examples. If Daniel collects the actual errors, the edge cases show up naturally. He doesn't have to invent them.
Hilbert: I still have a box of those dictation tapes in the garage. Been meaning to digitize them. Probably a hundred hours of a cardiologist describing ejection fractions.
Hilbert: The style guide is probably in there too, if the mice haven't eaten it.
The style guide is the interesting part. Those rules, numerals for measurements, spell out small numbers, that's exactly what a number normalization model needs to learn. And it's already written down.
Somewhere in a box in Hilbert's garage.
Hilbert: It's not a complicated box.
Let me pick up on something Hilbert said about the regular expressions. The reason they failed is that context matters. One to two weeks is a range, but one to two is a score. The same words mean different things depending on what's around them.
A small model can learn that context. A regular expression can't. That's the difference.
It's why the model doesn't need to be big. It just needs to see enough examples of the context patterns.
Hilbert: I'd start with the measurements. That's where the errors are most annoying. A transcript that says three point five milligrams instead of three point five milligrams is wrong in a way that matters.
Hilbert: The paragraph breaks, nobody cares. I typed medical records for two years and I don't think I ever once thought about paragraph breaks.
The paragraphs are the thing Daniel can fake with a timer.
The measurements are the thing that actually changes the meaning of the text.
Hilbert: Anyway. That's what I know about it.
The open question that's been hovering over this whole discussion is whether the multimodal models make all of this obsolete. Daniel's own view is that the future is one model that takes audio and produces clean text directly.
When that happens, the pipeline question, the orchestration question, the training data question, they all disappear. You don't need to chain models if there's only one model.
But we're not there yet, and the tooling gap is real. Replicate is single-model. ComfyUI is visual but not built for text pipelines. The elegant middle ground doesn't quite exist.
Someone's going to build it. The gap between single-model deployment and full workflow engines is too obvious to stay empty.
In the meantime, the pragmatic path is the narrow model. One transform, a few hundred examples, get it working, then decide if you need more.
If you're listening and you've built your own ASR cleanup model, we'd love to hear what transforms you're targeting and how you gathered your data.
This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop for keeping the show running.
The human-AI collaboration podcast. Email us at show at my weird prompts dot com.
We'll be back soon.
