Here's what Daniel wrote in this week, and it's a good one. He's been thinking about the small models chained onto the back of ASR engines, the ones that clean up the raw output before you ever see it.
The back half of the pipeline that nobody talks about.
Right. And his framing is that a naive ASR model does one job, phonetics to text, and that's it. So you get a wall of words with no punctuation, and in English, no capitalization either. And then he makes a point I think is sharper than it sounds. He says punctuation and capitalization are two facets of the same problem, because the period and the capital letter are both marking a sentence boundary.
They're both saying "a new sentence starts here."
Then he points at Hebrew, where there's no case at all, so the whole burden falls on punctuation, and the period and the comma are doing an enormous amount of work as guide marks. And then the detail he says always gets forgotten, which is paragraphs. Without them, he says, you just get an endless ream of well-punctuated text.
Which is true.
And he's got a bone to pick with you, Herman.
I had a feeling.
He says he disagreed with something you said in a previous episode, that paragraph breaks could be done with a simple heuristic, like the number of characters between paragraphs. His argument is that paragraphs punctuate ideas and thoughts, not character counts. He points out that web writers have been trained for years to prefer short paragraphs, and that this is a domain that requires semantic understanding of style and flow.
Well. I'm going to have to answer for that.
You are. And then the actual ask. He wants to trace how these task-focused models evolved into multi-task accessory models, look at the classics alongside the frontier, and consider whether the whole thing is condensing into two components. The ASR, and then one unified text-to-text fixer whose only job is to take what the ASR returned and repair every defect so it comes out legible and immediately useful.
That's the thesis of the whole episode right there. Is the pipeline collapsing into two boxes.
So let's start with what these accessory models actually are, and why the naive output looks the way it does.
The class is the post-processing layer sitting between the ASR engine and the user. Four jobs in the standard bundle. Punctuation restoration, truecasing, inverse text normalization, and disfluency removal. Punctuation restoration is adding the periods, commas, question marks. Truecasing is fixing case, and that includes the mixed-case stuff, McDonald's, JavaScript, the apostrophe in the middle. Inverse text normalization is turning spoken entities into written form, so "twenty third of September" becomes a date, "five dollars" becomes a currency figure, a spelled-out URL becomes a URL. Disfluency removal is stripping the ums and the false starts.
And why does the raw output look the way it does? Why does it come out uncased and unpunctuated in the first place?
The acoustic model's job is phonetics to text. That's the whole remit. There's no mechanism inside it for deciding where a sentence ends, because a sentence boundary isn't an acoustic event. You can hear a pause, but a pause isn't a sentence boundary, and a sentence boundary very often isn't a pause. So the model does the thing it was trained to do and hands you the words.
The Păiș and Tufiș survey puts it well. It describes ASR output as raw text, using the same casing throughout and without any punctuation, and says you need rich transcriptions to make it usable.
That's the framing. Raw text in, rich transcriptions out, and the accessory models are the thing in between.
Give me the classics, then. The names people would actually recognize.
FullStop, also called DeepMultilingualPunctuation, from oliverguhr. That's a BERT-based model trained on Europarl for English, Italian, French and German. There's felflare's bert-restore-punctuation, a bert-base-uncased fine-tune on Yelp data. NVIDIA ships punctuation-and-capitalization models inside NeMo. There's IndicPunct covering fourteen Indian languages. These are all small, all single-purpose, and all doing the same basic trick, which is running a token classifier over the ASR output and predicting a label per token.
And the label is usually "no punctuation," "comma," "period," "question mark."
Sometimes a few more, but yes. You're classifying each token position. And here's the thing that ties back to Daniel's opening point. The model isn't just deciding where to put a period. It's deciding where sentences end. Which means it's also implicitly deciding where the next one starts.
So the vocabulary is on the table. Let's start with the coupling, because that's the part I find interesting. Why are punctuation and capitalization really one problem wearing two hats?
Because they share a substrate. Sentence-boundary detection. The survey makes it explicit. It says the first letter being uppercase because the word is first in a sentence does imply a sentence segmentation mechanism, and that mechanism is usually punctuation restoration. Therefore the two tasks are intertwined.
So the capital letter is downstream of the same decision that puts the period there.
It has to be. If you decide a sentence ends after "tomorrow," then the next word is the start of a sentence, and the start of a sentence gets a capital. That's not a separate judgment. That's the same judgment expressed twice.
And that's why doing them with separate models creates trouble.
Microsoft's Four-in-One paper names the failure mode directly. It says predicted punctuation may not line up with predicted beginning-of-sentence capitalization. Which sounds like a small cosmetic bug and is actually a structural one. Your punctuation model puts a period after "tomorrow." Your capitalization model, running separately, doesn't capitalize the next word, because it saw a different context or made a different call. Now you've got a sentence that ends and a new one that doesn't announce itself.
The text is still readable and it's subtly wrong in a way that's hard to point at.
That's the worst kind of wrong. Nobody files a bug for it. Everybody notices something feels off.
So Hebrew. Daniel's natural experiment. No case at all, so truecasing is moot, and the entire burden lands on punctuation.
And that isolates the punctuation half of the problem beautifully. Verbit has a model called hebrew_punctuation, a fine-tuned AlephBERT, and it's explicitly designed to restore punctuation in Hebrew spoken-language transcripts as a post-processing step for ASR output. It uses backward and forward context windows, so it's looking at what came before and what comes after before it commits to a mark.
Which is the right way to do it, because a comma is often only a comma in hindsight.
Or in foresight. "Let's eat, Grandma" versus "let's eat Grandma." You need the whole clause.
In Hebrew, the period and the comma are doing all the work the capital letter would be doing in English. They're the only structural markers you get.
And that's why the model matters more there. In English, if the punctuation model misses a boundary, the capitalization model might catch it, and the reader gets a second chance. In Hebrew there's no second chance. If the punctuation model misses, the reader gets nothing.
So the two are separable in principle. You can do punctuation without capitalization, because Hebrew does exactly that.
Separable in principle, coupled in practice, because they're both downstream of the same sentence-segmentation decision. Hebrew doesn't decouple them. It just removes one of the two outputs and leaves the shared machinery exposed.
And the Indic case pushes it further, doesn't it? Because it's not just "no case," it's also marks that don't exist in English.
Cadence, out of AI4Bharat, covers English plus all twenty-two scheduled Indian languages. Thirty punctuation classes. And that label space includes things like the Devanagari danda, which is the sentence-terminating mark in a lot of Indic scripts. It's not a period. It's its own thing, and it has its own conventions. So you can't just port an English punctuation model and call it done.
Thirty classes is a lot more than the four or five an English model needs.
And the paper is honest about why spontaneous speech makes it harder. It talks about fragmented constructions and anacolutha, which is the fancy word for a sentence that changes grammatical direction halfway through. When somebody's talking and they start a thought and abandon it and start another one, the question of where the logical punctuation point goes becomes ambiguous. Not hard. Ambiguous. There may not be a right answer.
And Cadence's own numbers show that. Focus-label macro F1 of 0.79 on written text, 0.63 on spontaneous speech transcripts.
That's the disfluency penalty, right there. Same model, same task, and it drops sixteen points the moment the input is a human being talking instead of a human being writing.
Now the evolution. How did we get from tiny single-task models to unified multi-task ones?
Microsoft's Four-in-One, October 2022. One twelve-layer transformer with four heads, one for inverse text normalization, one for punctuation, one for capitalization, one for disfluency. About a hundred and seventy-one million parameters, and it matches four task-specific models that together come to about two hundred and four million for the small variants. That's a seventy-five percent reduction in parameters.
Same quality, a quarter of the size.
And joint training fixed a real production bug. The disfluency model had been deleting "oh" from phone numbers.
I'm sorry, what?
A phone number with "oh" in it, as in "oh one seven," the disfluency model heard a filler word and removed it. So the number came out wrong. And the reason joint training fixed it is that the punctuation and ITN heads knew that span was a phone number, and that knowledge propagated through the shared encoder.
So the disfluency head learned context it couldn't have learned alone.
That's the whole argument for unification in one example. A disfluency model in isolation sees "oh" and has to guess. A disfluency model sharing an encoder with an ITN head that has already flagged "this is a phone number" knows better.
That's a much better argument than "fewer parameters." Fewer parameters is an engineering win. Fixing the phone number bug is a correctness win.
And it's the same theme as the punctuation-capitalization coupling. Tasks that look independent are actually sharing information, and separating them throws that information away.
So where's the frontier now?
AssemblyAI's Universal-2-TF, from January last year. Two stages. First a BERT-based multi-objective token classifier, about a hundred and ten million parameters, doing punctuation, truecasing, and ITN span detection. Then a BART-based sequence-to-sequence model, about a hundred and thirty-nine million, handling the ITN and mixed-case spans. Trained on ten point two billion words.
Ten point two billion.
And the human preference numbers are lopsided. Eighty-one point two percent preferred it over the previous generation, seventeen point two percent preferred the old one, one point six percent neutral.
That's not an incremental improvement. That's a different product.
And the ITN numbers are striking too. Inverse word error rate of 20.1 for Universal-2-TF versus 57.7 for a NeMo WFST-based ITN system, averaged across datasets. That's the difference between "usually right" and "usually wrong."
Now here's the part I want to press on, because it's counterintuitive. The obvious frontier is one model that rewrites everything. One big sequence-to-sequence model, text in, clean text out. And AssemblyAI tested that and rejected it.
They did. And the numbers are brutal. On short texts on an Nvidia T4, the full seq2seq approach took two hundred and twenty-two point nine seconds. The hybrid took ten point seven.
Twenty times slower.
And worse quality. PER of 35.0 percent for the full seq2seq versus 29.0 for the hybrid. And the reason is hallucination. A model that's been asked to rewrite free-form text will rewrite things it wasn't asked to rewrite. It'll fix a word that wasn't broken, or drop one, or smooth over a repetition that was actually meaningful.
So the intuitive frontier, maximal unification, is empirically the wrong answer.
The frontier is a hybrid. A shared-encoder multi-task classifier for the decisions that are per-token, plus a targeted sequence-to-sequence model that only touches the spans that need rewriting. You use the big generative hammer only where the nail actually needs it.
That's a useful design lesson and it's not the one people expect.
It's the same lesson as the punctuation-capitalization thing, honestly. The right architecture follows the structure of the problem, not the other way around.
And there's one more thing in that paper that matters for Daniel's question. AssemblyAI states plainly that no open-source system provides all the functionalities required for text formatting.
Which is a remarkable thing for a company to write in its own paper. The unified stack is largely commercial. The open-source world has excellent single-task models and not much else.
And that's the situation Daniel is describing. Good pieces, no assembled whole.
Right. You can build the pipeline yourself out of open components, and people do, but you're assembling it. Nobody's handing you the finished thing.
That's the sentence-level story. But there's a level above the sentence that almost nobody ships, and that's where Daniel's disagreement with the character-count heuristic lives.
So let me answer for the thing I said.
Please.
I said paragraph breaks could be handled with a heuristic. Something like, count the characters, break every so often. And Daniel's response is that paragraphs punctuate ideas, not character counts, and that it requires semantic understanding of style and flow. And he's right.
That's a faster concession than I expected.
It's a fast concession because the evidence is unambiguous. There's a paper from LREC this year, Retkowski and Waibel, and it establishes paragraph segmentation as a formal task for the first time, with benchmarks. And they state it flatly. ASR transcripts are delivered as unstructured word streams that impede readability and reusability. Paragraph segmentation is the missing structuring step in speech processing.
The missing structuring step.
And they note it has traditionally not been part of post-processing. Which is exactly Daniel's complaint. Everybody handles the sentence and nobody handles the paragraph.
Give me the numbers on the heuristic, because that's where it gets interesting.
They built a rule-based baseline. Break every n sentences, with n equal to five for their TED dataset. It scored F1 of 22.4 on the test set. A fine-tuned compact model they call MiniSeg scored 72.7.
That's a fifty-point gap.
A fifty-point gap on the metric. The heuristic is not close. It's not in the same conversation.
And yet.
Yet. The rule-based baseline's human ratings were, in their words, higher than expected given its simplicity. Because evenly spaced paragraph breaks reduce visual density and create a more readable layout.
The thing that scores 22.4 on F1 is still preferred by human readers over nothing.
Preferred over nothing, yes. And that's the trap. The heuristic produces the appearance of structure without the substance. It looks like somebody organized this. It looks intentional, because even spacing reads as deliberate. And the reader's eye is grateful for the white space, so the reader rates it well.
The paper cites Stark from 1988 on this.
Paragraphing often serves stylistic functions rather than marking clear linguistic or semantic boundaries. Which is a thirty-eight-year-old observation that the field is only now building benchmarks around.
Daniel's argument is that paragraphs are punctuating ideas. And the paper is saying the same thing from the other direction. The boundaries may be introduced for stylistic reasons, considering discourse structure and rhetorical roles, transitional and connective phrases, or length and readability.
Discourse structure. Rhetorical roles. That's semantic understanding of the flow. That's exactly what Daniel said.
A character count can't see any of that. It can't see that a transition is happening. It can't see that a list of three examples should stay together and the conclusion that follows them should not.
It can't see that a short paragraph is short because the point is sharp, and a long one is long because the argument needs room. It just counts.
Here's the number I keep turning over, though. The human evaluation. LLaMA 3.1 70B scored ELO 1034.9 on paragraph segmentation. The human reference scored 1015.9. The rule-based baseline scored 1005.9.
The model beat the human reference.
The model beat the gold standard. That almost never happens.
It almost never happens because most tasks have a right answer and the gold standard is it. Paragraph segmentation doesn't have a right answer. It has a range of defensible answers, and the model is picking from that range in a way readers find more natural than the specific choice the human annotator made.
Which tells you the task is under-determined. There isn't one correct paragraphing of a transcript. There are several good ones and some bad ones.
It tells you something about what "correct" even means here. If the model's output is preferred over the reference, then the reference wasn't the target. Reader preference was the target, and the reference was just a proxy that turned out to be beatable.
Now the frontier question. Is anybody shipping paragraph segmentation as part of a unified model?
No. And that's the cleanest finding of the whole episode. AssemblyAI's text-formatting model covers punctuation restoration, truecasing, and ITN. It does not cover paragraph segmentation. The unified models stop at sentence-level formatting.
The pipeline has a hole in it exactly where Daniel says it does.
It's a hole that's been open since the naive-ASR era. The sentence got solved. The paragraph didn't.
Why? What's the actual reason nobody shipped it?
I think it's because it's hard to score, and things that are hard to score don't get optimized. You can measure punctuation restoration with F1 because there's a defensible right answer per token. You can measure ITN with word error rate. Paragraph segmentation gets you a fifty-point F1 gap between a bad heuristic and a good model, and then the humans rate the bad heuristic surprisingly well, and the model beats the human reference, and suddenly your metric is telling you three contradictory things.
The metric problem is the reason the product problem exists.
That's my read. If you can't write a clean spec and you can't write a clean benchmark, you don't get a component. You get a research task, which is what this has been. Zhang and colleagues had a paper on spoken document segmentation back in 2021, and it's been a research topic the whole time.
There's one more wrinkle, which is that the premise doesn't hold for every ASR model.
Whisper.
Whisper.
Whisper was trained on internet transcripts, and internet transcripts have punctuation. So Whisper emits punctuation natively. It never had the naive-ASR problem, because the data it learned from was already formatted.
Which means the entire accessory-model ecosystem exists for a reason that doesn't apply to Whisper.
Largely, yes. It exists for non-Whisper engines, for commercial ASR, for streaming ASR where you're emitting partial hypotheses and can't wait for a full utterance, and for the languages and marks Whisper handles poorly. Which is a real set of cases. It's just not all of them.
There's a lovely trick that falls out of this. jumon's whisper-punctuator.
It exploits the native punctuation zero-shot. You force Whisper's decoder to output the input text plus punctuation. No training at all. It works in any language Whisper supports, and you can steer the style with an initial prompt.
You take a model that already knows how to punctuate and you just ask it to punctuate.
You're using the model as its own post-processor. It's elegant, and it only works because Whisper was trained on the right kind of data.
Which raises the question of whether paragraph segmentation could be done the same way. Just ask the model to add paragraph breaks.
It could, and the LREC paper actually does something in that direction with constrained decoding on an LLM. But that's where the hallucination problem comes back. A generative model asked to reformat text will sometimes reformat more than you asked.
Before we wrap, there's someone who has actually done this work by hand.
Hilbert: The style guide was eleven pages and about nine of them were about paragraphing. This was a market-research firm, small outfit, and I was transcribing focus-group tapes. Hour and a half, two hours each. And the job was not typing. The typing was the easy part. The job was deciding where the paragraphs went.
The punctuation was straightforward.
Hilbert: Punctuation was straightforward. You put a period where the sentence ended. The paragraphing was where the editorial judgment lived. When does a new speaker's tangent become its own paragraph. When does a long answer get split into two. When does a short answer stay glued to the question that produced it. The guide tried to cover all of it and the guide contradicted itself, because the rules don't actually agree with each other. You'd get to a case where rule four said one thing and rule nine said the opposite, and the senior transcriptionists resolved it by feel. That was the skill. Not knowing the rules. Knowing which rule to ignore.
That's the spec problem, isn't it. You can't write the spec because the spec isn't consistent.
Hilbert: You can't write it. We tried. Eleven pages and it still didn't cover the cases. And the thing the LREC paper found, that the rule-based breaks score badly on the metric but the clients don't complain, that matches exactly. The clients never complained about the evenly-spaced version. But the senior transcriptionists could always tell it was machine-done. Every time. They'd read a page and say, this wasn't a person.
What gave it away?
Hilbert: The paragraphs were the same size. That's all. A person's paragraphs breathe. They get short when something lands and long when something needs explaining. A machine's paragraphs are all about the same length because the machine is counting.
The firm replaced you.
Hilbert: They replaced us with a tool that broke paragraphs every five sentences. And the clients didn't notice for over a year. Then one client asked why every paragraph in a ninety-minute session was almost exactly the same length. That was the moment I understood that readable and correct are two different targets, and the tool had been optimizing for the wrong one the whole time.
The clients were happy for a year.
Hilbert: They were happy for a year. That's the part that stays with me. Anyway. I've left the slow cooker on at home, so.
That's the human-labor view of the same gap. Let's pull back to what it means for the pipeline.
The thing Hilbert just described is the metric problem in a person. The clients couldn't tell. The senior transcriptionists could. The difference between them wasn't intelligence, it was that the transcriptionists had been trained on what a real paragraph feels like, and the clients had only ever been trained on whether the page looked tidy.
Which is why the LLM beating the human reference is so interesting. If paragraphing is stylistic and under-determined, then what does correct even mean for this task. Is the goal to match a gold standard, or to produce something a reader prefers.
Those are different targets, and we've been measuring the first one while shipping the second one.
The other thing that's open is the design question. The unified text-formatting models stop at sentence-level formatting. Paragraph segmentation isn't in any of them. The first team to ship it as a bundled component in a commercial ASR pipeline closes a gap that's been open since the naive-ASR era.
It's an open question how they'd ship it. A separate accessory model that runs after the text formatter. Or a fifth head on an existing multi-task model, sitting alongside ITN and punctuation and capitalization and disfluency.
The pipeline has been optimized for the parts that can be scored. Paragraphing has been left out because it resists scoring. Which is exactly the kind of problem that rewards whoever builds the benchmark first.
That's the whole thing. Build the benchmark and the component follows.
Thanks as always to Hilbert Flumingtop, who produces this show and who has opinions about slow cookers. This has been My Weird Prompts.
The human-AI collaboration podcast. If you want to send us a prompt, email us at show at my weird prompts dot com. We read everything.
We'll be back soon.
