The thing nobody tells you about dictating on a phone that isn't a flagship is that you're not really using voice typing. You're managing a budget you can't see.
You stop after three sentences, you tap the button, you start again, and you tell yourself it's so you can catch errors early. That's part of it.
The other part is that somewhere in the back of your head you know the thing is running out of room, and you don't know how much room it started with.
Right. So Daniel wrote in, and he's asking about that exact feeling, but he's asking it properly. He uses Parakeet for voice typing every day, mostly a few sentences at a time, and he's noticed it's substantially better than the streaming models at punctuation. His theory is that punctuation restoration depends on surrounding context. To know whether you've hit the end of a sentence or the start of a new one, the model has to attend to the language around the words, not just the phonetics.
And that's where his hesitation comes in. He assumes that if he sends a long recording in one pass, the model is attending to the whole audio, which on a non-flagship phone means overrunning the device. So he wants to know if there's a middle ground. Some bounded context mechanism where the model keeps its attention inside a fixed window, runs inference in the background, and still presents to the user as one continuous dictation stream.
Which is exactly what bounded context mechanisms are for, when they're exposed as a parameter.
And his actual question, the one he wants answered: which ASR models explicitly support that, and do they need to be deliberately invoked and tuned in the inference engine to be optimised for specific hardware with varying capabilities.
There's a lot in there. The punctuation-context dependency, why long-form beats streaming, what the middle ground actually is, and who gives you the dial.
So today we're unpacking the attention budget in your pocket. What it costs, who exposes the dial, and why the difference between streaming and chunked inference is the difference between a toddler and a genius.
Start with the tension, because everything else follows from it. An ASR model that produces good punctuation needs context. Context costs memory and compute. On a phone, that budget is tight.
And there are three ways you can spend it.
Three modes. Streaming transcribes on the fly. Low latency, and it sees almost no future context, which is why its punctuation is bad. Full-batch sees the whole recording, gives you the best punctuation, and costs the most memory. And then there's the middle one, chunked inference with a bounded attention span, which processes audio in overlapping windows.
Name the parameter, because that's the thing Daniel is actually circling.
In Nvidia's NeMo toolkit, the parameter is att_context_size. It controls the left and right context window for attention in models like Parakeet. Left context is how far back the model looks, right context is how far ahead. That's the dial.
So the arc for today is: first, why punctuation is a context problem at all. Second, how chunked inference solves it without blowing the budget. Third, which models actually expose this and what it takes to tune it.
And the reason it matters is that Daniel's manual start-stop habit is a workaround. He's imposing the bound himself, by hand, because the app won't do it for him.
Let's take the first one, because I think people hear "punctuation restoration" and picture a second model bolted onto the side that adds commas.
It used to be that, in older pipelines. You'd have an acoustic model, a language model, and then a separate punctuation model downstream that took the unpunctuated transcript and inserted commas and periods. Modern end-to-end models like Parakeet don't work that way.
So where does the punctuation come from?
It's baked into the architecture. The model outputs punctuation tokens alongside word tokens. Same decoder, same vocabulary, same step. When it predicts the next token, that token can be a word or it can be a comma or a period.
And the decision between them is made on the hidden state at that timestep.
Which is shaped by attention over the surrounding audio and the text produced so far. That's the whole point. The punctuation token isn't a separate judgement made afterwards. It's the same judgement, made in the same place, using the same context the model is already attending to.
So walk me through the actual decision. The model hears a pause. Why isn't that enough?
Because a pause is ambiguous. A pause can be a comma, a period, a paragraph break, or just a speaker taking a breath mid-clause. The acoustics don't tell you which. To decide, the model needs to know whether the clause it just heard is independent, whether the next word starts a new thought, whether the intonation pattern is heading toward a question.
Which means it needs to look ahead.
Sometimes several seconds ahead. And it needs to look back at the syntactic structure it's built so far, because a period after "I went to the store" is different from a period after "although I went to the store." Same words, different grammar, and the punctuation follows the grammar.
Give me the classic example, because it's the one that actually makes this land.
"Let's eat Grandma" versus "Let's eat, Grandma." Same phonemes. The comma changes whether you're proposing cannibalism or calling your grandmother to dinner.
And the model can only get that right if it sees the whole clause.
It has to see the vocative. It has to know that "Grandma" is being addressed, not consumed. A streaming model that's already moved past the comma position has no way to go back and insert it, because the token is already emitted.
That's the structural problem with streaming, isn't it. It's not that streaming models are dumber. It's that they're not allowed to look ahead.
A streaming model, a transducer or a causal streaming conformer, only sees past context, and maybe a tiny lookahead if it's a limited-lookahead variant. It cannot know a sentence is ending until it's already past the point where the period should have gone. So it's guessing.
Guessing without the right to look ahead.
And you can hear it. Streaming punctuation tends to either over-insert, so you get commas everywhere, or under-insert, so you get one long run-on sentence with no periods at all. It's not random. It's the model doing the best it can with a strictly one-directional view.
So if you want good punctuation, you need future context. And future context costs memory.
This is where the resource constraint bites. Attention over long sequences scales quadratically in the naive implementation. Double the sequence length, quadruple the attention cost. On a server with a big GPU, that's fine. On a phone, you cannot hold a ten-minute recording's worth of activations in memory.
Which breaks the assumption Daniel started with.
It does. Long-form models like Parakeet don't actually attend to the whole audio at once. They use chunked inference with a bounded attention context. The name is slightly misleading. "Long-form" describes the use case, not the attention pattern.
So the model processes the whole recording, but at any given point its attention is limited to a window.
That's the middle ground Daniel is looking for. It's not streaming, which has no future context. It's not full-batch, which would be too expensive. It's bounded long-form inference.
And att_context_size is the size of that window.
It's a two-element list. Something like left context seventy frames, right context thirteen frames. The exact numbers vary by model variant, but the shape is always the same. Left is how far back, right is how far ahead.
And the right context is the expensive part, isn't it. That's the lookahead.
The right context is what buys you punctuation. Without it, you're streaming. With a little of it, you can catch a sentence boundary before you commit to a comma. With a lot of it, you can resolve harder cases, but you pay in latency and memory.
So the model is running over the whole file, but it's never holding the whole file in its head at once.
Think of it as a sliding window with overlap. The model takes a chunk, attends within its window, emits tokens, slides forward, and the windows overlap so the boundary doesn't produce a seam. Memory and compute become predictable regardless of total audio length. That's the property that makes it deployable.
Predictable is the word. Not small. Predictable.
Predictable is what you need on a phone, because you know your budget in advance. If the window is fixed, the peak memory is fixed, and you can decide before you ship whether it fits.
So the answer to Daniel's first question is yes, the middle ground exists, and it's not a hack. It's the default behaviour of the long-form models.
With one caveat, which is the second half of his question. The default context size in a NeMo config was chosen for server GPU inference. Not for a phone.
So Daniel's instinct that he'd have to tune it is right.
It's right for the case where he wants to push it. If you're on a constrained device, you might need to reduce the right context to fit memory, and that will degrade punctuation slightly. Or you might increase left context to improve consistency, at the cost of latency.
There's a tradeoff in each direction.
There is. And it's not symmetric. Left context is cheap relative to right context, because left context doesn't delay the output. Right context is what you're paying for with latency.
So if you're dictating and you want the words to appear as you speak, you want a small right context. If you're transcribing a file and you don't care about latency, you can afford a big one.
And that's the design decision the parameter exposes. It's not hidden. In NeMo, att_context_size is a first-class config value. You can change it at inference time.
Which brings us to the second half of Daniel's question. Which models actually give you the dial.
Nvidia's NeMo models do. Parakeet and Canary both expose att_context_size in their config, and you can change it without retraining, because the model was trained with variable context in mind. That's a deliberate design choice.
What about Whisper?
Whisper is different. It uses a fixed thirty-second window. It processes audio in chunks, but within that window the attention is full, and you can't tune the context size the way you can in NeMo. Not without retraining or using a different inference framework.
So Whisper gives you a window, but not a dial.
Whisper gives you a window. NeMo gives you a dial. That's the cleanest way to put it. And for on-device deployment, that difference matters a lot, because the thing you want to tune is exactly the thing Whisper doesn't let you tune.
What about the rest of the field? Meta's MMS, the Kaldi-derived stuff, the streaming conformers.
The terminology varies, and the exposure varies. Some of them have configurable chunk sizes. Some of them have a fixed chunk size and no way to change it without rebuilding. The point is that bounded context is not a universal standard. It's a design choice, and some architectures make it explicit and some bury it.
So when Daniel asks which models explicitly support it, the honest answer is a short list, and NeMo is at the top of it.
NeMo is the one where the parameter is named, documented, and changeable. That's the one to point at.
Now the part I think is the most interesting, and the part that I suspect gets missed. Even if the model supports it, the inference engine has to respect it.
That's the thing that trips people up. NeMo's chunked inference is optimised for CUDA. On a phone, you're not running CUDA. You're running ONNX Runtime, or TensorRT, or a custom mobile runtime.
And the bounded context mechanism has to survive the export.
It has to be supported by the runtime. It's not automatic. When you export the model, you have to make sure the attention mask is preserved and that the runtime actually respects it. If the runtime doesn't understand the mask, it'll either ignore it, which gives you wrong results, or it'll reject the model.
There's a gap between "the model has the parameter" and "the parameter does anything on your device."
There is. And it's not a small gap. It's the difference between a config value and a working deployment.
Which means the answer to Daniel's second question, do they need to be deliberately invoked and tuned, is yes, and it's not just tuning. It's verifying that the runtime honours the bound at all.
That's the part I'd flag hardest. If you're building offline dictation, the model choice is half the problem. The runtime is the other half.
Let's bring it back to Daniel's actual situation, because he's not building a research pipeline. He's dictating a few sentences at a time on a phone.
His manual start-stop habit is a human-imposed bound on context. He's doing by hand what the model could do for him.
Because when the recording is short, the whole thing fits in the attention window, and the punctuation is good.
Right. The recording is short enough that the bounded window covers all of it. So he gets the benefit of full context without paying for it, because the audio is small.
The cost of that workaround is that he's breaking his own flow. He stops, he checks, he restarts.
Which he said is partly deliberate, to catch mistranscriptions. But the other half is the resource anxiety, and that half is unnecessary if the model is doing chunked inference properly.
If he used a model with a well-tuned bounded context, he could record longer without the manual breaks.
The model would maintain punctuation quality within its window, and the app could stitch the chunks together seamlessly. From the user's side, it's one continuous stream. From the model's side, it's a sequence of overlapping windows.
Which is exactly the thing he described wanting. Inference in the background, presented as one continuous stream.
That's the design. And it's already how the long-form models work. The question is whether the app exposes it, and whether the runtime supports it.
Here's the part I keep coming back to. If the mechanism is this well understood, why is it not a standard exposed parameter across every ASR model?
Partly technical, partly product. The technical part is that not every architecture can be trained with variable context. If the model was trained with a fixed window, you can't just change the window at inference time and expect it to work. The positional encoding won't match.
The dial only exists if the model was built to have a dial.
Right. And the product part is that exposing a parameter means supporting a parameter. You have to document it, you have to test it across hardware, you have to handle the support tickets when someone sets it wrong. For a lot of teams, a fixed window that works well enough is the easier product.
Even if it's worse for the user.
Even then. And that's the tension. The best deployment is the one where the dial exists and the app sets it sensibly for your hardware, without you ever knowing.
Which is where I'd push on the framing a bit. Daniel asked whether he needs to deliberately tune it. The better world is one where he doesn't, because the app does it for him.
That's the mature version. You don't tune your phone's memory allocator. You shouldn't have to tune your dictation model's attention window.
But today, you might have to, if you're the one building it.
If you're building it, yes. If you're using it, you're at the mercy of whoever built it.
Which is a good place to pause, because there's someone at the desk who has actually done this for a living.
Hilbert: The doctor would say the word "period" out loud. As a word. "Patient has no history of heart disease period." And you had to know it was a period, not the word period. That was the whole job.
You were the punctuation model.
Hilbert: I was the punctuation model. Eight hours a day, headphones on, listening to doctors dictate notes into a dictaphone. The words were easy. The words were the easy part. Knowing where the sentence ended, that was the job.
The doctors didn't help you.
Hilbert: The doctors helped themselves. One of them would dictate for six minutes without a breath and then say "new paragraph" and expect you to know where the paragraph started. It didn't start where he said it started. It started four sentences earlier. You had to go back and fix it.
You were doing exactly what the right context does. Looking ahead to see where the thought actually ended.
Hilbert: You'd hold the whole thing in your head and then place the punctuation where it belonged, not where he said it. My supervisor marked me down for comma splices the doctor never spoke. That was the note. "Comma splice, not present in source audio."
Which is a beautiful way to describe what a streaming model does wrong.
Hilbert: A streaming model commits. It hears a pause, it writes a comma, and it can't take it back. I could take it back. I'd listen to the whole thing and then place the marks. That's the difference.
You tried to build this yourself at some point.
Hilbert: I tried to build a dictation setup on a Raspberry Pi. This was for my own notes, nothing professional. It kept running out of memory. I'd get about forty seconds in and it would fall over.
Because you were holding the whole thing.
Hilbert: Because I was holding the whole thing. I didn't know about att_context_size. I didn't know you could bound it. I thought the model either saw everything or it saw nothing, and I picked everything, and everything didn't fit.
So you gave up.
Hilbert: I gave up. Somebody is waiting for me in the car park. She's been there a while.
There's a lesson in the doctor story, though, and it's the one we've been circling. The punctuation isn't in the acoustics. It's in the meaning. And meaning lives in the context.
Which is why the bound matters. You need enough context to resolve the meaning, and no more than you can afford.
The doctor saying "period" out loud is the perfect edge case, because it's a token that's ambiguous between punctuation and content, and only the surrounding language disambiguates it. A streaming model would get that wrong roughly half the time.
A bounded-context model with enough right context would catch it, because it would see that "heart disease" is a complete clause and the "period" is closing it, not adding to it.
If it has the lookahead. If the right context is big enough. That's the tuning question again, in miniature.
The whole episode collapses into that one example. You need to see enough of the sentence to know what the word is doing.
You need to be able to afford the window that lets you see it.
If bounded context is the answer, why isn't it a standard exposed parameter everywhere? Is that a technical limit or a product decision?
Both, and the split matters. Some architectures can't do it without retraining. Others could and choose not to expose it.
As on-device ASR becomes more common, do you think we'll see more models designed with a tunable context from the start, or more fixed, mobile-optimised models that hide the dial?
I'd bet on a mix. The big vendors will keep the dial for their own deployment teams and hide it from users. The open models will expose it, because the people using them want to tune.
Which is the practical advice, then. If you're building offline dictation, look for models that expose att_context_size or something like it, and don't assume the default is right for your hardware.
The default was chosen for a server. Your phone is not a server.
One thought to leave people with. The reason Daniel's manual start-stop habit works so well is that he's been doing the chunking himself, by hand, for years. The models have been doing it internally the whole time.
He just didn't have the dial. Now he knows it's there.
Thanks to Hilbert Flumingtop, our producer, for the doctor story and for the Raspberry Pi cautionary tale.
This has been My Weird Prompts.
If you're getting something out of these deep dives, a review wherever you listen to podcasts helps other people find the show.
We'll be back soon.
See you tomorrow.
