Daniel's been digging into on-device speech-to-text and he's got three questions stacked inside each other. First, what actually is NVIDIA Parakeet — not the marketing page version, but what's happening under the hood. Second, how does it perform on real smartphone hardware, not a data center GPU. And third, the one he really cares about — head-to-head against Whisper for the AI keyboard and dictation use case. Which one wins when your CPU is a phone chip and the user expects words to appear as they speak.
This is one of those comparisons where the spec sheet gives you the wrong answer unless you understand why the architectures are different in the first place. Most people see Whisper's been the default for years and assume it's optimized for everything. It's not. It was built for batch transcription on servers.
And Parakeet was built for streaming from day one.
So the question isn't which model is better in some abstract sense — it's which one was designed for the thing Daniel's actually trying to do. And dictation on a phone keyboard is about as streaming as a problem gets.
Let's start with what these things actually are. Whisper everyone knows — OpenAI dropped it in twenty twenty-two, trained on six hundred eighty thousand hours of multilingual audio. It's an encoder-decoder transformer. The encoder processes the whole audio clip into a representation, then the decoder generates tokens one at a time, attending back to that full representation at every step.
And that cross-attention step is the magic and the bottleneck. It's why Whisper can hear a word mumbled three seconds ago and correct it based on context that comes later. But it also means you cannot start decoding until the audio is done. The encoder needs the full input.
So you're sitting there after you finish speaking, waiting.
Waiting for the encoder to run, then waiting for the decoder to generate every token autoregressively — each one depends on the last, so you can't parallelize. On a server GPU this is fine. On a phone, it's the difference between dictation that feels like typing and dictation that feels like sending a fax.
Parakeet does it differently.
Radically differently. Parakeet uses CTC — Connectionist Temporal Classification. This is an older technique, but NVIDIA's implementation is modern. The key insight is that CTC models are non-autoregressive. The model processes audio frames as they arrive and outputs a probability distribution over tokens for each frame. Then you do greedy decoding — take the most likely token at each time step, collapse repeated tokens, remove blanks, and you've got your text.
No waiting for the end of the utterance.
No waiting. The moment an eighty-millisecond audio frame hits the model, you get token predictions. You can start displaying characters before the speaker finishes the sentence. For a keyboard dictation use case, that's the whole ballgame.
So that's the architecture split. But here's the thing that threw me when I first looked at the numbers. Parakeet-ctc-zero-point-six-b has six hundred million parameters. Whisper Small has two hundred forty-four million. Whisper Tiny has thirty-nine million. And yet Parakeet runs faster on-device than both of them.
This is the counterintuitive part that most comparisons miss. Bigger model, faster inference. The reason is that parameter count isn't what slows you down — it's the decoding algorithm. Whisper's autoregressive decoder has to run sequentially, token by token, each one attending back to the full encoder representation. For a ten-word sentence, that's maybe fifteen to twenty tokens, each requiring a full forward pass through the decoder with cross-attention.
And Parakeet?
One forward pass through the encoder, then an argmax over the time dimension. That's it. No decoder loop. No cross-attention per token. The six hundred million parameters are all in the encoder, which runs once. So you're comparing one large matrix multiplication against twenty smaller ones that each have to wait for the previous one to finish. On hardware that's good at parallel computation — which modern phone chips are — the single large operation wins.
It's like the difference between reading a page and copying it letter by letter.
That's a really clean way to put it. The CTC model reads the whole page at once and writes it down. The autoregressive model reads the page, then writes one letter, checks the page again, writes the next letter, checks again. The checking is the cross-attention, and it's expensive.
So let's put some numbers on this. What does this actually feel like on a phone?
Whisper Tiny through whisper.cpp on something like a Pixel eight — you're looking at roughly three hundred milliseconds of latency for a five-second phrase. And that's Tiny, the thirty-nine million parameter version that's been quantized and optimized to within an inch of its life. The latency scales with audio length too, because the encoder processes the full clip. A fifteen-second dictation chunk might push you past half a second.
And that's the version where you've already sacrificed accuracy for speed.
Right. Whisper Tiny gets about nine percent word error rate on LibriSpeech clean. That's usable, but you'll be correcting things. Whisper Small gets closer to five or six percent, but the latency jumps because you've got two hundred forty-four million parameters to run through an autoregressive decoder.
Parakeet?
NVIDIA claims a real-time factor below zero-point-one on a single GPU. On a Snapdragon eight Gen three in a phone, you can sustain sub-hundred-millisecond streaming latency. The model's outputting tokens every eighty milliseconds of audio. That's fast enough that characters appear as you speak — the dictation feels instantaneous.
And accuracy?
On LibriSpeech clean, Parakeet-ctc-zero-point-six-b gets around five-point-five percent WER. That's better than Whisper Tiny and competitive with Whisper Small. But — and this is where the architecture tradeoff bites — CTC models are weaker at modeling long-range dependencies.
Explain what that means in practice.
Say you say the word "read" but the sentence is in past tense, so it should be pronounced "red" not "reed." Whisper's attention mechanism sees the whole sentence — it knows from context which pronunciation was intended and can correct the transcription accordingly. Parakeet makes a decision per audio frame based on local context. If the acoustic signal is ambiguous, it might get it wrong because it can't look ahead three seconds to disambiguate.
So Whisper is better at the kind of correction where later context clarifies earlier audio.
Homophones, proper nouns, code-switching — anything where local acoustic information isn't enough. CTC models are fundamentally local decision-makers. That's the tradeoff for streaming speed.
Which brings us to the keyboard use case specifically. Daniel's not asking about transcribing meeting recordings. He's asking about tapping the microphone button and having words appear as he speaks.
And that use case is streaming-native. The user expectation is that characters show up in real time. Not after a pause. Not in bursts. Character by character, word by word, as you speak.
Whisper's architecture fights this.
It really does. The community workaround with whisper.cpp is to use a voice activity detector and a sliding window — you chunk the audio into pieces, transcribe each chunk, and stitch the results together. But now you've introduced a new problem: chunk boundaries. If you cut a chunk in the middle of a word, the model loses context and you get transcription errors at every boundary. You can overlap the chunks to mitigate this, but overlapping means processing the same audio twice, which eats into your latency budget.
So the streaming retrofit has a ceiling.
A hard ceiling. And the user experience reflects it — words appear in bursts after each chunk boundary, not smoothly. It feels janky. Compare that to Parakeet with a simple greedy decoder: every eighty milliseconds of audio produces tokens, the keyboard inserts them as they arrive, and the user sees continuous text generation. The difference in perceived responsiveness is dramatic.
Perceived responsiveness might matter more than a one or two percent word error rate difference for this use case.
I think it does. Users tolerate typos more than they tolerate lag. A typo you can fix with a tap. Lag you can't fix — you just have to wait. And waiting breaks the flow state that dictation is supposed to enable. The whole point of speaking instead of typing is speed and fluidity. If the system makes you wait, you might as well have typed.
There's another dimension here that doesn't get enough attention. Punctuation.
Oh, this is where it gets interesting. Whisper was trained on six hundred eighty thousand hours of subtitled video and audio with transcripts. That training data included punctuation — periods, commas, question marks, the works. So Whisper learned punctuation implicitly. It just knows where the periods go because it saw them in the training transcripts.
It's baked into the model.
Deeply baked in. Parakeet, being a CTC model, doesn't do punctuation natively. The CTC loss operates at the token level on audio-text alignment — it's not modeling the kind of semantic structure that tells you where a sentence ends. So NVIDIA's solution is to ship a separate Punctuation and Capitalization model as part of the Riva platform.
A bolt-on.
You could call it that. Or you could call it clean engineering separation. The PC model runs after the CTC model, takes the raw word stream, and inserts punctuation and capitalization. On a phone, you can choose to run it or not — if you're memory-constrained, you skip it and get a wall of lowercase text. If you've got the headroom, you add it and get properly formatted output.
What's the memory hit?
Depends on the PC model size, but you're adding a second neural network to your inference pipeline. On a flagship phone with eight or twelve gigs of RAM, it's fine. On a budget device with four gigs, you might have to choose between punctuation and having other apps stay alive in the background.
Whisper gives you punctuation for free, architecturally.
For free, but also without control. Whisper's punctuation is whatever the model learned from its training distribution. If you want different punctuation behavior — say, you want semicolons inserted in a specific way for legal dictation — you can't tweak it without fine-tuning the whole model. With Parakeet's separated PC model, you can swap in a domain-specific punctuation model without touching the ASR core. That's useful for enterprise deployments.
So the architectural separation that looks like a hack in a spec sheet comparison turns out to have real deployment advantages.
It's the kind of thing that only becomes visible when you actually try to ship a product. The Riva platform also includes separate models for inverse text normalization — converting spoken forms like "three PM" to "3:00 PM" — and those are swappable too. Whisper does some of this implicitly, but again, you get what the training data gave you.
Let's talk about the ecosystem around these things. Whisper has whisper.cpp, which is remarkable — a community-led project that's optimized Whisper inference to run on basically anything with a processor.
Gerganov and the contributors have done incredible work. whisper.cpp supports quantization down to four bits, CoreML delegation on Apple hardware, ARM optimizations — it's the reason Whisper runs on phones at all. But it's still a retrofit. The streaming mode with voice activity detection and sliding windows is clever engineering working around a fundamental architectural constraint.
And Parakeet's ecosystem?
NVIDIA Riva is the official stack. It's more polished — you get the ASR model, the punctuation model, the inverse text normalization, all designed to work together. But it's also NVIDIA's playground. The current smallest published Parakeet model is zero-point-six billion parameters. That's too large for budget phones. There's no Tiny equivalent yet — no sub-hundred-million parameter Parakeet variant that NVIDIA has open-sourced.
So if you're building a keyboard app that needs to run on a two-hundred-dollar Android phone, Whisper Tiny through whisper.cpp is your only option right now.
Your only option, and it'll work, but it won't feel great. The latency will be noticeable, the accuracy will be mediocre, and you'll be fighting the streaming problem constantly.
What about fine-tuning? Daniel's done work with fine-tuned Whisper models.
This is where Whisper's maturity pays off. There's a huge ecosystem of fine-tuned Whisper variants on HuggingFace — models tuned for medical dictation, legal transcription, specific languages and accents. The fine-tuning pipeline is well-understood. You take a pre-trained Whisper checkpoint, run it on your domain-specific audio with transcripts, and you get meaningful accuracy improvements.
And Parakeet?
Much smaller fine-tuning ecosystem, simply because the model's newer and the CTC architecture requires different tooling. It's possible — NVIDIA provides fine-tuning scripts in the NeMo framework — but there aren't hundreds of community fine-tunes to choose from. If you need domain-specific accuracy out of the box, Whisper's ecosystem is a real advantage.
But for the keyboard use case, domain specificity might matter less than responsiveness.
I think that's right. The keyboard dictation use case is general-purpose by definition — people dictate texts, emails, notes, search queries. The vocabulary is broad but not deeply technical. A five to six percent word error rate on clean speech is acceptable for that use case if the latency is near-zero. What kills the experience isn't the occasional wrong word — it's the system feeling sluggish.
Let's do a concrete comparison. Same sentence, both models, on a phone.
Alright. The sentence: "Hey team, let's schedule a follow-up for next Tuesday at three PM." With Parakeet, you tap the microphone, start speaking, and "Hey" appears before you've finished saying "team." By the time you say "PM," the full sentence is on screen, properly streaming, each word appearing roughly as you articulate it. The punctuation model — if you're running it — inserts the comma and period a fraction of a second after the words appear.
And Whisper?
You tap the microphone, speak the sentence, release. Then there's a pause. Maybe three hundred milliseconds on a good phone with Whisper Tiny. Then the full sentence appears at once. Correctly punctuated, probably accurate, but it all lands in one burst. If you used the streaming mode with chunking, you might see "Hey team" appear, then a brief pause, then "let's schedule a follow-up," then another pause, then the rest. The chunk boundaries create visible stutter.
The first experience feels like the phone is reading your mind. The second feels like the phone is thinking about it.
And that's the whole thing. The perceptual difference between streaming and batch processing is enormous for dictation, even if the total time from speech-end to text-complete is similar. Users don't measure latency with a stopwatch — they measure it by whether the system feels responsive.
There's an edge case that actually illuminates the architectural difference nicely. The phrase "I need to record a record."
Perfect example. Same spelling, different pronunciation, and the correct interpretation depends on part-of-speech context that spans the whole sentence. Whisper's attention mechanism sees the full utterance, recognizes the syntactic structure, and can disambiguate — it knows the first "record" is a verb and the second is a noun. Parakeet's CTC model processes each audio frame locally. It might output the same token for both instances because the acoustic signal is similar, and it doesn't have the global context to know they should be different.
So Whisper wins on homophone disambiguation.
Wins clearly. And this extends to proper nouns, technical terms, anything where local acoustic information is ambiguous and you need sentence-level or paragraph-level context to get it right. CTC models have a fundamental ceiling on this kind of accuracy that attention models don't.
Which suggests the real answer to Daniel's question might be: it depends on what you're dictating.
For short, conversational dictation — texts, quick emails, search queries — Parakeet's streaming advantage dominates and the accuracy difference is negligible. For long-form dictation where accuracy on proper nouns and technical vocabulary matters — medical notes, legal documents, anything you'd previously have typed carefully — Whisper's attention mechanism might be worth the latency cost.
And for code-switching? Daniel lives in Jerusalem. People switch between English and Hebrew mid-sentence constantly.
Neither model handles code-switching well out of the box. Whisper was trained on multilingual data, so it has some capacity, but it's not optimized for mid-utterance language switching. Parakeet's CTC architecture is inherently monolingual per model — you'd need a model specifically trained on code-switched data. Both would require fine-tuning for that use case.
So on-device STT isn't solved. Neither model handles accents, code-switching, or domain jargon well without additional work.
Not solved. Improved dramatically in the last three years, but not solved. And the ecosystem layer might matter more than the model architecture in the long run. whisper.cpp versus NVIDIA's TensorRT for mobile — that's the battle that's going to determine what ships on phones.
TensorRT is NVIDIA's inference optimization framework. It's what would make Parakeet run fast on ARM chips.
Right. And whisper.cpp is the community equivalent for Whisper. Both are trying to solve the same problem — take a model built for GPUs and make it run efficiently on phone CPs and NPUs. The quality of that optimization layer might matter more than which architecture you start with. A perfectly optimized CTC model beats a poorly optimized attention model, and vice versa.
Which brings us to the open question. Will NVIDIA release smaller Parakeet variants — sub-hundred-million parameters — that can run on mid-range and budget phones?
That's the thing I'm watching. Right now, Parakeet-ctc-zero-point-six-b is the smallest published model. Six hundred million parameters is a lot for a phone. You can quantize it, you can optimize it, but there's a floor on how small you can make a six-hundred-million-parameter model and still have it run well on a two-hundred-dollar Android device.
Whisper Tiny at thirty-nine million parameters runs on basically anything.
Runs on a smartwatch, probably. The parameter efficiency of the CTC architecture means you might be able to build a hundred-million-parameter Parakeet variant that matches Whisper Small's accuracy while keeping the streaming latency advantage. But NVIDIA hasn't released that model. Whether they do — and whether they open-source it or keep it behind the Riva platform — is going to determine how widely Parakeet gets adopted for on-device use.
The other possibility is that someone in the community trains a small CTC model from scratch using the same architecture. The CTC approach isn't proprietary.
It's not. CTC has been around since twenty-sixteen — Graves et al. The innovation in Parakeet is the scale, the training data, and the integration with Riva. But the architecture is published. A motivated community could absolutely train a CTC-based streaming ASR model optimized for mobile.
So we might end up with a world where the keyboard on a Pixel phone uses one architecture and the keyboard on a Galaxy uses another, and users never know or care — they just notice whether the dictation feels snappy.
That's probably the right way to think about it. The model is infrastructure. The user cares about the experience. And the experience of dictation is about latency first, accuracy second, and punctuation third — in that order, for most people, most of the time.
I want to circle back to something you said earlier about the punctuation model being a bolt-on. Because I think there's a deeper point there about engineering philosophy.
Go on.
Whisper's approach is to bake everything into one model — transcription, punctuation, formatting, even some translation capability. It's the "one model to rule them all" philosophy. Parakeet's approach is modular — ASR here, punctuation there, inverse text normalization somewhere else. And I think people default to assuming the integrated approach is more elegant.
But modularity has real advantages when you're shipping a product. If the punctuation model has a bug, you update just the punctuation model. If a customer needs different formatting for a specific vertical, you swap the inverse text normalization module. You don't have to retrain the ASR core. That's valuable in production.
And the integrated approach has the advantage of simplicity — one model file, one inference pass, you're done. No pipeline to manage.
Which matters a lot for an open-source project like whisper.cpp that's maintained by volunteers. The simplicity of "one model does everything" is a feature when your development resources are limited.
So the philosophical difference maps onto the organizational difference. OpenAI ships integrated models because they're a research lab that releases artifacts. NVIDIA ships modular platforms because they're an enterprise infrastructure company that builds systems.
And Daniel, as someone who builds with these things, probably cares more about the integration experience than the philosophy. Can he drop it into a pipeline? Can he customize it? Can he debug it when something goes wrong?
The modular approach gives you more knobs to turn. The integrated approach gives you fewer things that can break.
Pick your poison.
Hilbert: They both miss punctuation on "um" and "uh."
Go on.
Hilbert: I spent six months in twenty nineteen labeling disfluencies for a startup called Voxly. They were building a CTC-based dictation system — thought they were going to beat Dragon NaturallySpeaking. The office was above a dry cleaner in San Jose. We'd sit there with headsets on, marking every "um," every "uh," every false start, every repeated word. The model was supposed to learn to strip them out automatically.
Did it work?
Hilbert: It learned to strip them. It also learned to strip commas, periods, and the word "I" about forty percent of the time. The CTC loss doesn't distinguish between "this token is disfluency" and "this token is punctuation" — it's all just tokens the model decides whether to emit. So you'd get these clean, fluent transcripts with no pauses, no sentence boundaries, just a wall of words. They tried bolting on a punctuation model — this is twenty nineteen, mind you — and it kept inserting periods after the word "the."
So the same problem NVIDIA's solving with a separate PC model in twenty twenty-six was already the known failure mode of CTC systems seven years earlier.
Hilbert: Same problem, same solution, more GPUs. The Voxly CEO pivoted to crypto in twenty twenty. I still have the headset. It's in a box somewhere.
The thing that strikes me about that story is that the punctuation problem isn't a minor edge case — it's central to whether dictation is actually usable. A wall of lowercase text with no sentence boundaries is exhausting to read. You have to do the punctuation yourself mentally, which defeats the purpose of dictating in the first place.
And Whisper solved this by training on subtitles, which are already punctuated. Six hundred eighty thousand hours of data where the commas and periods are just there in the training signal. No separate model needed.
Hilbert: Subtitles also have speaker labels and sound effect descriptions. I always wondered if Whisper ever hallucinates a "door slams" into a transcript.
It actually does, occasionally. There are documented cases of Whisper generating non-speech tokens from its training distribution — laughter, music notes, sound effects. It's rare, but it happens.
So both approaches have their ghosts in the machine. Whisper might occasionally think it heard applause. Parakeet might give you a wall of text with no periods. Pick your artifact.
Hilbert: The Voxly model also couldn't handle anyone with an accent. We trained on LibriSpeech, which is mostly American English read by people from the Midwest. First time we tested it on someone from Glasgow, the word error rate hit about sixty percent. The CEO said that was an edge case.
That's not an edge case. That's most of the world's English speakers.
Hilbert: He drives a Tesla now. Does something with NFTs.
The accent point connects to something we haven't touched on. Both Whisper and Parakeet were trained primarily on English-language data from specific demographics. The benchmarks we've been citing — LibriSpeech clean, five percent, nine percent — those are on curated datasets that don't represent real-world dictation diversity.
That's fair. Real-world dictation includes accents, background noise, crosstalk, people trailing off mid-sentence, people correcting themselves. The published word error rates are best-case numbers. On actual smartphone dictation in a coffee shop, both models are probably double those figures.
Which makes the latency advantage even more important. If both models are going to make errors, the one that lets you see and correct them faster wins.
The correction loop is underated in these comparisons. With Parakeet's streaming, you see an error appear in real time and you can stop speaking, correct it, and continue — all within the same dictation session. With Whisper's batch output, you see the full transcript after a pause, then you have to go back and find the error, which might have scrolled off the screen if it was a long dictation.
The UX of error correction is where the architectural difference becomes a product difference.
And product people care about that more than they care about word error rate benchmarks. I've seen product teams pick a technically inferior model because the correction experience was better. Users don't judge accuracy in the abstract — they judge how annoying it is to fix mistakes.
Where does this leave Daniel's question? Parakeet versus Whisper for the AI keyboard dictation use case.
If you're building a keyboard app today and you care about the dictation feeling responsive — which you should, because that's what users notice — Parakeet's architecture is the better fit. The streaming-native design maps directly onto the user expectation of characters appearing as they speak. The accuracy is competitive on clean speech, and the modular punctuation approach gives you deployment flexibility.
But.
But Whisper has the ecosystem. whisper.cpp runs everywhere. There are dozens of fine-tuned variants. The community tooling is mature. If you're a small team building a keyboard app, you might choose Whisper not because it's architecturally better for the use case, but because you can actually ship it on every device without negotiating an NVIDIA enterprise license.
If NVIDIA releases a sub-hundred-million-parameter Parakeet variant under an open license?
Then the calculus flips. A small, fast, streaming-native model with competitive accuracy and a permissive license — that's the thing that would make Parakeet the default choice for on-device dictation. The architecture is right. The model size and licensing are the blockers.
The ecosystem war you mentioned earlier — whisper.cpp versus TensorRT for mobile — that's going to determine whether that open variant matters even if it ships. The best model in the world doesn't help if the inference runtime is slow or hard to integrate.
That's where I think the next year gets interesting. NVIDIA has the inference optimization expertise — TensorRT is good. But whisper.cpp has the deployment footprint — it's already in apps, it's battle-tested on hundreds of devices. The runtime that wins is the one that makes it easiest for a developer to add a microphone button and get good-enough dictation with zero configuration.
The misconception I keep seeing is that bigger models are slower on-device, so you should always pick the smallest model for mobile. The truth is that architecture dominates parameter count — Parakeet at six hundred million parameters runs faster than Whisper Tiny at thirty-nine million because it avoids the autoregressive decoding loop. Parameter count tells you about memory, not about latency. The decoding algorithm tells you about latency.
The second misconception is that Whisper is real-time. It's not. It's batch-first. The streaming support in whisper.cpp is a retrofit — clever and useful, but a retrofit. If you need actual streaming dictation where words appear as you speak, you need a model architected for that from the start.
The open question I'm left with is whether the on-device STT race gets decided by model quality at all, or by the inference runtime layer underneath. whisper.cpp versus TensorRT — that's the battle that determines what ships in the next generation of phone keyboards. And neither of those projects is standing still.
The model is the headline. The runtime is the product. And Daniel, as someone who actually builds with these things, probably figured that out three prompts ago.
This has been My Weird Prompts. Thanks to our producer Hilert Flumintop for keeping the show running — and apparently for six months of disfluency labeling that prepared him for exactly this conversation.
You can find every episode at myweirdprompts dot com. If you've got a question you want us to dig into the way we dug into this one, email the show at show at myweirdprompts dot com.
We'll be back soon.
