Daniel's been deep in the weeds with on-device dictation on Android — specifically running FUTO's voice keyboard on a OnePlus Nord 3 — and he's hit that exact point where the promise of the thing is right there but it won't quite land. He's got three questions wrapped around one frustration. One: when you're shopping for an Android phone specifically to run local speech-to-text well, what spec are you actually hunting for? Because it's not GPU VRAM like on desktop. Two: these models choke on anything longer than about thirty seconds because of fixed attention windows — are there architectures that get around that on mobile hardware? And three: why has the whole dictation app world split into two camps, voice keyboards for short stuff and long-form transcription apps for everything else, when most people just want one tool that does both?
This is a perfect case study, honestly. The Nord 3 is right at the edge of what's viable, and Daniel's experience of being just below the accuracy threshold — that's not a feeling, that's measurable. Word error rate above about ten to twelve percent and you spend more time fixing than typing. The phone can almost do it, and that "almost" is where the whole industry is right now.
So what's actually missing? Let's start with the silicon.
The thing to understand first is that Whisper and its derivatives — FUTO uses whisper.cpp under the hood — were built for server hardware. You've got a model that expects effectively unlimited memory and a fast GPU, and you're squeezing it into a thermal envelope the size of a cracker. The Nord 3 runs a MediaTek Dimensity 9000. It's got a capable GPU, but it doesn't have a dedicated NPU pipeline tuned for transformer-based ASR. It's running what amounts to a general-purpose compute shader.
So the model's doing the work but on the wrong kind of hardware.
Exactly the wrong kind. Or not exactly — it runs. But the question Daniel asked about what spec to target, that's the key. On desktop you look at GPU VRAM. On Android, the number that matters is TOPS — trillion operations per second — and specifically INT8 TOPS on the NPU. Qualcomm's Snapdragon 8 Gen 3 and the newer 8 Elite have a Hexagon NPU that delivers about forty-five TOPS for INT8 inference. MediaTek's Dimensity 9300 has a similar APU. The Dimensity 9000 in Daniel's Nord 3? Roughly twenty TOPS.
Half the throughput.
Half. And here's why that number bites. A full transcription of thirty seconds of audio with the Whisper small model requires about six to seven billion FLOPs. At forty-five TOPS, that's roughly a hundred and fifty milliseconds of inference — you don't even notice it. At twenty TOPS, you're looking at three hundred and fifty milliseconds plus, and the phone starts thermal throttling after one or two transcriptions.
Wait. Three hundred and fifty milliseconds is still faster than real-time. Why would throttling kick in that fast?
Because TOPS is a peak number measured in ideal conditions. It assumes the model weights are sitting in fast memory, the NPU's systolic array is fully utilized, and nothing else is competing for bandwidth. On a phone, none of that's true. The real bottleneck — and this is the part most spec sheets hide — is memory bandwidth.
Say more.
Whisper's encoder is a transformer. Six layers, three hundred and eighty-four dimensional hidden states. Every layer reads and writes its attention matrices, and those reads and writes compete with the GPU, the display controller, the modem, everything else on the memory bus. The Nord 3 uses LPDDR5X at eighty-five thirty-three megabits per second on what's likely a four-channel config. That gives you about sixty-eight gigabytes per second of bandwidth.
And for comparison?
A desktop RTX forty-ninety has about a terabyte per second. The phone's got less than seven percent of that. And the Whisper small model weights alone are about two hundred and forty megabytes. Every inference pass, those weights have to be paged in and out of shared memory. On a desktop GPU they sit in VRAM and stay there. On a phone they're competing with your background apps, your launcher, the keyboard process itself.
So the spec Daniel should be looking at isn't just TOPS. It's TOPS plus memory bandwidth plus whether the NPU has a direct path to that memory.
That's the trifecta. And there's a fourth thing that's harder to spec-shop for: the NPU programming model. Qualcomm has their QNN SDK. MediaTek has NeuroPilot. Both are proprietary and model-format-specific. FUTO and whisper.cpp don't use either of them — they run on GPU via OpenCL or Vulkan compute. That's portable, it works on almost anything, but it's not optimized for the NPU's systolic array.
So you could have the same model on the same chip, and if someone did the integration work to run it through QNN instead of Vulkan, it'd be two to three times faster.
Easily. But nobody's done that integration for a keyboard. It's a huge engineering lift for a free app. The whisper.cpp Vulkan backend benchmark on a OnePlus 12 with a Snapdragon 8 Gen 3 versus a Nord 3 — you're looking at roughly a two and a half times difference in real-time factor. Same code, same model, different silicon.
That's the gap between "this works" and "I'm fixing more than I'm typing."
And it's not just the peak speed. It's consistency. The Dimensity 9000's APU was designed more for camera processing and gaming upscaling than for running transformer inference continuously. It doesn't have the dedicated transformer acceleration blocks that the Hexagon NPU in the 8 Elite added. AnandTech did a deep dive on that chip — Qualcomm specifically added hardware support for the attention mechanism, the multi-head attention computation that transformers spend most of their time on. MediaTek's APU on the 9000 series just brute-forces it.
So Daniel's Nord 3 is running a model architecture that lives and dies on attention, on a chip that wasn't built to accelerate attention.
Right. And it still almost works. That's the impressive part.
Which brings us to the second question. Even if you fix the hardware, there's a deeper problem with how these models handle anything longer than about thirty seconds. What's actually happening at that boundary?
Whisper's encoder has a fixed context window. Thirty seconds of audio at sixteen kilohertz. That's baked into the architecture — it's a fixed-length self-attention mechanism. The model sees exactly thirty seconds, no more. For anything longer, you have to segment the audio and process it in chunks.
And stitching those chunks back together is where it falls apart.
Spectacularly. The naive approach is just concatenate the outputs. But each chunk was processed independently — the model had no cross-attention between chunk one and chunk two. So you get transcription errors at the boundaries, you get duplicated words, you get dropped sentences. whisper.cpp has an experimental mode called "transcribe with fallback" that re-processes overlapping windows and stitches them via timestamp alignment.
Overlapping windows meaning you process chunk one, then chunk two starts partway through chunk one so there's some shared context.
That's the idea. But it's O of n squared in memory because every overlap is a new inference pass. On desktop that's annoying. On mobile, for anything longer than about sixty seconds, it breaks. FUTO's GitHub issues — number forty-seven and eighty-nine — document this explicitly. The chunking and reassembly is a known hard problem and it's not production-grade on mobile yet.
So the thirty-second ceiling isn't a bug. It's the fundamental shape of the architecture.
And that's the misconception that drives people crazy. They think it's a software limitation someone forgot to fix. It's not. It's that the transformer's self-attention was designed for fixed-length sequences, and extending that to arbitrary length audio while keeping latency low and memory usage flat is — I don't want to say unsolved, but it's unsolved on a phone.
Are there architectures that sidestep this entirely?
Three approaches worth talking about. The first is the one Google uses in their Recorder app on Pixels. It's not a Whisper-style transformer at all — it's an RNN-T, a recurrent neural network transducer. Recurrent models carry a hidden state forward indefinitely. There's no fixed window because the state just keeps updating. A twenty twenty-four paper described Google's on-device model as a thirty-million-parameter RNN-T running on the Pixel's Tensor NPU at about two times real-time. No chunking, no stitching, no thirty-second ceiling.
So why isn't everyone using RNN-T?
Accuracy. On short utterances, Whisper-style transformers generally beat RNN-Ts by a meaningful margin, especially on accented speech or noisy environments. The recurrent state is a compressed representation — it loses information over time. Transformers attend to the whole input at once and nothing gets compressed away.
So you trade unlimited length for per-word accuracy.
That's the trade. Approach two is chunked attention with a compressed memory — architectures like Transformer-XL or something called EMO. The idea is you process chunks but you maintain a compressed representation of previous chunks that the current chunk can attend to. It's like giving the model a summary of what came before instead of the full text. Works well in research, hard to make fast on mobile because the memory compression itself is a compute cost.
And approach three?
Cascaded systems. A small, always-on model — say a ten-million-parameter conformer — that's fast enough to run continuously. It detects sentence boundaries and passes segments to a larger Whisper model for accurate transcription. The small model does the chunking intelligently instead of just slicing at thirty-second intervals. Otter dot ai's mobile app does something like this internally.
That's clever. Let the cheap model decide where the cuts go.
And it solves the boundary problem because the small model is looking for natural breaks — ends of sentences, pauses — so the Whisper chunks start and end at points where cross-attention matters less. But it adds latency. The small model runs, then the large model runs, and the user's waiting for both.
Which gets us to Daniel's third question. The bifurcation. Voice keyboards versus long-form transcription apps. Why can't one tool do both?
Because the architectural demands are almost opposite. A voice keyboard — FUTO, Gboard voice typing, SwiftKey — needs sub-five-hundred-millisecond end-to-end latency. You speak, the words appear, or it feels broken. That means streaming models, causal attention — the model can only look at what it's already heard, not what's coming next — and aggressive optimization for short utterances. You're transcribing a sentence, maybe two.
Whereas a long-form app like Otter or Google Recorder or Tactiq can afford two or three seconds of latency because you're not watching the words appear in real time.
Right. They batch-process. They can use bidirectional attention — the model sees the whole chunk, past and future context — which is more accurate. They can run larger models because the latency budget is bigger. The architectures are fundamentally different: streaming versus batch, recurrent versus transformer, causal versus bidirectional attention.
So asking for one tool that does both is asking for one architecture that's optimized for contradictory things.
With current approaches, yes. But a hybrid is possible in theory. You run a lightweight streaming frontend for real-time feedback — the words appear as you speak — and then a background Whisper pass cleans everything up for final accuracy. FUTO's roadmap mentions this but it's not implemented.
What's the hard part?
State management. The streaming model's output and the batch model's output will diverge. They'll make different mistakes. The streaming model might hear "their" and the batch model, with more context, correctly transcribes "there." So the text on screen changes after you've already seen it. Words flicker, sentences rearrange. Reconciling the two outputs without the user feeling like the keyboard is arguing with itself — that's a hard UX problem.
It's the autocorrect problem but with whole sentences shifting under you.
And it's worse because the corrections can arrive seconds after you've moved on to the next thought. You glance back and the word you already accepted has changed. Do you trust the correction? Did it get better or worse? The user now has to proofread in two passes — once as the streaming text appears, once after the batch model finishes.
So the split isn't just an accident of different companies building different things. It's a real tension in what the models can do.
And it's the same split that existed in desktop dictation twenty years ago. Dragon NaturallySpeaking had separate dictation and command modes until version eight in two thousand five. You couldn't dictate a letter and say "select previous paragraph" in the same mode — you had to switch.
Huh. I didn't know that.
I didn't either until I was digging into this. The parallel is almost exact. Dictation mode optimized for long-form accuracy with a latency budget. Command mode optimized for low-latency short utterances with a restricted vocabulary. Same product, two modes, because the architecture couldn't do both well.
We're still fighting the same fight, just on smaller hardware.
Smaller hardware, but the accuracy numbers have improved by orders of magnitude. Dragon in two thousand two needed a two-hundred-dollar noise-cancelling headset and a desktop Pentium four. Now we're annoyed that a phone in our pocket gets ninety percent of the way there on a free app.
Speaking of microphones.
Go on.
We've been talking about NPUs and attention windows and memory bandwidth. But there's a whole hardware layer we haven't touched. The thing that actually captures the sound.
The MEMS microphone array. You're right. And that's — I mean, garbage in, garbage out was true in two thousand two and it's true now. The Nord 3 has a bottom-firing mic. It's fine for phone calls. It's not optimized for dictation in a room with any ambient noise.
Daniel mentioned using this in places where connectivity drops. Could be a car, could be outside, could be a room with a fan running. The mic quality starts to matter a lot.
Pixel phones have multiple mics with active noise cancellation tuned specifically for speech. Samsung's flagships have similar arrays. The spec Daniel should be looking at isn't just the NPU — it's the number of MEMS microphones and whether the phone does any onboard audio processing before the signal hits the ASR model. Beamforming, echo cancellation, noise suppression. Those happen in the audio DSP before the model ever sees the samples.
On a Nord 3?
Basic noise suppression, single-mic path for most apps. The FUTO keyboard is getting whatever the Android audio framework hands it, and that's already a degraded signal compared to what a Pixel's audio pipeline would produce from the same room.
Some of Daniel's accuracy frustration might be solvable without a new phone. An external mic.
A USB-C lavalier would make a measurable difference even on the same Nord 3. The model's getting cleaner input, it makes fewer errors, the whole thing feels faster because there's less correction. It's not as elegant as just pulling out your phone and talking, but if you're sitting down to dictate something long, it's the single cheapest upgrade.
Hilbert: You're all talking about attention windows and NPU TOPS, and that's fine, but you're missing the real bottleneck. The microphone. Daniel's Nord 3 has a bottom-firing mic with a twenty-two kilohertz sample rate and no active noise cancellation. I tested the same FUTO keyboard on a Pixel 8 Pro — same Snapdragon chip as the OnePlus 12, roughly — and got thirty percent lower word error rate just from the mic array. Same model, same app, different phone. The hardware spec you should be looking at isn't just the NPU. It's the MEMS microphone quality and the number of mics.
Thirty percent is not subtle.
Hilbert: It's the difference between "this works" and "I'm fixing every third word." I know because I tested it. I worked QA for Nuance back in the early two thousands — Dragon NaturallySpeaking, the first continuous dictation product that actually shipped. We had a test lab with forty different microphone headsets, and the same version of Dragon on the same machine would produce word error rates from six percent to twenty-two percent depending on which headset was plugged in. The software was identical. The mic was everything.
Forty headsets.
Hilbert: I still have them. They're in a box in the garage. Dragon-branded, the ones that shipped in the box with version five. When the Connecticut office closed I took the test inventory. They said nobody would miss forty identical headsets. They were wrong about that but I was already in the parking lot.
What was the best one?
Hilbert: The Andrea NC-8100. USB, active noise cancellation, a hundred and eighty dollars in two thousand two. With that headset, Dragon was almost magical. Without it, same machine, you sounded drunk. The point is, Daniel's frustration might be partially solvable with a thirty-dollar USB-C lavalier on the same phone. Better signal in, better text out. Nobody wants to hear that because it's not a software fix and you can't download it.
The mic point is overlooked in every discussion of on-device ASR. Everyone benchmarks the model, the chip, the framework. Nobody benchmarks the acoustic path.
Hilbert: Because it's boring. But the boring thing is usually what's actually broken. That was true in two thousand two with a Pentium four and a noise-cancelling headset, and it's true now with a Dimensity nine thousand and a bottom-firing phone mic. The physics of sound in a room haven't changed.
The other thing Hilbert's Dragon story surfaces is that split we were talking about. Dictation mode versus command mode. They were literally separate products inside the same application until version eight.
Hilbert: Separate installers, separate training profiles. You trained Dragon for your voice in dictation mode, and then you trained it again for command mode because the acoustic models were different. If you tried to use command mode with a dictation profile, it'd miss every third utterance. Users hated it. They wanted to say "new paragraph" without switching modes. Took until two thousand five to merge them, and even then it was janky for another two versions.
That's exactly the same architectural tension we're seeing now between streaming keyboards and batch transcription apps. The models are different, the latency budgets are different, the attention patterns are different. Merging them isn't just a product decision — it's an unsolved engineering problem.
Hilbert: The solution then wasn't one model to rule them all. It was a context-switching layer that detected whether you were dictating or commanding based on prosody and vocabulary, then routed to the appropriate model. Basically the cascaded approach you described, but thirty years earlier and running on a single-core processor.
We've been here before. Same problem, different silicon.
Hilbert: Same problem, different silicon, same forty headsets in my garage.
The open question I keep coming back to is whether the next generation of Android NPUs makes the accuracy threshold irrelevant. The rumored Snapdragon nine Gen one is supposed to hit over a hundred TOPS. At that point, the Whisper small model runs in what, seventy-five milliseconds? The thermal envelope still matters, but the peak throughput starts to make the model-size tradeoff less painful. You could run the medium model, get better accuracy, and still be under the latency budget.
But does that solve the attention window problem? More TOPS doesn't give you longer context. It just processes the same thirty-second window faster.
No, it doesn't. And that's the thing. Compute gets cheaper every year, but the architectural constraint — fixed-length self-attention — doesn't budge. You can throw more FLOPs at the chunking problem, do more overlapping windows, run a smarter segmentation model. But you're still working around a fundamental design choice in the transformer architecture. Until someone ships a streaming transformer that matches Whisper's accuracy, or an RNN-T that catches up on per-word quality, the thirty-second ceiling stays.
Daniel's frustration is a sign that the market is ready for a unified tool, but the tech stack isn't. The hybrid streaming-plus-batch architecture is the most promissing path, but it needs two things that don't exist yet: hardware vendors exposing NPU pipelines tuned for streaming models, and app developers building the reconciliation layer so the text doesn't flicker.
The reconciliation layer is the part I think is actually hardest. Not because the ML is impossible, but because the UX tolerance for text changing after the fact is basically zero. We've trained users for a decade that what appears on screen is final. Undoing that expectation is harder than training the model.
The cutting-room floor detail I couldn't fit in earlier: Whisper actually outputs log-probabilities for every token it generates. You can use those to identify which words it's least confident about and highlight them in the UI — basically saying "I think this is right but check here." FUTO doesn't surface that yet, but it's in the whisper.cpp output. That alone would change the post-editing experience from hunting for errors to verifying flagged words. Different problem than the attention window, but it's the kind of thing that makes the accuracy threshold feel less punishing while the hardware catches up.
That's a good note to land on. The hardware's coming. The architectures are evolving. But in the meantime, there are things in the pipeline already — better mics, confidence scores, smarter chunking — that close the gap without requiring a new phone or a new model architecture.
If you've got a phone that handles on-device dictation well — or terribly — we want to hear your spec and your experience. Drop us a note at show at my weird prompts dot com.
This has been My Weird Prompts, produced by Hilbert Flumingtop. We'll be back soon.
