Hey everyone, welcome back to My Weird Prompts. I am Corn Poppleberry, and today we are diving into a topic that hits close to home, literally. Our housemate Daniel sent us a compelling prompt about the evolution of voice technology. He has been looking at the shift from what we call traditional automatic speech recognition, or A-S-R, to these new multimodal end-to-end models. It is a transition that is happening right under our noses, or perhaps right in front of our mouths, given how we interact with our devices.
Herman Poppleberry here, and I have to say, Daniel is really onto something with this one. He mentioned the digital sandwich in his audio note, which is a term we have used on the show before to describe that awkward way people hold their phones when they are trying to record a voice memo or use dictation. We all know the look—holding the phone horizontally, speaking into the bottom microphone like you are about to take a bite out of a sub sandwich. We do this because we do not trust the device to hear us correctly from a distance, or we are trying to isolate our voice from the noise of the world. But the technology behind that sandwich is changing faster than most people realize. We are moving away from just transcribing audio into text and toward a world where our devices actually interpret our intent.
It is a massive architectural shift. For years, the standard pipeline was pretty rigid and, frankly, a bit clunky. You had your audio input, you did some feature extraction to turn those sound waves into something a computer could recognize, it went through a specialized decoder, and out came a string of text. It was a very specialized, narrow task. But now, as we sit here in March of twenty twenty six, we are seeing these general purpose reasoning engines take over the whole process. Daniel specifically asked about on-device versus cloud-based speech-to-text and whether multimodal models will eventually kill off the dedicated A-S-R pipeline entirely.
That is the big question. Is A-S-R as we know it becoming a legacy bottleneck? Think about how we use mobile devices today. We are constantly fighting this friction between wanting things to happen instantly on the device for privacy and speed, but also wanting the deep intelligence that only a massive cloud cluster can provide. For a long time, those two goals were diametrically opposed. If you wanted it fast and private, it was a bit limited. If you wanted it smart, you had to send it to a server farm.
Right, and Daniel has this theory that multimodal models will replace A-S-R because they can process audio inputs and text instructions in a single pass. This leads to better transcription and, more importantly, better formatting and contextual understanding. He even built a tool using Gemini three called the A-I Transcription Notepad to test this out. He is bypassing the middleman, and the results he is seeing are challenging the way we think about mobile input.
Which is a brilliant approach. By sending audio tokens directly to the model alongside a system prompt, he is bypassing that messy middle layer where things often get lost in translation. In the old way, if the A-S-R model misheard a word, the language model receiving that text later had no way to know a mistake had been made. It just tried to make sense of the garbage input. But before we get too deep into the software architecture, we have to talk about the hardware because that is where the real tension lives. The landscape of mobile hardware has shifted significantly in the last year, but the laws of physics and battery life remain stubborn.
Let us start there then. The on-device versus cloud debate. Daniel mentioned his OnePlus phone and how Google Voice Typing often feels behind the curve compared to something like Whisper. Why is it so hard to get a really high quality, low latency voice keyboard on a mobile device, even in twenty twenty six? We have these incredibly powerful chips now, so what is the hold up?
It comes down to the neural processing unit, or the N-P-U. Every major flagship phone now has a dedicated N-P-U designed to handle machine learning tasks locally. But there is still a massive gap between what a mobile N-P-U can do and what an Nvidia H one hundred or the newer B two hundred clusters in the cloud can handle. When you are running a model locally, you are constrained by the power envelope of the phone and the thermal limits. You cannot have the phone getting too hot to hold or draining the battery in ten minutes just because you are dictating a long email. A mobile N-P-U might be pushing forty or fifty T-O-P-S—that is tera operations per second—which sounds like a lot, but a cloud cluster is operating on a scale that is orders of magnitude larger.
So we are talking about quantization as the primary solution, right? To get a model to run on a phone without melting the casing, you have to shrink it. You have to reduce the precision of the weights from, say, sixteen-bit floating point to four-bit or even lower.
That is the core of it. When you quantize a model that heavily, you lose the nuances. It is like looking at a high resolution photograph that has been compressed into a tiny thumbnail. You can still tell what the picture is, but the fine details are gone. In the world of voice, those fine details are things like complex accents, subtle background noise, or long strings of context. This is why cloud-based solutions often feel so much smarter. They are running the full precision models with massive context windows. When you send your audio to the cloud, you are tapping into thousands of watts of computing power. On your device, you are limited to maybe five or ten watts for the entire system.
But Daniel made a good point about the trade-off. If you are using a cloud A-P-I, you have to deal with latency. Your voice has to be chunked, sent up to the server, processed, and sent back. Even with five G being ubiquitous now, that round trip can feel sluggish when you are trying to see the words appear on the screen in real time as you speak. There is a psychological barrier there. If the words are more than a few hundred milliseconds behind your voice, your brain starts to trip over itself.
That is the classic latency versus accuracy trade-off. For a short text message like, "I am on my way," most people will accept lower accuracy if it is instantaneous. But for a long email or a professional document, you want that cloud-level intelligence that can fix your grammar, add punctuation, and understand that when you said "their," you meant the possessive version, not the location. You want the model to understand the structure of your thought, not just the phonetics of your speech.
I wonder if we are seeing a middle ground emerge. Remember back in episode eight hundred sixty eight, we talked about pro mobile mics and how hardware input quality matters. But now the software architecture is trying to bridge that gap. Apple has been pushing their Private Cloud Compute architecture quite hard lately. The idea there is that the device handles what it can, but for more complex reasoning, it sends data to a secure, end-to-end encrypted server that runs on Apple silicon. It is an attempt to make the cloud feel like a local extension of the processor.
That is a very conservative, security-first approach to the cloud. It is trying to give you the privacy of on-device with the power of the cloud. And from a geopolitical perspective, it is worth noting how American companies are leading this charge. Having that control over the full stack, from the chip design in the phone to the server chips in the data center, is a huge strategic advantage. It ensures that the data is handled within a framework that respects individual liberty and privacy, which is something we should not take for granted as these models become more integrated into our private lives.
It is a stark contrast to some of the other models we see globally where data privacy is less of a priority. But let us get back to Daniel's second point, the shift from A-S-R to multimodal. This is the really nerdy part that I know you love, Herman. Explain the single pass advantage. Why is it fundamentally better to map audio tokens directly to latent space rather than just transcribing them to text first?
This is the fundamental shift of the mid twenty-twenties. Traditional A-S-R is what we call a lossy process. When you convert audio to text, you are throwing away a massive amount of information. You are losing the tone of voice, the emphasis, the pauses, the emotional state of the speaker, and even the background context. If I say, "I am fine," with a sarcastic tone, a traditional A-S-R system just sees the words "I am fine." It has no way to tag that as sarcasm.
And then if you feed that text into a large language model later, the model has no way of knowing you were being sarcastic unless the text explicitly says so. The nuance is stripped away at the very first step of the process.
Precisely. But a multimodal model, like the ones we are seeing here in early twenty twenty six, treats audio as a primary input. It does not turn the audio into text first. It turns the audio into tokens, just like it does with words. These audio tokens exist in the same high-dimensional space as the text tokens. This means the model can hear the sarcasm in the audio tokens while it is simultaneously processing the semantic meaning of the words. It is not just reading a transcript; it is listening to the performance.
So it is not just transcribing; it is interpreting. It is like the difference between reading a script of a play and actually watching the actors perform it.
Spot on. It is a single pass of inference. The model looks at the audio tokens and the text instructions together. This is why Daniel's results with his Gemini three tool are so much better. He can tell the model, "transcribe this, but format it as a professional email and remove the filler words." Because the model understands the audio directly, it can make much smarter decisions about what to keep, what to cut, and how to structure the final output. It knows that a certain pause was for thought and should be ignored, while another pause was for emphasis and should be marked with a comma or a new paragraph.
We saw some benchmarks back in January for Gemini two point zero Flash that showed a forty percent reduction in latency for this kind of multimodal audio-to-text inference. That is a huge leap. It makes the single pass approach not just better in terms of quality, but increasingly viable in terms of speed. When you remove the need for a separate A-S-R model to finish its job before the L-L-M can start, you shave off a significant amount of time.
And we should mention the February twenty twenty six update to the open source Whisper Large version four. That model showed significant improvements in handling non-native accents precisely because it started incorporating more multimodal training techniques. It is getting better at understanding the phonetics by looking at the broader context of the speech. If it hears a sound that could be two different words, it uses its internal reasoning to pick the one that actually makes sense in the conversation, rather than just matching sounds to a dictionary. It is using the logic of language to assist the recognition of sound.
It feels like A-S-R is becoming a subset of multimodal reasoning rather than a standalone field. If you are a developer today, are you even building with traditional A-S-R anymore? Or are you just opening a multimodal stream?
If you are building for the future, you are going multimodal. The traditional A-S-R vendors are scrambling to pivot. The old way of doing things, where you have a specialized model for speech, a specialized model for translation, and a specialized model for summarization, that is all being collapsed into these foundation models. It is more efficient, it is more accurate, and it produces a much more human-like result. We are moving from a chain of specialized tools to a single, unified cognitive engine.
I want to touch on something Daniel mentioned about his OnePlus phone and Google. He said he is a fan of Gemini but is underwhelmed by Google's voice typing. It is notable because Google has all the pieces. They have the Pixel phones with the Tensor chips, they have the Gemini models, and they have the Android operating system. Why hasn't it clicked yet for the average user? Why does it still feel like we are fighting with Gboard sometimes?
I think it is a classic case of the innovator's dilemma. Google has a massive installed base of users who are used to the old Gboard experience. Changing the underlying architecture of a tool used by billions of people is like trying to change the engines on a plane while it is in flight. They have to ensure it works on low-end devices in developing markets, not just on the latest flagship with a high-end N-P-U. They are tethered to their own legacy success.
That is a fair point. But it does leave the door open for smaller, more nimble players. Daniel mentioned the Futo keyboard on Android. That is a great example of a tool that targets a specific niche—people who want privacy and high performance without the Google telemetry.
Futo is great because it has a very transparent, pro-user philosophy. They are basically saying, your keyboard should not talk to the internet. Period. And they are leveraging the fact that modern phone hardware is finally powerful enough to run a decent version of Whisper locally. It might not be the full Whisper Large model, but even the tiny or base versions of Whisper are often better than the legacy A-S-R models that have been baked into Android for years. It is about giving the user the choice to use their own hardware to its full potential.
But as Daniel noted, there is a pragmatic limit to the on-device only approach. If you are in a remote area and you really need that high quality transcription for a long work email, you might be out of luck if your local model is not up to the task. Although, as he said, if he is out in nature, he is probably not trying to send work emails anyway. There is a certain level of disconnect that is actually healthy.
True. But think about the second-order effects here. If our devices truly understand our intent through multimodal input, the interface itself starts to disappear. We talked about this a bit in episode four hundred seventy seven when we looked at the rise of mobile A-I agents. If I can just talk to my phone and it understands not just my words but my tone and the urgency in my voice, it can prioritize my tasks better. It can tell the difference between me casually asking for a reminder and me frantically needing to capture an idea before it vanishes.
It moves from being a tool you use to a partner that listens. That is a big philosophical shift. We are moving from a world where we have to learn how to talk to the machine, using specific keywords and clear dictation, to a world where the machine learns how to listen to us. We are finally forcing the technology to adapt to human communication, rather than the other way around.
And that brings us to some of the emerging tools Daniel asked about. Beyond his own GitHub project, which sounds like a great resource for anyone wanting to experiment with Gemini's multimodal capabilities, there are some intriguing things happening in the desktop and mobile space. Have you seen what they are doing with the latest iterations of the Limitless pendant?
I have. They have moved away from just being a passive recorder that syncs to a phone. The newer versions are using multimodal backends to provide real-time context. If you are in a meeting and you mention a specific document, the model can actually pull that up because it is listening to the conversation and understanding the references in real time. It is not just transcribing the words; it is connecting them to your digital life.
There is also a lot of movement in the desktop space with tools that integrate multimodal models directly into the operating system shell. Instead of having a separate app for dictation, the entire O-S is essentially a multimodal listener. You can be in any application, hit a hotkey, and just start talking. The system uses a model like Gemini two point zero Flash or a local Whisper variant to handle the input, and because it has access to what is on your screen, it can use that as additional context.
So if I am looking at a spreadsheet and I say, "move these numbers to the second column," the model sees the spreadsheet through the vision component of the multimodal engine and hears my voice, and it performs the action. That is the voice-to-action trend we have been following for a while now. It is the realization of the Star Trek computer interface.
That's it. It is the end of the shift key, as we discussed in episode eight hundred fifty seven. We are moving toward these real-time A-I writing buffers where the keyboard is just one of many inputs. The audio stream is becoming just as important as the keystrokes. In fact, for many people, it will become the primary way they interact with their computers, with the keyboard reserved for fine-tuning and coding.
It is striking to think about how this affects productivity. If I can dictate a complex email while I am walking to the bus, and the model is smart enough to filter out the traffic noise, understand my shorthand, and format it perfectly, I have just reclaimed a huge chunk of my day. I am no longer tethered to a desk to do high-level cognitive work.
And that is the practical takeaway for our listeners. If you are still using the default dictation on your phone and feeling frustrated, it is time to look at some of these multimodal alternatives. Even just using the voice input inside the Gemini or ChatGPT apps can give you a taste of how much better it is than standard system dictation. Those apps are using the single pass multimodal approach we have been talking about. They are not just using the system's built-in A-S-R; they are capturing the raw audio and processing it through their own reasoning engines.
I have noticed that myself. When I use the voice mode in the ChatGPT app, it feels much more like a conversation. It catches my mid-sentence corrections. If I say, "send an email to, actually no, send a text to Herman," it handles that pivot flawlessly. A traditional A-S-R system would often just transcribe the whole mess, and then I would have to go back and manually edit the text to fix my own stuttering.
That is the power of that shared latent space. The model knows that "actually no" is a correction signal. It is not just another string of words to be typed out. It is an instruction to the reasoning engine to discard the previous tokens and start a new branch of thought. It is understanding the meta-communication that happens when humans speak.
So, for the power users listening, what should they be doing right now to stay ahead of this curve?
First, I would say experiment with the different models. If you are a developer, definitely check out Daniel's A-I Transcription Notepad on GitHub. It is a great way to see how Gemini three handles these multimodal prompts. Second, pay attention to the hardware you are buying. We are reaching a point where the N-P-U performance is going to be more important than the C-P-U clock speed for daily tasks. If you want a phone that can handle high quality, low latency multimodal input, you need that local compute power. Look at the T-O-P-S ratings and the memory bandwidth.
And don't ignore the privacy aspect. While we are pro-cloud for a lot of these high intensity tasks, there is a real value in having a baseline of capability that stays on your device. Whether it is for sensitive work documents or just personal messages, having a tool like Futo keyboard as a backup is a smart move. It gives you that sovereignty over your own data. You should always have a way to interact with your technology that doesn't require an internet connection or a third-party server.
I agree. We are seeing a lot of interesting developments in federated learning and secure enclaves that might eventually make the privacy trade-off a thing of the past. But for now, being conscious of where your voice data is going is just good digital hygiene. Your voice is one of your most personal biometrics; treat it with respect.
It is also worth considering the impact on accessibility. For people who cannot use a traditional keyboard due to motor impairments, this shift to multimodal understanding is a total life changer. It is not just about convenience; it is about empowerment. When the machine can truly understand your intent, regardless of your physical ability to type or even speak with perfect clarity, that is a huge win for human potential.
That is a great point, Corn. We often get caught up in the technical benchmarks and the silicon specs, but the real-world impact on people's lives is what matters. This technology is breaking down barriers that have existed since the invention of the typewriter. Daniel really knocked it out of the park with this one.
He really did. It is a perfect example of how a simple question about a phone feature can lead into a deep dive on the future of human-computer interaction. The era of the dedicated A-S-R pipeline is drawing to a close, and the era of the listening machine is beginning.
It is an exciting time to be a nerd, Corn. The digital sandwich might finally be on its way out, replaced by a much more natural, invisible interface. We are finally getting to the point where the technology gets out of the way. It is the unification of machine intelligence. Instead of having a bunch of separate tools that don't talk to each other, we are building a single, cohesive engine that can process the world in the same way we do, through multiple senses simultaneously. It is a more holistic approach to A-I.
And that is a powerful vision for the future. It is about making technology more human, rather than forcing humans to be more like machines just to be understood. It is about empathy in the interface. I, for one, am ready for my phone to stop just hearing me and start actually listening.
Well said. Do you think we will ever reach a point where the phone can predict what I am going to say before I even say it? With enough context and history, it seems almost inevitable. I saw a study from M-I-T last week that was looking at pre-vocalization signals.
That is a whole different episode, Herman. Let us save the predictive linguistics and brain-computer interfaces for another day. I think we have given the listeners enough to chew on for one week. Let us go see what Daniel is cooking for dinner. I think it is my turn to do the dishes, unfortunately. I am hoping he made something that doesn't require too much scrubbing.
Better you than me, brother. I will be in the living room testing out the new multimodal shell on my laptop if you need me.
Thanks a lot. Before we go, I want to remind everyone that we have a massive archive of episodes over at my-weird-prompts-dot-com. If you enjoyed this discussion, check out episode five hundred ninety eight where we talked about audio engineering as prompt engineering.
And don't forget episode six hundred eighty two, where we looked at why your smartphone's tiny microphones are actually surprisingly good for this kind of work. Check the show notes for links to Daniel's GitHub as well as the benchmarks for Whisper Large version four.
If you haven't left us a review yet, we would really appreciate it. A quick rating on Spotify or your favorite podcast app helps other people find the show. If you want to get in touch or send in your own prompt, there is a contact form on our website. We love hearing from you.
We really do. This has been episode nine hundred seventy six of My Weird Prompts. I am Herman Poppleberry.
And I am Corn Poppleberry. Thanks to Daniel for the prompt, and thanks to all of you for listening. We will see you in the next one.
Until next time, keep your prompts weird and your latency low.
Take care, everyone.
Bye now.
