I was thinking the other day about how much time we spend as a species just translating things from one format to another. It is like we are all stuck in this constant state of administrative overhead. For the last decade, if you wanted a computer to understand a voice, you had to pay what I call the transcription tax. You take the audio, you crunch it into text, you lose all the soul and the tone, and then you hand that flat text to a model and hope it can guess what the person actually meant. But looking at the latest data and the releases coming out of Google today, on March twenty-sixth, twenty twenty-six, it feels like that tax is finally being repealed.
It is not just being repealed, Corn. It is being abolished entirely. We are moving into a world where the intermediate step of text is becoming optional, or in some cases, a hindrance. Today's prompt from Daniel is about an evaluation he just ran on Google's Gemini one point five Flash, and it really highlights this shift from what we call cascaded pipelines to native multimodality. Daniel ran forty-nine different analytical tasks against a single twenty-minute audio file, and the results are honestly a bit of a wake-up call for anyone still building on the old speech-to-text stack.
It is interesting that he is focusing on the one point five Flash version. We have seen the release of the Gemini three series recently, including the three point one Flash Lite preview that dropped just a few weeks ago, and the Live version that came out today. But one point five Flash has remained this foundational benchmark because it was the first one to really nail that high-speed, long-context audio task at scale. Before we get into Daniel's specific Irish-accented brain dump of a test, explain the architectural difference here for the listeners. When we say native multimodality versus a cascaded system, what is actually happening under the hood that makes it different from, say, sticking OpenAI's Whisper in front of GPT-four?
The cascaded system is like having a translator who only knows how to write down words but cannot hear the music. Whisper or any traditional speech-to-text engine listens to the audio and outputs a string of text. That text is all the large language model ever sees. So if the speaker is crying, or being sarcastic, or if there is a loud siren in the background, the model usually has no idea unless the transcriber adds a little tag like "loud noise" or "sobbing." Native multimodality means the model, in this case Gemini, is actually processing the raw audio waveforms directly. It sees the audio as a series of tokens just like it sees words. It hears the pitch, the cadence, the background hum of a refrigerator, and the specific timbre of a human voice all at once. It is not reading a script of the movie; it is actually watching the movie. Or at least listening to it with both ears.
That is a great way to put it. And Daniel's test was pretty rigorous. He took a twenty-minute recording of himself talking about everything from rocket sirens in Jerusalem to voice cloning and deepfakes, and then he hit the model with questions ranging from emotion detection to forensic audio analysis. What I find wild is that he used the Flash version, which is essentially the high-efficiency version of the Gemini family, and it still managed to navigate this massive context window. We are talking about a one-million-token context window. Herman, put that in perspective for us. How much audio can you actually fit into a million tokens?
It is roughly nine and a half hours of audio in a single prompt. To give you the technical breakdown, Gemini represents audio at a rate of thirty-two tokens per second. That is about one thousand nine hundred and twenty tokens per minute. When you have a million tokens to play with, you can feed it an entire workday of meetings in one go and ask it who sounded the most annoyed during the three p.m. sync. Or, to use a more scientific example, you could feed it a five-hour field recording from a rainforest. A cascaded system would try to transcribe the "chatter" of the animals into text, which is useless. Gemini, because it hears the waveform, can actually identify specific bird species based on the frequency and pattern of their calls. It is a completely different level of data ingestion.
I would pay good money to have an AI tell me exactly who is being passive-aggressive in my emails, so having it do that for audio sounds like a productivity dream or a social nightmare. But let's look at the benchmarks Daniel was referencing. In the technical reports, Gemini one point five Flash was hitting ninety-nine point one percent recall in what they call the Audio Haystack test. Explain what that test actually looks like in practice.
The Audio Haystack test is the ultimate stress test for long-term memory. You take hours and hours of audio—the "haystack"—and you bury a specific spoken phrase or a unique sound—the "needle"—somewhere in the middle. Then you ask the AI to find it and tell you exactly what was said or what happened at that timestamp. A combined Whisper and GPT-four pipeline only hit ninety-four point five percent recall. That five percent gap might not sound huge, but when you are searching through ten thousand hours of call center logs or legal depositions, that is a lot of missed needles. The reason Gemini wins here is that it does not have to rely on a perfect transcription. If the "needle" was whispered or obscured by static, the text-based model might just see a gap in the transcript. Gemini hears the faint signal in the waveform itself.
It is the difference between searching a text document for the word "bang" and having a guard who actually hears a gunshot in a noisy warehouse. But how does a "smaller" model like Flash achieve this? Usually, when we talk about AI, bigger is better. We expect the massive Pro models to be the ones with the best memory. How is Flash hitting these ninety-nine percent recall numbers?
That comes down to two major technical pillars: online distillation and the Mixture of Experts architecture, or MoE. Demis Hassabis, the CEO of Google DeepMind, has been very vocal about how they built Flash. It is not just a truncated version of Gemini Pro. It is "online distilled," which means during its training, it was essentially a student watching a teacher. It was trained to mimic the reasoning and multimodal understanding of the much larger Pro model but within a more efficient parameter set. Then you have the MoE structure, which Jeff Dean has championed. Instead of the whole model firing for every audio token, it routes the data to specific "expert" layers. If it hears speech, it activates the speech experts. If it hears background noise, it routes to the environmental experts. This allows it to maintain that one-million-token window without the computational cost exploding.
So it is like a hospital where you have a general triage desk that sends you to a specialist immediately, rather than one doctor trying to do everything. That explains the speed, but what about the quality? Daniel's report mentioned that the API downsamples the audio to sixteen kilobits per second and collapses it into mono. As an audio guy, that sounds like a nightmare. Sixteen kilobits is basically "early two-thousands internet radio" quality. Does that not mess with the model's ability to do this forensic work?
You would think so, but the model is surprisingly resilient. It is trained on "noisy" data—phone calls, compressed YouTube videos, old archival recordings. It has learned to look past the artifacts of compression to find the underlying signal. It is more interested in the "shape" of the sound than the perfect reproduction of every frequency. In Daniel's test, he asked the model to perform a speaker profile. Even with that sixteen kilobits per second compression, it correctly identified his accent, his age range, and even his technical background based on the terminology he used. But more impressively, it could distinguish between his natural voice and the segments where he was playing back AI-generated voice clones of himself.
That is the forensic part that gets interesting. If the model can hear the artifacts of a voice clone that a human ear might miss, we are looking at a built-in deepfake detector. But I want to poke at the methodology for a second. Daniel's test used a single Irish-accented voice. As we know from every voice-activated elevator joke ever made, Irish accents can be a notorious "black box" for traditional speech-to-text. Did the native multimodality help with the accuracy there, or does it still struggle with the lilt and the speed?
From what we see in the outputs, the native approach is significantly more robust. Because the model is not trying to force the audio into a phoneme-to-text map before understanding the meaning, it can use the context of the entire waveform to resolve ambiguities. If a word is mumbled or obscured by a siren—which Daniel actually had in his recording because he was recording in a conflict zone—the model can use the surrounding audio context to infer the word, rather than just outputting "unintelligible." It is also worth noting that Daniel's evaluation covered thirteen different categories. We are talking about speaker demographics, health and wellness inferences, and even audio engineering tasks like identifying the frequency balance of the recording.
Wait, it can do EQ analysis? Like, it can tell you if your recording is too heavy on the low end or if you have a "muddy" mid-range?
It can. It can describe the acoustic environment. In Daniel's test, it correctly identified that he was in a small residential room. It picked up on the lack of reverb and the specific noise floor. This is where the "weird" in "My Weird Prompts" really shines. Most people think of an AI assistant as something you talk to, but we are moving toward a world where the AI is an expert observer of the audio itself. It is an audio engineer, a forensic linguist, and a sentiment analyst all rolled into one. And it is doing all of this for fifteen cents per million tokens.
Fifteen cents. That is the part that blows my mind. We are talking about sixteen times cheaper than the Pro variant. At that price point, the economic barrier to "audio mining" just evaporates. You could have a system that listens to every single customer service call in real-time, not just to transcribe them, but to flag when a customer's voice indicates they are about to cancel their subscription based on the tension in their vocal cords, even if their words are polite. That is a massive shift for business.
It is a total paradigm shift. Up until now, companies only analyzed a tiny fraction of their audio data because it was too expensive to transcribe and then process. Now, you can just dump the raw audio into Flash. This leads us to the bigger landscape shift we are seeing here in March twenty twenty-six. The move toward Audio-to-Audio, or A2A. We have been living in this world of sub-optimal latency where you speak, the computer thinks for two seconds while it transcribes and processes, and then it speaks back. Google's release of the Gemini three point one Flash Live preview today is aimed squarely at killing that latency. They are pushing for sub-five hundred millisecond response times.
That is faster than the human brain's natural conversational gap in some cases. If you get below half a second, the "uncanny valley" of AI interaction starts to disappear. It stops feeling like a walkie-talkie conversation with a robot on Mars and starts feeling like a real-time exchange. But this also brings up the privacy elephant in the room. Google's "Personal Intelligence" feature that launched last month allows these models to access your entire Workspace—your Drive, your Gmail, and now, presumably, your recorded meetings and voice notes. If the model is this good at "hearing" things, it is hearing a lot more than just your grocery list.
It is hearing the background of your life. If you have a recording of a family dinner in your Google Photos, a native multimodal model could potentially identify that your uncle sounds like he has a persistent cough that might be worth checking out, or that your dishwasher is making a sound that indicates a failing pump. The health and wellness category in Daniel's report touched on this. He asked the model to infer the speaker's physical state. It picked up on his breathing patterns and vocal clarity. While that is incredible for personalized health monitoring, it is also a massive amount of data to hand over to a single entity. We are moving from "Google knows what you search for" to "Google knows how you breathe."
It is the ultimate "read the room" technology, but the room is your entire life. Now, let's talk about how this fits into the broader AI ecosystem. You mentioned Gemini Embedding two earlier. How does that change the way we actually use this audio data?
This is a massive piece of the puzzle that launched on March tenth. Gemini Embedding two is the first truly multimodal embedding model. In the past, if you wanted to do a search across your data, you had text embeddings for your documents and maybe some separate audio embeddings for your sounds. They did not speak the same language. Embedding two maps audio, video, and text into the exact same vector space. That means you can search your audio files using a text query, or vice versa, and the model understands the semantic relationship across modes. If you search for "sounds of tension," it can find a video of a heated argument, an audio recording of a violin string about to snap, and a text document about a geopolitical crisis. It is all one unified understanding.
That is a bit terrifying but also incredibly powerful for organization. Imagine searching your entire life's recording history for "that time I sounded really happy" and having it actually work. But I want to go back to Daniel's specific test. He's originally from Ireland, living in Jerusalem. He's got this mix of technical jargon and casual speech. One of the categories he tested was "Language Learning." How does a native audio model help with something like learning a dialect compared to a text-based one?
It is all about the prosody and the phonetics. A text-based AI can tell you how to spell a word and what it means, but it cannot tell you if your mouth is shaped correctly to produce the right sound. A native audio model like Gemini can listen to your attempt at a word and compare the waveform of your speech to a native speaker's waveform. It can say, "You are putting the emphasis on the second syllable instead of the first," or "Your vowel sound is too flat." It is the difference between reading a book about how to play the piano and having a teacher listen to you play and say, "You are hitting that key too hard." It is the difference between information and coaching.
And when you combine that with the sub-five hundred millisecond latency we are seeing with the Live API, you basically have a real-time dialect coach in your ear. But Herman, what are the limitations? We talked about the sixteen kilobits per second downsampling. Is there anything else developers should be wary of?
The main thing is that while it is great at speech and general environmental sounds, it is not a musicologist yet. Google has a separate model called Lyria for high-fidelity music generation and analysis. If you try to use Gemini Flash to analyze the subtle harmonic distortion of a tube amplifier, you are going to be disappointed. It is optimized for "semantic" audio—sounds that carry meaning or information. Also, while the one-million-token window is huge, the model can still suffer from "middle-of-the-document" loss if you are not careful with how you prompt it. You still need to be specific about what you are looking for in those nine hours of audio.
So, if I am a developer listening to this, and I am currently using a pipeline where I send audio to Whisper, get a JSON file back, and then send that to an LLM, your advice is basically... stop doing that?
Precisely. That is the core takeaway. If you are only interested in the words spoken, the old way is fine—it is cheap and reliable. But if you want to build anything that feels "intelligent" or "empathetic," you are leaving eighty percent of the data on the table by using a text-only intermediate step. You are losing the emotion, the sarcasm, the ambient context, and the speaker's physical state. Using something like Gemini one point five Flash or the new three point one Lite allows you to query the audio directly. You can ask, "At what point did the speaker sound most confused?" or "Was there any background noise that suggested they were outdoors?" You cannot do that with a text transcript.
It is like trying to describe a painting to a blind person and then asking them to critique the brushwork. You are losing the primary source material. And that brings us to the "Audio Haystack" again. If you can only remember ten minutes of audio, you do not have a haystack; you have a small pile of grass. Being able to cross-reference something said at minute five with a tone shift at minute fifty-five is where the real insights happen.
And that is exactly what Daniel's evaluation proved. He was asking questions that required the model to synthesize information from across the entire twenty-minute recording. It was not just "what did he say at the end?" It was "how did his attitude toward AI change throughout the recording?" That requires the model to maintain a coherent understanding of his emotional state over time. It is a level of temporal awareness that we have never seen in audio processing before. It is almost like the model is developing a sense of "personality" for the speaker.
It is building a mental model of the human on the other end. Which, again, is both cool and a little bit like a sci-fi horror movie depending on how you look at it. But let's look at the competition. We are focusing on Gemini because that is what Daniel's prompt was about, but how does this stack up against the other players? Is OpenAI or Anthropic doing this native waveform stuff as well?
OpenAI's GPT-four-o was the first big move in this direction, but Google has really pulled ahead in terms of the context window and the cost efficiency with the Flash series. Anthropic's Claude four point five, which we are expecting later this year, is rumored to have native audio, but right now, Gemini is the only one giving developers this level of long-form audio access for pennies. The one million token window is the real differentiator. Most other models cap out much earlier, which means you cannot feed them an entire hour-long podcast, let alone a nine-hour recording of a conference.
So, to wrap this up, the "transcription tax" is essentially dead. We are moving into the A2A era. What are the three things our listeners should take away from Daniel's experiment?
Number one: Stop building pipelines that rely on intermediate text transcription if you need more than just the words. You are throwing away the most valuable parts of the audio. Number two: Leverage these long-context windows for what we call "audio mining." Instead of just summarizing a meeting, ask the model to identify the moments of highest tension, or to flag when someone's tone contradicts their words. And number three: Audit your current stack. If you are still using Whisper for everything, you are likely losing a massive amount of non-verbal data that could be making your product better. The economic barrier is gone—fifteen cents per million tokens means you can afford to be curious.
It is like we have been looking at the world in black and white and someone just handed us a color camera. Sure, the black and white photos were functional, but you missed the fact that the sky was blue and the grass was green. Moving to native multimodality is that color shift for audio. I can finally have an AI tell me exactly how many times I say "um" and "uh" in this podcast and then give me a sentiment score on my own jokes.
It would probably tell you that your jokes are "consistent," Corn. Which is a nice way of saying they are all equally cheesy.
I will take "consistent." It is better than "unintelligible." But in all seriousness, the future of the audio interface is here. We are moving away from the "chatbot in a box" and toward an "omni-modal" assistant that truly hears us. The race to sub-five hundred millisecond latency is the final frontier. Once we hit that, the distinction between talking to a human over a digital line and talking to an AI will become almost purely philosophical.
And that is a world we need to start preparing for, both in terms of how we build and how we protect our privacy. The tools are here, the pricing is right, and the models are getting smarter by the day. Daniel's evaluation of Gemini one point five Flash is a perfect snapshot of where we are right now: on the cusp of audio finally becoming a first-class citizen in the world of AI.
Well, I for one am ready for my AI sloth-to-human translator. If it can make me sound as smart as you think you are, Herman, it will be worth every cent of that fifteen-cent-per-million-token price tag.
I think even Gemini has its limits, Corn.
Ouch. I felt the sentiment on that one without any help from a multimodal model. Alright, I think we have covered the depth of Daniel's audio experiment. It is a fascinating look at the "under the hood" mechanics of how these things are actually hearing us.
It really is. And big thanks to Daniel for running these tests. It is one thing to read a Google white paper; it is another to see how it handles a real-world Irish brain dump in the middle of a conflict zone. It grounds the tech in reality.
And that is a wrap for this deep dive. Thanks as always to our producer, Hilbert Flumingtop, for keeping the gears turning behind the scenes. And a big thanks to Modal for providing the GPU credits that power the research and generation of this show. If you want to keep up with these episodes, search for My Weird Prompts on Telegram to get notified the second a new one drops.
This has been My Weird Prompts. We will catch you in the next one.
Stay weird. Bye.
