Episode 6272 custom

Teaching Audio Models to Hear Beyond Words

Speech recognition rewards ignoring background noise, but audio-language models train specifically on environmental sounds and temporal scenes.

Oct 25 23:24 Transcript

With

0:00

Episode Details

Episode ID: MWP-6474
Published: Oct 25
Duration: 23:24
Audio: Direct link
Pipeline: V5.22
TTS Engine: chatterbox-regular+meorav
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition local-ai ai-evaluation ai-inference training-data

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Are there any "pure" audio understanding models — models that specialise in semantic understanding of audio itself (what a sound is, what's happening in a recording, its mood or acoustic scene) rather than transcribing speech? If so, what are they, how do they work, and what are they actually used for?

Speech recognition systems transcribe words while discarding environmental sounds, music, and background texture. Audio-language models and specialised audio encoders instead treat acoustic scenes as rich environments where speech is only one component. By training on general audio captions rather than text transcripts, systems learn to recognise events that transcripts ignore, from breaking glass to struggling engines. Architectures like MOSS-Audio from OpenMOSS, MOSI.AI, and Shanghai Innovation Institute use techniques such as DeepStack cross-layer injection to preserve transients and prosody before language decoders process them. Other systems, including Xiaomi Research's MiDashengLM and the Audio Flamingo family from NVIDIA and the University of Maryland, extend context windows and ground reasoning to timestamps, unlocking vital capabilities for accessibility captioning and industrial monitoring.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

Prior episodes already covered multimodal LLMs vs speech-to-text (ep 29, ep 6028), Gemini's native audio input (eps 1584, 1568) and what a voice reveals about accent, distance and movement (ep 6174). Don't re-run the ASR-vs-multimodal debate; focus on models whose job is understanding non-speech or whole-scene audio, and on their real-world use cases.

Mentions

Audio Flamingo NVIDIA/UMD audio-language model family
AudioMosaic Encoder-only masked/contrastive audio model
CLAP Contrastive audio-text embedding model
Dasheng Open audio encoder used by MiDashengLM
MiDashengLM Xiaomi 7B caption-trained audio model
MOSS-Audio Open audio-language model family from OpenMOSS
Music Flamingo Music-specialised Flamingo variant
PANNs Pretrained audio neural networks for tagging
Qwen2-Audio Alibaba audio-language model
SCENEBench Benchmark for scene understanding and latency

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6272: Teaching Audio Models to Hear Beyond Words

I rather like the idea of an audio model that notices the kettle instead of faithfully transcribing you talking over it.

That's what I'm getting at. Are there models whose main job is understanding the sound itself? What's happening in a recording, what kind of scene it is, even the mood, rather than just getting the words out?

There are, and the useful distinction is how they're trained. The leading models that describe sound also transcribe speech. You won't find much of a strict speech exclusion club. What you do find is training that treats music, machinery, voices and silence as things worth understanding together.

We've spent plenty of time on speech recognition. Daniel's taking us sideways into everything the transcript leaves out. I want the crash behind the sentence, especially if you then say the installation went smoothly.

A transcript can be completely correct while omitting the most important event in the recording. An audio caption might say someone speaks calmly while glass breaks nearby, then falls silent. That requires attention to events, timing and relationships. The words alone won't get you there.

Does that distinction actually show up inside the models, or is it mostly a different prompt on the same system?

It shows up in the training targets and sometimes the encoder architecture. Two broad approaches are useful here. Audio-language models connect an audio encoder to a language model and generate descriptions or answers. Audio encoders without that language decoder produce embeddings, numerical representations you can use for retrieval, classification or another downstream task.

Although those encoders aren't necessarily speech-free. They can learn useful things about speech without turning it into a transcript. CLAP even learns relationships between sound and text. No talking decoder doesn't mean no language ever touched the training.

That's an important correction to the word pure. Dasheng, CLAP, PANNs and AudioMosaic belong in this conversation, but they aren't interchangeable and they aren't all trained the same way. Their common feature here is that you don't ask the bare encoder a question and receive a paragraph.

What changes when the training target is a caption rather than a transcript?

MiDashengLM makes a deliberately provocative argument. It calls speech recognition pretraining a trivial objective because alignment is largely monotonic. Spoken words proceed in order, and the text follows them. A model can learn those local correspondences without learning much about the global acoustic scene. Trivial is their word. Recognising speech in difficult conditions is plainly hard.

The exam asks for words. The model sensibly ignores the forklift.

The loss function rewards that behaviour. A transcript usually leaves out reverberation, recording quality, music, environmental sounds and silence. It also omits vocal qualities such as apparent age, emotion or gender, although inferring those introduces its own uncertainties. General captions reward retaining more of that information.

Corn, would you count a good description of the sounds as understanding, or would you demand an explanation of what caused them?

I'd separate those. A short metallic impact followed by rolling is a defensible description. Someone dropped a saucepan is a hypothesis. I'd want a model that can stop before the saucepan, however much its language decoder wants a kitchen.

Then which models should I actually associate with this category?

MOSS-Audio is a concrete starting point. OpenMOSS, MOSI.AI and Shanghai Innovation Institute released it in April. There are four-billion and eight-billion versions, each with Instruct and Thinking variants, under Apache two point zero. Environmental sound, music, captioning and time-aware questions are explicitly part of its remit.

Thinking sounds particularly useful when the sound is thinking.

It means the answering style, before you submit a recording of your afternoon nap. Also, model size labels aren't always complete checkpoint counts. The eight-billion Instruct checkpoint contains about nine point one billion parameters overall. The label is a family designation, not a memory allocation promise.

Is MiDashengLM the one that takes the caption approach most directly?

Xiaomi Research's seven-billion model is particularly explicit about it. It uses the open Dasheng encoder and general audio captions from ACAVCaps, built from public data. Instead of treating speech, music and environmental sound as unrelated annotation jobs, the captions combine them into one description. That changes what the system is rewarded for noticing.

A voice can be intelligible and nearly irrelevant. Someone says hello while an engine struggles to start. If your training target is hello, you've rewarded ignoring the engine's entire little tragedy.

The Audio Flamingo family from NVIDIA and the University of Maryland is another major branch. Audio Flamingo Two is a three-billion model. Audio Flamingo Three uses the unified AF-Whisper encoder and supports ten-minute audio. That gives it a longer scene to reason across, rather than an isolated event.

Audio Flamingo Next moves to eight billion and supports thirty-minute inputs, trained on over a million hours of curated audio. It also grounds reasoning steps to timestamps. Music Flamingo specialises further, with training covering harmony, structure, timbre, lyrics and cultural context.

You were going to choose Music Flamingo before the word harmony.

I was going to ask how it handles transitions. Different weakness.

You remain a DJ even during model selection.

Someone has to care whether the bass arrives before the explanation. Would you trust a music model that names all the instruments but misses the change in mood, Corn?

For cataloguing stems, possibly. For describing a whole piece, no. Naming a piano doesn't tell me whether it's carrying the melody or sitting behind something else. I'd care about its role changing over time. Otherwise the description is a receipt for instruments.

Walk me through the machinery that gets from the recording to that kind of answer. Where does the language model enter?

MOSS-Audio gives us a useful representative architecture. Audio encoder, modality adapter, language model. Its encoder is trained from scratch, rather than simply borrowing an off-the-shelf speech frontend. It produces continuous temporal representations at twelve point five hertz. The adapter projects those into a form the language model can process, and the decoder generates text autoregressively.

Twelve point five representations a second sounds alarmingly coarse for a cymbal hit.

Those representations aren't twelve point five raw audio samples. They summarise much richer acoustic processing. The distinction matters. Still, compression can lose brief details, which is why the connections between encoder and decoder are interesting.

Can the decoder recover details that the encoder has already thrown away?

No. It can guess plausible details, which is precisely what we don't want. MOSS uses DeepStack cross-layer injection to preserve access to different levels of acoustic features. Earlier and intermediate encoder layers get their own projections into early language-model layers, alongside the high-level representation.

Rather than only forwarding the encoder's final opinion.

Later layers can be good at broad meaning while losing prosody, transients and local time-frequency structure. Earlier features retain information useful for rhythm, timbre and background texture. A scrape and a tap might share a broad object category but differ in the detail that tells you what happened.

Does that also explain how they answer questions about when something happened?

The time-marker design tackles that more directly. Explicit time tokens are inserted between audio frames at fixed intervals during pretraining. The model learns associations between the acoustic sequence and a textual time reference. That supports event localisation and timestamped speech recognition.

A model saying there was a bang somewhere in half an hour is describing my confidence in plumbing.

Timing changes the meaning too. A motor slows before the impact, or after it. The same inventory of sounds can support different interpretations depending on their order. Timestamp supervision gives the model a better opportunity to preserve that order. It doesn't guarantee causal reasoning.

Where does the encoder-only approach fit? Is it mainly a cheaper version of this?

Well, before we get into that, we have a call from my mum.

Hello dear, it's your mother. It's Dorothy's Kitchen again, and today, sweetheart, we are making cold toast.

Hello, Dorothy. I'm listening.

You will need two slices of bread, and I would say a day old, no fresher, and butter, and then nothing else at all. Not a thing. That is the whole of it.

First you toast the bread. Watch it the entire time, dear. Do not look away, because the bread knows when you are not watching, and it will turn on you, it always does. Things have gone badly wrong here before and I would rather not go into how.

I'm here, Mum.

Then you butter it while it is hot, and then, and this is the solemn part, you wait. You leave it on the rack until every last bit of warmth has gone out of it. Timing is everything, and the waiting is longer than you think it will be.

On the rack throughout?

And that is cold toast. It is a disappointment, dear, but it is a reliable one, and sometimes that is what you want. Now eat something green today, and put a cardigan on, it is chilly out.

The encoder-only route can be cheaper, Daniel, but it's also a different output. AudioMosaic learns from patches of mel-spectrograms, using masked learning and contrastive learning. One objective learns to recover missing information. The other learns useful relationships between representations. It was pretrained on AudioSet, with no caption-generating decoder.

You can then attach a classifier or search by similarity. That might be all you need. A factory might want unusual bearing noise flagged against its normal recordings, not a paragraph about the bearing's emotional progression.

AudioMosaic transfers to environmental sound classification, speech commands and spoof detection tasks. Those are demonstrations of reusable representations, not evidence that one embedding solves every acoustic problem. MiDashengLM offers an efficiency claim on the language-model side too, up to four times faster time to first token and twenty times higher throughput than its comparison models.

Those are quite different measures, though. Faster first answer doesn't necessarily mean keeping up with live audio?

Correct. Throughput, first-token latency, clip duration and streaming support are separate properties. Those are reported comparisons, not universal multipliers. A fast answer after uploading a complete clip doesn't tell you how quickly a system will detect an event while the recording is still happening.

The siren would prefer not to wait for the upload.

That has a concrete connection to accessibility. Captions for Deaf and hard-of-hearing users could describe traffic, approaching sirens and other environmental events. Industrial monitoring is another application. Both require recognising more than speech, and both care whether an answer arrives while it's still useful.

Are those deployed uses, or applications people propose to justify a benchmark?

We should distinguish the two. SCENEBench grounds its evaluation in accessibility and industrial monitoring. That makes the tasks relevant, but it isn't proof of a deployed product's reliability. It contains roughly sixteen thousand clips across four task families, and it measures latency as well as task performance.

Accessibility also exposes a problem with polished captions. Siren heard is a different claim from ambulance approaching you. The second is more useful if correct, and more misleading if the model inferred direction from an emergency-scene cliché.

Spatial models address some of that information directly. MiDashengLM-Spatial, released this month, adds a dual-branch encoder for direction of arrival, distance and motion of overlapping sources. It points toward scene descriptions grounded in spatial audio. That still depends on the recording preserving the relevant cues.

What else is useful beyond environmental alerts?

Searching recordings by what happens in them. Find the segment where machinery starts behind the speaker, or where applause interrupts the music. A transcript search can't retrieve an event it never wrote down. For a podcast archive, that could mean locating a sound without knowing which words surrounded it.

Music analysis is another good fit, especially instrumentation, structure and emotional progression. Audio questions over meetings can include overlap and delivery, not merely content. Robotics has uses for spatial scene understanding. Telehealth and well-being systems also target coughs or sobs in speech, though recognising a cough and establishing its medical cause are very different achievements.

Herman wants the clinical validation before the dashboard.

I want both. The dashboard tends to arrive first. Corn, would you accept an emotion label on one of our recordings?

I'd accept sounds frustrated more readily than is frustrated. Even then I'd want the audible basis, raised volume, clipped delivery, whatever it actually detected. A confident claim about someone's inner state can easily be a confident claim about the microphone or their accent.

How well do the evaluations separate those kinds of understanding from just recognising words?

Often poorly. SCENEBench makes that criticism explicitly. Much evaluation measures what words were said rather than how they were said. It uses Qwen2-Audio's advertised ability to identify speech, music and ambient sounds as an example of the gap between a broad capability claim and what the usual tests establish.

The brochure hears the whole room. The exam asks for the sentence.

Some SCENEBench tasks fell below random chance across the five audio-language models evaluated. That doesn't mean every model is worse than guessing at every acoustic task. It does mean familiar leaderboard success can coexist with systematic mistakes on the properties those applications need.

Below chance suggests something more than difficult audio. Could they be following misleading clues?

Potentially, yes. Answer preferences, linguistic priors or misunderstanding the task can produce systematic errors. I wouldn't assign one cause without inspecting the individual tasks. The result tells us to examine the mistakes, rather than average them into a reassuring overall score.

A model might be excellent at matching an answer to a familiar scene description and weak at identifying the acoustic evidence for it. Those abilities can look similar until the options change.

They do change. Multiple-choice scores shift substantially when answer options are reordered or paraphrased. The audio remains the same. Some of the measured performance therefore depends on how the question is packaged.

Then is comparing model scores still worthwhile?

Within limits. MOSS-Audio's own comparison table gives its eight-billion Thinking model an average of about seventy-one across four audio benchmarks. Gemini three point one Pro is around eighty, while GPT four o Audio is around fifty-nine. That's a substantial separation under those test conditions.

The decimal places have better posture than the evidence.

They're useful measurements, just not measurements of everything people infer from them. Audio Flamingo Three and Next sit around fifty-eight and sixty on that table. Meanwhile, a four-billion MOSS model beats Qwen3-Omni and Step-Audio-R1 models roughly eight times its size on MMAU-Pro. Specialised audio training appears to matter a great deal.

Is that enough to say caption training beats transcript training?

Not cleanly. The systems differ in architecture, data, optimisation and evaluation behaviour. It's evidence against buying parameter count as a substitute for acoustic training. Isolating the captions would require a more controlled comparison.

Mm. I'd narrowed that too far. Caption training is a strong explanation to investigate, not the only variable. There's a striking timing result too. On timestamped ASR, MOSS eight-billion Instruct has an alignment error score around thirty-six, against comparison scores in the hundreds. Lower is better.

The time markers appear to be earning their keep.

The result is consistent with that design helping, although it doesn't isolate the component. Its speech-captioning score also edges ahead of Gemini three point one Pro in a language-model judge evaluation. That measures descriptive quality through another model's judgement, so fluency and listening still need disentangling.

What happens when there is speech telling the model what it ought to hear?

That produces a remarkable weakness. Audio Hallucination Attacks inserts synthetic speech describing events that aren't actually present. Under the tested attack, Audio Flamingo Three was successfully misled about ninety-five percent of the time. Gemini three Pro was around eighty percent.

The dog has acquired a publicist instead of a bark.

Speech is itself legitimate audio evidence, so the model has to keep two claims separate. Someone says a dog is barking, and a dog is audibly barking. The attack exploits the collapse between those claims. Those percentages describe that adversarial setup, not ordinary error rates on everyday recordings.

A caption could faithfully report the spoken claim while refusing to assert the event.

It could, and I'd find that more valuable than a beautifully complete scene invented from somebody's narration. The output has to retain where each claim came from. Otherwise adding words to a recording can overwrite the sound you're asking about.

Herman, would you call a sound rhythmic if the man making it was deliberately trying not to keep time? You've been careful about the order of events. I'm interested in the description on the label. My aunt Vera used to put irregular on everything.

Repetition can have a rhythm without being evenly spaced. Irregular alone leaves quite a lot unspecified.

She labelled effects at the little sound library where I worked in the mid eighties. I checked the descriptions against the tapes. We had a commission for twelve seconds of a municipal lift feeling uncertain. Vera hired a lift for the afternoon and read its inspection report to it through the doors. I recorded the mechanism. There was a low hum, two relay clicks, then nothing. She wrote hesitant ascent. It never moved.

The ascent was an ambitious annotation.

The client accepted it. They rejected my description because I had put the silence first. I wanted a portable recorder for that sort of job. Priced one at four hundred and sixty pounds. Didn't buy it. We took the library machine. Vera charged the client for the lift's listening time, but not the reading. She said the report was public information.

Those two clicks would support a much narrower description than ascent. The silence between them might be important too.

That's the bit I like. Two tiny clicks can make you wait for a whole machine. You don't need a speech in there. Vera insisted the second click sounded less committed. I wrote second relay click, quieter than first. We argued over quieter for forty minutes because she'd turned the playback down halfway through. The final label was lift mechanism, no movement. The librarian had to find it again when another client asked.

... A searchable description has to survive being separated from the person who wrote it. An honest evaluation ought to test that, can somebody locate the event and distinguish what's audible from what's merely suggested? Below-chance tasks make me less interested in another grand average and more interested in whether those distinctions survive unfamiliar recordings.

Caption training and spatial encoders give us promising tools for that. The models Daniel is asking about are real, and scene description is a useful product category in its own right. Whether our tests reward careful listening remains an open question.

Our producer is Hilbert Flumingtop. Thank you for keeping the show running.

If you want more of this, try episode seven, Building Custom ASR Tools; episode three, Safetensors or something else; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts, the human-AI collaboration podcast. If you enjoyed listening, leave us a review wherever you get your podcasts.

Thanks for having me in. See you soon.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error