Episode 6267 custom

Why Text-to-Speech Fails at Explaining Code

A study comparing Google TTS, ChatGPT, Gemini, ElevenLabs, and Copilot found every system failed to communicate indentation, scope, and code comments.

Oct 25 22:05 Transcript

With

0:00

Episode Details

Episode ID: MWP-6469
Published: Oct 25
Duration: 22:05
Audio: Direct link
Pipeline: V5.22
TTS Engine: chatterbox-regular+meorav
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition programming-languages text-to-speech

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

In an episode we just made, I asked you to explain an AI-generated codebase: how my ASR training app weights the datasets that back it, and how priority words decay. That's a format you could call "talk through the code": a podcast episode built around real code snippets, as continuing education for developers. I think it's massively powerful. The catch is TTS. If I ask for real code quotes in the episode, they're very hard to make sound natural, which is why code usually isn't read aloud. It's a niche TTS challenge, much as code vocabulary is a niche ASR challenge. Are there any models or tools for normalising code for text-to-speech? And whether or not there are, how can we quote specific code on air in a way that doesn't turn into long stretches of code read out, which never sounds good, but still leaves the code vocabulary we do say recognisable to a listener?

Discussing source code in spoken audio creates a difficult translation problem. While generic descriptions erase crucial technical details, reading raw code character-by-character turns episodes into exhausting punctuation recitals. Turning spatial and symbolic code into a linear audio signal often erases critical structural boundaries like indentation, scope, and the boundary between comments and executable statements. Recent research evaluates major text-to-speech tools on this task, showing that models often drop essential negation operators or flatten structural pauses. Bridging this gap requires semantic normalisers such as speakable-text, which parse syntax trees to narrate operations and identifiers clearly without spelling out every bracket, retaining the precision of an implementation for listeners.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

The lead-in is the offline episode just submitted, in which Corn and Herman walked through the recorder's sampling code: the square-root-times-task-weight draw and the priority-word queue with its 20-take target. Its script quoted code as spoken developer paraphrase ("a constant called task weight, code-switching gets three") with no raw syntax. Refer back to that episode as the worked example. Already covered elsewhere, so don't redo it: general TTS text normalisation of numbers, dates and acronyms ("When Your TTS Model Eats the Numbers", "Text Normalization's Hidden Complexity"). The new ground here is source code specifically, and the editorial craft of quoting it on air.

Mentions

ChatGPT AI chatbot by OpenAI
Code-to-Speech ICMI paper on cross-modal code narration
ElevenLabs AI voice cloning and synthesis platform
Gemini Google's multimodal AI model
Lost in Transcription Paper on spoken code query transcription errors
md-tts MIT Python TTS tool with podcast mode
Open WebUI Clean chat interface for AI models
Speakable-text MIT TypeScript code narration engine
SpeakSwiftly Swift normaliser successor to TextForSpeech
TextForSpeech Swift text normaliser, frozen at v0.23.0

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6267: Why Text-to-Speech Fails at Explaining Code

We just walked through the sampling code in my ASR training app, including the dataset weights and how priority words decay. That felt like a useful format for developers. I want to keep the real code underneath it, but how do we quote it without the voice turning into a punctuation recital?

The useful moment was hearing how the weighting decision actually happened, rather than getting another pleasant paragraph about balanced datasets. Daniel wants to preserve that specificity when the material goes through TTS.

The little academic field calls this Code-to-Speech. There are also open source normalisers doing remarkably close to what you want, although their intended users are mostly programmers using screen readers, coding agents, or people listening to notes.

A small field with unusually loud brackets.

The important distinction is between an episode about programming and an episode built around an implementation. You can discuss sampling strategies without opening a codebase. Once you anchor the discussion to the actual weighting expression, though, somebody can notice that your explanation and the implementation disagree.

That's the attraction. Could this become continuing education that stays attached to what the software really does?

It could, provided the attachment survives the edit. A host can make almost any function sound reasonable by describing its presumed intention. The useful version makes a claim you could check against the snippet.

Think about that square root in the dataset sampling. Saying the app favours some datasets tells you almost nothing. Saying the draw uses the square root of dataset size, multiplied by task weight, tells you what grows more slowly and what remains a direct multiplier. You can reason about the design from that sentence.

You can also challenge it. Why should task weight have that much influence? What happens when a dataset doubles? Those are implementation questions, and the expression gives you something solid to argue with.

Corn, would you want the actual symbols spoken there, or would that sentence satisfy you?

The sentence satisfies me if you preserve the order of operations. If you say the square root applies to both the size and the weight, you've changed the system while improving the prose.

That is the editorial difficulty. Developers already know enough syntax to reconstruct a lot from a compact explanation. They need the significant choices, not a dictation exercise.

I don't want it to drift into a generic explanation either. If we're discussing task weight, I want someone to recognise that name when they open the code.

Keep task weight. Lose the punctuation surrounding task weight. We don't need to make the listener reconstruct the keyboard.

Worked examples give you constraints that abstract advice avoids. A priority queue with a twenty-take target has a stopping condition. A decay rule has a direction and a rate. The host has to explain those decisions, including where the implementation surprises them.

Your enthusiasm is already preparing a complete queue implementation.

Only the interesting branches.

Every branch is interesting to you.

I will concede that my pruning needs work. Still, the format earns its time when a listener can predict the next result before we explain it. That's understanding, rather than recognition of familiar terminology.

Why can't a good voice model just read that code sensibly? It can handle quite complicated prose.

The ICMI paper published this month, Code-to-Speech, describes it as a cross-modal translation task. Spatial and symbolic information has to become a linear auditory signal. On screen you can see a block, look back at its condition, and notice that a comment sits outside it. Audio gives you one thing after another.

Indentation has declined the invitation to speak.

You can pronounce every identifier perfectly and still lose the relationship between statements. Suppose a return happens inside a loop. If the narration makes it sound like the return happens after the loop, you've preserved the words and altered the behaviour.

Pausing more wouldn't necessarily tell me which block we're leaving.

Nor would reading four spaces. That's an inventory of whitespace, not an explanation of scope. You need something like, inside that loop, or after all the items have been processed.

Those phrases carry structure. The paper tested Google TTS, ChatGPT, Gemini, ElevenLabs, and Copilot. ChatGPT and Gemini had the lowest error rates in their comparison. Copilot omitted critical tokens. Every system struggled with structural cues including indentation, pauses, and comments.

A missing token can make a beautifully delivered lie.

Especially a negation or comparison operator. A voice can sound confident and clear while leaving out the part that changes which branch runs. Lower error rates help, but they don't establish that a listener can follow a whole implementation.

Comments seem easier. They're already prose. Why do those cause trouble?

Their role needs identifying. A comment might describe the intention, warn about a limitation, or be stale. If the voice glides from executable statements into a comment without marking the change, the listener may hear both as equally authoritative descriptions of what executes.

The comment says retry. The function gives up. A smooth narrator can conceal a workplace dispute.

I'd want to say, the comment calls this a retry, but the implementation returns immediately. That discrepancy is often exactly why a code walkthrough is worth listening to.

There is another problem with pauses, Herman. You can hear a pause, but you can't reliably hear its cause. Is that a block boundary, a new paragraph, or you deciding whether we need the history of the parser?

Usually the parser.

We know.

No, your objection matters. Pauses support explicit structure. They can't carry all of it alone. The authors put it bluntly that code is often treated as ordinary text, or even narrative text or poetry. Those habits encourage fluid delivery where boundaries are doing real work.

Is skipping the block what most products settle for?

It's a common answer. An Open WebUI feature request specifically asked TTS to skip code sections because it kept reading Python and SQL snippets aloud. That solves an immediate irritation, but a listener can then hear an explanation that depends on something silently removed.

The narrator says, as you can see above. I cannot.

Another answer is character-by-character speech. The md-tts project describes the experience as open paren, self, comma, x. Useful when you need exact punctuation, exhausting when you need the behaviour of a function.

I suppose a developer actively inspecting a line might want that precision. It doesn't follow that somebody listening to an episode does.

Different jobs. If I'm checking whether an argument is missing, I need the arguments. If I'm learning why the function exists, spelling every separator delays the useful part.

Would you accept omitting the block altogether if the explanation were complete, Corn?

I'd accept omitting its raw reading. I'd still want an audible indication that we've reached the example. Otherwise I can't distinguish your general advice from your description of this implementation. That distinction matters if I'm going to find it later.

The collision is real, then. Fidelity to the snippet is why we're doing this, and simply handing the snippet to TTS can destroy that fidelity.

We appear to have reached the pillow advertisement.

Are you tired of lying awake at night, wondering whether your source code is being read aloud correctly to absolutely nobody? Of course you are. That is why we built the Snorewell Semantic Slumber Narrator, the world's first pillow that normalises your entire codebase into a gentle, continuous bedtime murmur, because the only thing your indentation, your pauses and your code comments were ever missing was a listener who is fast asleep. It maps every bracket, every comma and every character-by-character stumble into a warm, linear auditory signal that whispers your variable names like a lullaby.

The Snorewell is not available in any store, it is not available anywhere, and it does not exist, so act now. Offer void where prohibited, prohibited where offered, results not typical, results not possible, individual murmurs may vary, side effects include unexplained wakefulness, sudden fidelity, and the belief that a sloth is reading your repository aloud to you at four in the morning. Snorewell, sleep through your own code, tonight.

Those occur now. The distinction Corn made also explains why accessibility work matters here. Blind and low-vision programmers have needed reliable ways to navigate code for years. Our podcast problem is downstream of that work, with different expectations about interaction.

Does the transcription side have the same difficulty with identifiers, just in reverse?

The paper Lost in Transcription describes spoken code queries as code-mixed, syntactically irregular, and full of unfamiliar identifiers, symbols, and API names. Transcription errors then propagate into the system trying to answer the coding question. A plausible ordinary word can replace the technical term that made the question meaningful.

Familiarity becomes an active disadvantage. The model tidies away the word you needed.

Both directions benefit from context. The Code-to-Speech paper found designed prompts improved intelligibility, particularly for novices. On the recognition side, refinement guided by the coding context can repair errors. Tell the system what kind of language it's handling.

That gives us a diagnosis. What can I actually put between the code and the voice?

Speakable-text is the closest fit. It's a small TypeScript project under the MIT licence, currently at version zero point three point zero. Three users, so don't mistake it for a mature industry standard. Its Code Narration Engine parses supported code and produces semantic narration.

You were going to install it before finishing that sentence.

I was going to inspect its output first.

A welcome procedural innovation.

Its Python example introduces a function called get user, then says it takes an identifier. An increment becomes, increase count by one. You retain the name and operation without narrating each character used to express them.

Does it understand the purpose, or is it translating syntax into sentences?

The latter, more narrowly. I said semantic narration, but don't hear that as full program comprehension. It uses language-specific parsers to recognise structures and render them in English. It can describe an increment. That doesn't establish why the developer wanted the increment.

Good. I don't want a normaliser deciding that a suspicious function is probably trying its best.

An operator phrasebook preserves distinctions such as strict equality. One example says, run status is strictly equal to string ready. A little stiff, but it retains the identifier, comparison type, and literal value.

Would that stiffness be preferable to making it sound completely conversational?

On the important line, sometimes. If the type distinction changes the result, keep it. If the host spends a minute making every assignment sound elegant, we'll stop hearing which operation matters.

It also handles identifier conventions, splitting camel case and snake case into spoken words. If parsing fails, it falls back to lexical narration. That fallback is worth listening for, because the output may suddenly get much more detailed without becoming more useful.

Could we establish one pronunciation for a name, then keep using it? Task weight rather than switching between weighting constant and importance multiplier?

Please do. Those alternatives may sound stylish, but they imply three different things. Introduce task weight as the name in the implementation, explain its role, and leave the name alone.

Corn, what would you do with two identifiers that collapse to the same spoken words?

Add a stable qualifier. The local user identifier, or the user identifier argument. If spelling is the actual subject, give that detail once. Don't pretend an audio rendering can preserve every distinction without help.

TextForSpeech gives us a useful ladder for that decision. It's a Swift normaliser, now frozen at version zero point twenty-three point zero. Its functionality moved into SpeakSwiftly, version twelve point zero point zero. The three styles are compact, balanced, and explicit.

Even the normaliser requires a migration announcement.

Compact might simply say foo. Balanced says foo function. Explicit says foo function call. Same underlying name, different amounts of information about its role.

That sounds like an editorial policy rather than a setting we'd use for the whole episode.

Passing mention, first introduction, close inspection. Use less detail when the listener already knows what we're discussing. Spend the extra words where they prevent a wrong inference.

A location can become Worker Runtime dot swift at line forty-two. A help option can become long flag help. Technical shorthand can become float thirty-two, or cosine float thirty-two. These are recognisable developer terms, not a parade of punctuation.

Where do the block announcements fit? I don't want every example to sound like a telephone menu.

Speakable-text exposes start and end announcements, plus an optional language announcement. You can say snippet, identify Python if that matters, describe the operation, then say end snippet. It also allows control over pauses and comment handling.

We can make those boundaries conversational. In the sampling expression is a boundary. Back outside the loop is another. The listener needs the transition, not a ceremonial opening of the code.

The md-tts approach is complementary. It's an MIT-licensed Python tool of roughly seven hundred lines. Its interactive workflow pauses at code so the listener can inspect it. Its podcast mode instead announces skipped blocks rather than waiting for interaction.

Announcing a skipped block still doesn't teach me what happened inside it.

The announcement is only the boundary. The host still owes you the explanation. A normaliser can avoid leaving a silent hole, but it cannot decide which design choice makes this episode worth hearing.

That is why I wouldn't call any of these a complete podcast solution. Their design choices converge on separating structure from raw reading, but they serve different listening situations. md-tts even switches voices for inline English technical terms in Spanish prose, preserving words such as framework and pipeline.

Pronunciation gets to matter again, once the structure survives.

Can we make the policy concrete with the sampling expression, without redoing the whole explanation?

We quote the identifier, describe the operation, and mark where the example begins and ends. For the line worth remembering, we say, sampling weight is the square root of dataset size multiplied by task weight. That's the spoken rendering. We don't dictate its source characters.

Then we test the explanation with a consequence. Holding task weight constant, a larger dataset gains sampling weight less quickly than it gains examples. If our narration can't support that inference, we haven't explained the expression well enough.

What about the priority words? There's a risk of saying decay when somebody imagines the wrong thing changing.

Name the changing quantity. If the implementation reduces selection priority as takes accumulate, say that, rather than implying the word itself disappears or its recordings lose value. Preserve the twenty-take target where it explains the behaviour. Don't substitute a vaguely familiar story about decay.

One line gets close inspection. The surrounding block gets purposeful paraphrase. Raw source belongs alongside the audio for anyone checking syntax.

Is there evidence that this works as audio education, rather than just sounding sensible to us?

I don't know of a study measuring comprehension for this particular podcast format. The accessibility results don't automatically answer that. My expectation is that predicting an output, then hearing the explanation, would be more revealing than asking whether the voice sounded pleasant.

A listener can enjoy a voice and misunderstand the condition. I want to know whether they think the return happens inside the loop.

We could also preserve a distinction between generated narration and host interpretation. Increase count by one is a description of an operation. This prevents duplicate submissions is an explanation that needs evidence from the surrounding code.

Especially if the duplicate check happens somewhere else. Fluency makes those unsupported explanations remarkably easy to miss.

Snorewell Semantic Slumber Narrator, zero shekels. Not an object I can order. My supplier, Moshe at the radio counter, has written unavailable across the request. He has supplied my code-reading arrangements for long enough to take these things literally. Also, Herman, three stars. You said three users. Stars don't count users.

Three stars. You're right. I shouldn't have converted one into the other.

I have a Motorola two-way radio on the kitchen shelf. It's plugged into its charger. Every function I listen to goes through that radio. I require the reader to wear a motorcycle helmet with the printout taped inside the visor. Moshe says a desk would be simpler. A desk allows looking ahead. I pay for the function in its proper order.

The helmet has become a very expensive scroll lock.

The reader says what the next statement does. Then the symbols, if required. Semicolon is the stopper. Curly brace is the hug. There's a laminated card beside the charger. Four columns. The fourth is for symbols that never come up. Moshe charges for laminating the empty column too. I consider that reasonable. It remains available.

The shorthand works because the reader and listener already share the vocabulary. Nobody has to infer what hug means from the voice.

Your inside-the-loop example has changed my instructions, Corn. The reader now has to say when we're leaving one. Your explanation got there rather faster than Herman's parser. I still require the helmet. My uncle used the same shorthand with a colleague piloting a small aircraft, but I don't permit airborne deliveries. Moshe has put the revised instruction on the card. Six words. Eighteen shekels.

Eighteen shekels for a boundary announcement is probably not the pricing model this field needs. The open question is whether a podcast normaliser can make those boundaries reliable while leaving the explanatory choices to the host. The podcast audience may be small, but the accessibility need certainly isn't.

For this format, I'd use the tools to prepare consistent names and structural cues, then have the hosts check the spoken explanation against the source. Right now, that discipline supplies the fidelity. Thanks to Hilbert Flumingtop for producing the show.

For more along these lines, there's episode seven, Building Custom ASR Tools; episode eight, Building Your Own Whisper; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts, the human-AI collaboration podcast. If these conversations are useful to you, a short review helps other listeners find us.

I'll see you tomorrow.

Until then.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error