Episode 6365 custom

Teaching Speech Models to Transcribe Bilingual Slang

Fine-tuning Parakeet with 77 minutes of speech cut Hebrew-English code-switching errors nearly in half, but later gains required far more audio.

Oct 30 22:56 Transcript

With

0:00

Episode Details

Episode ID: MWP-6567
Published: Oct 30
Duration: 22:56
Audio: Direct link
Pipeline: V5.25
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition ai-evaluation training-data fine-tuning text-to-speech

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

As we've talked about in recent episodes, over the course of the past couple of weeks I've been working hard on gathering training data from my fine-tuned Nvidia Parakeet for enhanced accuracy and voice recognition. As I've also mentioned, reading hours of sentences does get a bit mind-numbing over time, but I'm keeping motivated because this is a long-term project and I can actually see that the model is getting better. Every time I record a few more hours of audio and do a new fine-tune, the WER is decreasing, but I also see the accuracy getting better. I'm training on two things: one of them is nish vocab like tech vocab and code-switching English Hebrew, and the other is stuff like number recognition, in which I'll record single The beer cost$4.50, and with enough sentences like that, we're hoping to render that with a dollar sign and with a decimal place in the right place, which Raw parakeet doesn't do. I'd been thinking that if I could find a way to integrate my training objectives into sentences that were actually interesting it would be much easier to read them. Or maybe, as a wild idea, we could even create a book or something educational that actually leveraged the sentences, so while I'm sitting here reading them, I'd actually be learning something at the same time. The thing about AI training and gathering training data like this is that volume really does make a difference in my experience. More interesting.

Fine-tuning Parakeet for personal dictation highlights the friction between brief GPU training times and grueling human data capture. The effort focuses on speech mixing English and Hebrew, pushing recorded prompts to Hugging Face under Read Aloud Voice Sentences. Because raw Parakeet v2 lacks Hebrew characters in its tokenizer, the model is trained to output consistent romanized spellings for words like Bituach Leumi. Early audio drastically cut code-switching word error rates and boosted seen-term accuracy, while also establishing a generalized spelling habit for unseen words. However, diminishing returns led to stricter release thresholds. Meanwhile, formatting numbers and currency remains an inverse text normalization challenge rather than a speech recognition issue, better solved through synthetic text pairs, NVIDIA's Thutmose Tagger, or Dual-Form ASR.

Sources

What the research for this episode read before the script was written. Primary sources first.

Mentions

Common Voice Mozilla crowdsourced multilingual speech dataset
Dual-Form ASR Paired spoken/written supervision for normalization
Hugging Face Platform for AI models and datasets
inverse text normalization Converting spoken form to written form
librispeech Standard English ASR benchmark dataset
NVIDIA Thutmose Tagger Token-tagging inverse text normalization approach
OnePlus Android phone used for recording
Parakeet ASR model for dictation transcription
Sabich Daniel's code-switching model release version
Whisper OpenAI's speech-to-text model

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6365: Teaching Speech Models to Transcribe Bilingual Slang

Twenty minutes of training after months of talking feels personally insulting.

Daniel has been recording sentences into his phone to fine-tune Parakeet for the vocabulary he actually dictates, especially tech language and English mixed with Hebrew. The model keeps improving, which keeps him going, but the reading itself is getting mind-numbing. He wants to know whether those training sentences could carry something worth learning, perhaps even become a book, rather than merely give his mouth another shift.

The GPU finishes its work while the person supplying the data is still somewhere in the middle of a very long commitment. That's an unusual engineering constraint. Faster training barely touches it.

His other target is written numbers. He says a beer costs four dollars fifty, and wants the transcript to produce the currency symbol and decimal amount. That sounds adjacent to teaching it Bituach Leumi, but the solution may live somewhere quite different.

It does. The code-switching results already tell us something concrete about how recording time pays back. The currency target needs a separate test. We shouldn't let a falling word error rate quietly claim credit for a decimal point nobody has benchmarked.

The decimal has hired a publicist.

His capture loop is deliberately plain. A phone web app shows a sentence, he reads it, listens back, accepts the take or records it again, then moves on. It follows the Common Voice pattern. Accepted takes get pushed in batches to a public Hugging Face dataset called Read Aloud Voice Sentences, with an English Hebrew Codeswitching Sentences subset.

That playback step matters. People imagine gathering speech means talking while doing something else. Here you're performing, checking the performance, and deciding whether the mistake was yours or the recording's. It's more like editing than conversation.

The bank contains about seven thousand seven hundred prompts, usually built around one or two Hebrew terms. The selection leans toward words stock Parakeet mishandles. He's concentrating the scarce human effort on known weaknesses rather than reading whatever happens to be nearby.

Before you propose an automatic curriculum scheduler, Herman, you're going to fall in love with the selection mechanism and forget somebody still has to say every sentence.

I was going to propose grouping them by difficulty.

There it is.

Grouping could help the person too. Although... yes, I've already started optimizing the queue instead of the afternoon.

The actual training takes roughly twenty minutes on one GPU. One version took nineteen, another twenty-three. Neither changes the production schedule much. The limiting resource is accepted speech, including the time spent making recordings that don't survive.

A GPU cannot clear its throat for him.

Would you find the visible progress enough to keep recording, Corn? Even if the prompts stayed dull?

Enough to start. Not enough to sustain an indefinite project. A new release is satisfying; another sentence about submitting a form is work. Daniel has a practical advantage here: he dictates extensively, so he experiences the benefit between recording sessions. A corrected word later in the day is a better motivator than a dashboard congratulating him.

That connects the collection loop to actual use. The benchmark measures the improvement, but fewer repairs in daily dictation make it tangible. Still, we need to look at what those improvements contain.

The starting point is stranger than an accent mismatch. Raw Parakeet version two has no Hebrew characters in its tokenizer, so when it hears makolet it's essentially blind as a bat. Apologies to the bats.
Gloria: Standards and practices clarification. A listener complaint has been filed regarding the phrase about the tokenizer having no Hebrew characters and being, quote, blind as a bat. We remind hosts that bats are not blind, and that implying otherwise constitutes a technical defamation of a protected flying mammal under policy four four seven one. This is the fourth clarification I have issued today.

The available output characters constrain the fix. Replacing the tokenizer would reinitialize the decoder and demand far more training data than Daniel has. Instead, he teaches a consistent romanized spelling. Bituach Leumi stays Bituach Leumi, rather than wandering into Bituah Leumi whenever the model feels adventurous.

The grocery becomes Micollette in the stock output. It sounds considerably more expensive.

That's its response to makolet. It finds a plausible English-looking sequence for an unfamiliar sound. Before fine-tuning, it got twenty-eight percent of the Hebrew term occurrences right across more than eleven hundred occurrences.

Those correct answers can conceal quite a lot. If it recognizes Shabbat and Bamba but can't handle the words you use to arrange childcare or contact an office, the average flatters its usefulness.

The baseline vocabulary audit makes that visible. There were a hundred and sixty-five terms it always recognized, including names and words already familiar in English. There were three hundred and ninety it never got right. Everyday expressions such as Misrad Hapnim, tzaharon, rofe mishpacha, Motzei Shabbat, and Pikud Haoref.

It can transcribe the snack and lose the appointment.

Word error rate averages across all the words. Lots of ordinary English can be perfect while the Hebrew term carrying the purpose of the sentence is wrong. That's why term recall belongs alongside it.

Then volume can improve two different things: how often it spells a familiar term correctly, and whether it can attempt a term it hasn't been taught. I'd care more about the second, though I suppose the first pays the bills.

Both matter, and the early results separate them. On the older evaluation split, stock code-switching word error rate was ten point nine percent. With seventy-seven minutes of training audio, it fell to six percent.

Seventy-seven minutes bought a very good first impression.

Increasing the audio to a hundred and twenty minutes brought it to five point eight percent. At a hundred and ninety-six minutes, it reached five point three. Improvement continued, but at a much smaller rate.

That's the part I'd find psychologically difficult. You remember the first jump, then expect another hour to repeat it. The machine hasn't promised that; your memory has.

A newer evaluation split gives a separate comparison. Stock scored eleven point three percent, the third proof of concept scored six percent, and the first stable release reached five point one.

Separate comparison is doing important work there. We can't splice the most flattering points from different tests into one magnificent graph.

Correct. Well... comparable checkpoints within each split, rather than one continuous curve. The latest Sabich version two point two reports four point nine one percent code-switching word error rate, with about seventy percent term recall. Its code-switched training audio totals roughly four hundred and seventy-one minutes.

The first seventy-seven minutes bought more than the next several hundred. That doesn't make the later hours wasted. It makes them expensive in a different currency.

Daniel's roadmap describes gains as growing roughly with the logarithm of the data. That's a useful reading of the observed pattern, not a fitted law guaranteeing the next result. What would persuade you to do those later hours?

Errors concentrated in words I say constantly. A tiny average gain could remove an annoyance I encounter every morning. A larger gain on vocabulary I never use could leave my actual experience unchanged. I'd want the benchmark, but I'd also keep track of which repairs disappear.

That's why the vocabulary weighting is sensible. On terms present in training, accuracy rose from twenty-nine to sixty-six percent across the early checkpoints. Familiar examples were being learned much more reliably.

Useful memorization, then. Nobody complains their phone has memorized the right spelling of their name.

On terms excluded from training, accuracy jumped from twenty-seven to forty-seven percent early on. That indicates a transferable mapping between his Hebrew pronunciation and the romanized output, rather than only a list of stored words.

The model has acquired a spelling habit. A fairly personal one.

Daniel puts it that way too: it learned how his Hebrew maps to the romanization, not only the training words. Later held-out gains were much less convincing. Between the third proof of concept and the first stable release, the held-out improvement was about two percentage points, within noise, while the gain on seen terms was over nine points.

You have your Hebrew-etymology face on. I predict you'll want to explain why one spelling deserves to win.

I would happily discuss it, but consistency wins here even when the transliteration offends me.

A painful professional concession.

Deeply. The practical response to the smaller gains is a new release trigger: collect one and a half times the previous release's code-switched audio, rounded up to the hour, before publishing another.

That prevents twenty minutes of training from becoming an excuse to announce every afternoon's recording.

The next threshold is seven hundred and twenty minutes, or twelve hours. At the current pace of eleven to sixteen minutes of code-switched audio daily, the forecast is mid-to-late November. Volume matters, yes. The later hour has to work harder to justify itself.

Currency formatting may not deserve another hour of speech at all. Hearing four dollars fifty correctly and writing the amount in financial notation are separate achievements.

The name is inverse text normalization, ITN. It converts a spoken-form transcript into conventional written form. Raw Parakeet doesn't perform that conversion for Daniel's example. More familiarity with Hebrew vocabulary won't fix it.

Nor will an educational chapter make the decimal less ambiguous. Four fifty might be a price, a time, or a pair of numbers. The surrounding sentence has to carry the decision.

NVIDIA's Thutmose Tagger approaches this through token tagging, selecting transformations rather than freely generating a replacement transcript. That constrains the opportunity to invent content. You want the amount reformatted, not a language model helpfully renegotiating the beer.

Four dollars fifty is already enough negotiation.

Numeric training pairs can also be generated from text. Work on that approach reported an overall ITN accuracy improvement of about fourteen percent. That doesn't predict Daniel's result, but it demonstrates that additional useful supervision needn't require an additional human recording.

I'd be slightly annoyed if my decimal collection turned out to be a text-generation job.

Would you still want formatting built into the recognizer, or would you prefer a separate stage you can inspect?

Separate, initially. I'd want to see whether it heard the right amount before checking whether it formatted it correctly. Otherwise, one wrong result leaves me guessing which component failed. Later, if a joint system worked better, I'd happily stop caring about that distinction.

Dual-Form ASR explores the joint route with paired spoken and written supervision. It puts extra training pressure on normalization-sensitive spans. Its evaluation also distinguishes places where normalization is required from places where it must be forbidden.

Version numbers would enjoy that last category. Tech dictation contains enough dots without every one becoming money.

Right... a product identifier shouldn't be beautified into an amount. The dual-form approach gives Daniel's idea a close technical relative, but his published code-switching results don't yet include a currency benchmark. We have evidence for the Hebrew improvement, not for the dollar sign.

Then the educational material should avoid becoming seven thousand excuses to buy beer. I'd build short chapters around things he actually wants to understand: how municipal services work, a piece of computing history, the mechanics of a health appointment. Each passage teaches something, and the target vocabulary earns its place.

You'd favor connected passages over independent sentences?

Connected ideas, independently understandable clips. If the app shows only the next sentence, that sentence still needs to make sense. No opening with “therefore it did so” because the explanation was three screens ago. The reader should get continuity without the dataset needing chapter context.

That preserves the existing pipeline. The bank already has targeted vocabulary and selection weights, so there's a rudimentary curriculum. Rewriting and sequencing could add intellectual continuity without replacing the capture app.

I'd also vary the rhetorical shape. A definition, an example, a disputed claim, an explanation of why a rule exists. Reading endless declarative sentences makes even interesting material sound like inventory.

Common Voice supplies the adjacent collection model, but its priorities are speech coverage and volunteer contribution, not teaching the reader. I don't know an established pedagogically sequenced corpus designed for this exact double purpose. The learning benefit is plausible, not demonstrated.

Reading aloud can also occupy enough attention that you retain surprisingly little. I'd rather learn one coherent mechanism over ten clips than recite ten unrelated facts. The prose has to carry me, not administer an exam while I'm checking microphone levels.

You would defend the book over a new interface.

The book rarely asks me to grant microphone permission twice.

Fair criticism of my installations. Although your preferred paperback would struggle with uploading the takes.

It could supervise.

There is a real constraint on making everything connected, though. Repeated contexts might let the model rely too heavily on recurring phrases. The interesting version still needs varied sentence structures and uses of each term.

Different contexts across the book, then. Teach Bituach Leumi in an explanation of institutions, revisit it in a discussion of forms or employment. Not the identical sentence wearing a different chapter heading.

The recording distribution also limits what the project can claim. There are three thousand three hundred and thirty-one recordings, all Daniel. Almost all were made on his OnePlus phone, with only a small laptop contribution.

More beautifully written sentences won't produce another speaker.

There are no external microphone recordings and no female voices. Outdoor speech is scarce; noisy-room speech is scarcer. This can be an excellent personal dictation adaptation without proving it works for English speakers across Israel. Cross-speaker transfer remains untested.

I'd keep those ambitions separate. Personal usefulness is a complete achievement. Recruiting other speakers becomes another project, with different reasons for people to keep showing up. An educational corpus might help recruitment, but we can't assume his interests are everybody's reward.

His capture app is exploring multiple users. Meanwhile, another constraint already consumes recording capacity: retaining English. Fine-tuning only on code-switched speech can damage performance on ordinary English.

You teach it the unfamiliar words and it sends the familiar ones home.

One Whisper-small experiment saw English word error rate move from three point four to thirteen point three percent after code-switched fine-tuning. That's not a forecast for Parakeet. It shows why checking the original capability matters.

Daniel's mixture is roughly seventy percent code-switching, twenty percent his own monolingual English, and ten percent LibriSpeech replay. The replay transcripts preserve casing and punctuation. Even a modest share of original-domain examples can sharply reduce forgetting.

Which means not every educational chapter can be engineered around Hebrew terms. Some plain English belongs in the collection because keeping a capability also costs examples.

His publication gate checks both his own English and LibriSpeech against stock. Neither is allowed to worsen, and so far they haven't. That protects against celebrating a Hebrew gain paid for by an English regression.

The boredom is doing real damage if it limits accepted recordings. Sentence design deserves engineering attention for that reason, not merely because a pleasant interface would be nice.

Corn, would you count a sentence as new if the council changed only the reference number? I'm arranging a recording session this week, and that's what they've sent me. You were talking about repeated contexts. I've got repeated invoices.

A new reference number is new training material. It's very little new reading.

That's what I thought. Yael Ben-Ami at the council records desk is helping me dictate the descriptions for a batch of scanned documents. A correction has taken three visits because the room number was copied wrongly. She's answering the counter bell between every batch. She once cleared seventeen callers before her tea cooled. I counted. I've started arriving with hers in a flask.

She's doing the checking as well as receiving the documents. That would interrupt any recording rhythm.

She has my sentence log in the bottom drawer at her desk. The recorder it started with has stopped working. The log hasn't. I write down every complete sentence I read into a device. I don't read one twice. Changing a reference number doesn't qualify. This week I'll record fresh descriptions on the phone, then bring her the audio. I've already paid for the cable she needs. I won't read customer satisfaction slogans. The council can find another mouth for those. Herman, you separated hearing the amount from formatting it. Her screen shows the amount correctly. She still has to listen through each description to find where it belongs.

Your no-repeats rule rules out retakes too. A cough becomes a writing assignment.

I replace the sentence. It takes longer, but I'm not spending an afternoon saying the same thing.

Then the log might make a repeated task slower even when the recording tool improves. Searching it is another step before an accepted clip.

Only if she hasn't left the right page open. She usually has.

At the twelve-hour release, I'd watch whether better sentences have made the collection sustainable, alongside whatever improvement the model delivers. A log can keep growing while the person reading it has had quite enough. The later hours will still cost human attention, even if a text-side fix eventually takes the currency examples off the schedule.

Thanks to our producer, Hilbert Flumingtop, for keeping the show running.

If you enjoyed this one, try episode eight, Building Your Own Whisper; episode fifteen, AI Gets Personal; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts, the human-AI collaboration podcast. No question too obscure, no rabbit hole too deep.

You'll find the episodes and more at my weird prompts dot com.

We'll be back soon.

Until then.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error

Teaching Speech Models to Transcribe Bilingual Slang

Sources

Mentions

Downloads

Recommended Listening

You Might Also Like

#6365: Teaching Speech Models to Transcribe Bilingual Slang