# Why Speech Model Fine-Tuning Demands Clean Restarts

My Weird Prompts — an AI-generated podcast. Daniel Rosehill sends in a prompt; AI hosts Corn and Herman discuss it. Verify important claims independently.

- Episode: 6408
- Published: 2026-11-02
- Duration: 18:20
- Audio: https://episodes.myweirdprompts.com/audio/speech-model-checkpoint-fine-tuning-96009.m4a
- Tags: speech-recognition, fine-tuning, local-ai
- Web page: https://myweirdprompts.com/episode/speech-model-checkpoint-fine-tuning-96009/
- JSON: https://myweirdprompts.com/api/episodes/speech-model-checkpoint-fine-tuning-96009.json

## Summary

Fine-tuning Parakeet on evolving vocabulary often tempts developers to train solely on new deltas using saved checkpoints. Yet resuming training from PyTorch Lightning checkpoints risks carrying stale optimizer states and decayed learning rates. This breakdown explores the mechanical distinctions between NeMo archives and training checkpoints, the hazards of cumulative model drift, and why full joint retraining from the base model outperforms complex regularization schemes like elastic weight consolidation when compute costs remain low.

## The prompt

> As you know, I undertook a significant project fine-tuning Parakeet for my own domain vocabulary recently. This was a significant undertaking, but I was able to get the word error rate on Niche Tech Vocab down to 4.7% on our latest benchmarks, and I'm approaching the point at which more samples don't help, and I'm happy to consider the job done the time being, with the intention of doing periodic retrainings as I identify new words I want to add. During the first few iterations I didn't even consider the idea of saving checkpoints but on the last run I did. The principle seems fairly straightforward. I start from a checkpoint but then I'm not sure the dataset structure goes from there. Am I training subsequent versions on the Delta between the dataset that went into the last checkpoint and what I've gathered since. And is there any unanticipated gotchas with this approach? The trainings have so far cost under$5, but they become a little more expensive as the dataset grows. Using serverless GPU on Modal, the last one took approximately two hours. So, given how important the project is to me, the idea of using a checkpoint approach is not to save money. So if it's technically cleaner to do each one scratch, I'd be tempted to simply do that. In this episode, let's look at the technical pros and cons of incremental training on ASR fine-tunes.

## Show notes

Iterative fine-tuning of automatic speech recognition models introduces critical workflow decisions around checkpoints and dataset management. In NeMo, restoring from a PyTorch Lightning .ckpt inherits existing optimizer states and scheduler positions meant for crash recovery, while initializing from a .nemo archive cleanly loads model weights for a fresh run. Training strictly on new vocabulary delta triggers catastrophic forgetting, an issue continual learning methods attempt to mitigate through replay buffers or regularization techniques like Elastic Weight Consolidation. However, when full training runs are fast and inexpensive, joint retraining from base Parakeet on the entire accumulated manifest remains the gold standard. This avoids optimizer skew, chained model drift, and benchmarking errors caused by unversioned evaluation sets.

## Sources

- [NVIDIA, model card, last modified 2026-08-05](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)
- [NeMo docs, Checkpoint Formats (current main)](https://raw.githubusercontent.com/NVIDIA/NeMo/main/docs/source/checkpoints/intro.rst)
- [NeMo docs, ASR model checkpoint list (current main)](https://raw.githubusercontent.com/NVIDIA/NeMo/main/docs/source/asr/asr_checkpoints.rst)
- [NeMo Speech docs index (updated 2026-10-05): `init_from_checkpoint` config, `restore_from`, `setup_optimization` fine-tuning patterns](https://context7.com/nvidia-nemo/speech)
- [Chang, Lee & Lee, Towards Lifelong Learning of End-to-end ASR, Interspeech 2021](https://arxiv.org/abs/2104.01616v3)
- [Vander Eeckt & Van hamme, Continual Learning for Monolingual End-to-End ASR, EUSIPCO 2022](https://arxiv.org/abs/2112.09427v4)
- [Plantinga, Yoo & Dhir, Continual Learning for End-to-End ASR by Averaging Domain Experts, 2023](https://arxiv.org/abs/2305.09681v1)
- [Ahadzi et al., Continuous Learning for Children's ASR: … EWC and Synaptic Intelligence, Interspeech 2025](https://arxiv.org/abs/2505.20216v1)
- [Vander Eeckt & Van hamme, Inverse-Hessian Regularization for Continual Learning in ASR, ICASSP 2026](https://arxiv.org/abs/2601.14751v1)
- [Vander Eeckt & Van hamme, Efficient Rehearsal for Continual Learning in ASR via Singular Value Tuning, IEEE TASLP 2026](https://arxiv.org/abs/2601.18266v1)
- [Vander Eeckt & Van hamme, Parameter-Efficient Continual Learning for ASR, Interspeech 2026](https://arxiv.org/abs/2606.09342v1)
- [Adethya & Nirmala, A Study on Regularization-Based Continual Learning Methods for Indic ASR, 2025](https://arxiv.org/abs/2508.06280v1)

## Transcript

Corn: Daniel's got a Parakeet fine-tune sitting at 4.7 percent word error rate on his niche tech vocabulary, and the returns from adding more samples have flattened out. He saved checkpoints on the last run for the first time. Now he wants to add new words periodically, and he's wondering whether the next round trains on just the delta or on everything. Cost isn't the motive. A run is under five dollars and about two hours on Modal, so if starting from scratch is technically cleaner, he'll do that.

Herman: Herman Poppleberry, for anyone keeping score at home. It sounds like a workflow question, but it's a question about what a checkpoint is. In NeMo the word covers two different objects, and most of the confusion dissolves once you separate them.

Corn: Two objects with one name. Naturally.

Herman: A .nemo file is a tar archive. Config, weights, tokenizer, whatever artifacts the model needs. Loading it is an initialization. A PyTorch Lightning .ckpt saved during training also carries the optimizer state, the scheduler position and the step counter. Loading that is a resumption.

Corn: So which one Daniel saved decides what he inherits.

Herman: Yes. With a .ckpt he's inheriting training state tuned for the old dataset. With a .nemo he's starting a fresh run from good weights. NeMo documents both as first-class patterns, so neither is a hack.

Corn: I'd have guessed a checkpoint was a checkpoint. I've been wrong about less.

Herman: Everyone guesses that. And the default tooling doesn't help, because both files get called checkpoints in logs and folder names. You can end up with a directory full of files that look like siblings and behave like strangers. Let me walk through resuming mechanically, because the dataset answer falls out of it.

Herman: When you call trainer fit with a ckpt path, Lightning restores the weights, the Adam moments, the scheduler state and the step count. You pick up mid-run as though the training never stopped. That's perfect if a job crashed at hour one of two. It's the wrong tool for a new round of data.

Corn: Why wrong? It sounds like the most faithful version of continuing.

Herman: Because the learning rate schedule is partway down its curve, or finished. If it finished, you may be resuming at a learning rate near zero, and the new words barely register. The Adam moments also encode gradients from the old loss landscape on the old dataset size. You hand the optimizer a memory of a different problem. It may behave fine, but you've no idea, and it's hard to audit.

Corn: Like inheriting a thermostat someone else set.

Herman: Roughly. And you can't see the setting without digging into the state dict. The documented clean-restart pattern is the init from checkpoint option in the model config. It loads weights from a previous checkpoint but starts a fresh optimizer, fresh scheduler, fresh step counter. The other route is to restore from the previous .nemo, then set up optimization again with a smaller learning rate. The NeMo fine-tuning tutorials do exactly that, copying the optimizer config and lowering the rate before fitting.

Corn: Why lower? The model already knows the old words.

Herman: Because it's already near a good solution. A big learning rate shoves it out of that basin in the first few hundred steps, and then it spends the rest of the run climbing back in. Lower is a nudge instead of a shove.

Corn: Is there a way to tell which one you got? Short of waiting for the results?

Herman: You can watch the first few hundred steps. If the training loss spikes well above where the last run ended and then slowly settles, the rate was too aggressive. The model got shoved. If it starts near the old value and drifts down, you were gentle. It's a crude diagnostic, but it costs nothing, because the logs are already there.

Corn: So the warm start only pays off if you're gentle with it.

Herman: Right. Otherwise you've kept the weights and thrown away the reason you kept them.

Corn: Fine. Now the question Daniel asked outright. Train on the delta, or on everything?

Herman: Everything. Training only on new data is how you get catastrophic forgetting. The model gets better at the new words and worse at the old ones, because nothing in the loss reminds it the old ones exist. Chang, Lee and Lee put it plainly in their Interspeech paper from twenty twenty-one. Collecting new data and fine-tuning on it leads to higher error rates on the earlier datasets. Their lifelong approach, which keeps some past data around, got a 28.7 percent relative WER reduction over the plain fine-tuning baseline.

Corn: Twenty-eight percent sounds like the savings from a good coupon, not a research result.

Herman: Relative, so on a base rate of a few percent it's small in absolute terms. But the direction matters more than the size. And there's a structural point I like. In the continual learning papers, the upper bound, the thing everyone measures against, is joint training on all the data.

Corn: Which is Daniel's from-scratch option.

Herman: Yes. His option isn't a lazy alternative to the research. It's the reference the research is trying to approximate. Vander Eeckt and Van hamme closed more than forty percent of the gap between naive fine-tuning and joint training while using just point six percent of the original data. Their whole motivation is that you shouldn't have to store everything. Daniel can store everything. A vocabulary dataset is a few gigabytes, not a privacy-restricted call-center archive.

Corn: So the exotic methods solve a problem he doesn't have.

Herman: Mostly. Think of a hospital that can't keep patient audio for legal reasons. For them, forgetting is a real constraint, so they need clever methods. Daniel's constraint is a hard drive.

Corn: A very cheap constraint.

Herman: And the cheap insurance is rehearsal. A twenty twenty-six paper from the same group shows replay works even with minimal memory, outperforming fancier methods with a single utterance per previous task. If Daniel ever did want delta-only training, a handful of old utterances mixed in gets him most of the protection.

Corn: Though why would he, when the full set is two hours?

Herman: He wouldn't. The default is the full combined set. New-data-only needs an explicit mitigation, and for him it's a solution to a nonexistent cost problem.

Corn: Good. So the mechanism is clear. That's the easy half, I suspect.

Herman: It is. The gotchas don't show up until round three or four. Start with the optimizer state, which we covered. The deeper one is drift. Each chained round starts from a model that has already moved away from base Parakeet. Whatever quirks round one introduced, say it over-prefers a tech term, get baked in, and round two builds on them. There's no path back to the original distribution.

Corn: What does that look like in practice? Say round one makes it too fond of a word.

Herman: Say it starts writing a framework name every time it hears something vaguely similar. Round two adds more data, but the model's prior is already skewed, so the new examples get interpreted through that skew. By round four you're debugging a habit whose origin is three checkpoints back, and nothing in your logs says where it started.

Corn: Couldn't he just spot it early? Read the transcripts after each round?

Herman: He could, if he knew which words to look for. But drift is rarely one dramatic error. It's a slight lean, a few extra substitutions spread across hundreds of utterances, and each one looks like an ordinary mistake. You only see the pattern by comparing against a fixed reference, which a chain doesn't give you.

Corn: Unless he keeps the base.

Herman: Which is exactly the from-scratch argument. Fine-tune from base Parakeet, the six hundred twenty-seven million parameter model, every time, on the whole accumulated manifest. Then every version has the same origin.

Corn: I distrust anything that can change while I'm not looking, and a chain of checkpoints is a thing that changes a little every round. I'd rather own the same worse thing.

Herman: You'd rather own the same thing, full stop. Though to be fair, this time you're right on the merits.

Corn: I was going to claim indifference.

Herman: You were not going to be indifferent. The next gotcha is the quiet killer, and it's the eval set. If the niche vocab benchmark grows between rounds because Daniel added new words, then 4.7 percent in round five isn't comparable to round four. You've changed the test and the model at once.

Corn: That one hurts, because it's invisible. The number goes down, you feel good, and the exam got easier.

Herman: Or harder, and you panic for nothing. Imagine round four adds fifty rare product names to the benchmark. The model might have improved, but the new names are hard, so WER ticks up and you conclude the round failed. The discipline from the literature is a fixed held-out set, plus a fixed set from the original domain. New words get their own separate set. Looking past a single WER number is the whole point there, same instinct as in that earlier episode on benchmarking beyond WER.

Corn: So three sets. The old vocab, the general domain, and a new-words set that starts life small.

Herman: Yes, and the new-words set only joins the frozen ones once it's stable. Until then you report it on its own line. Then when a number moves, you know which exam moved it.

Corn: Do the regularization tricks help? Elastic weight consolidation, synaptic intelligence, the ones with names like a lawyer's firm.

Herman: Modestly. A children's ASR study reported relative WER reductions of 5.21 percent for EWC and 4.36 percent for SI against plain fine-tuning. Real complexity for small gains. Inverse-Hessian regularization is newer, but it needs a Kronecker-factored approximation of an inverse Hessian, and that's a lot of machinery for a solo project.

Corn: I'm not factoring anything with a Kronecker, whatever that is.

Herman: Neither would I, honestly. The intuition behind EWC is simple enough, though. It estimates which weights mattered most for the old task and penalizes moving them. It's a rubber band on the important parameters. The cost is computing that importance estimate and tuning how stiff the band is.

Corn: And if you tune it wrong?

Herman: Too stiff and the model can't learn the new words. Too loose and you've got plain fine-tuning with extra steps. There's also the parameter-efficient family, where you restrict adaptation to the low-energy tail of the singular values so the dominant components stay put. It reduces forgetting nicely, but it's a different training setup than plain full fine-tuning.

Corn: So what's the verdict for a guy with a two-hour, five-dollar run?

Herman: From scratch, meaning from base on the full dataset. It's reproducible from one base checkpoint plus one manifest. Rolling back means picking an earlier manifest, not untangling a chain of N checkpoints. And runs are directly comparable when the eval set changes. The continual learning literature is mostly about accent, language or domain shift, where old data is expensive. Vocabulary is cheap to keep and cheap to replay.

Corn: What would you do if the run took forty hours?

Herman: Different answer. I'd chain, with a .nemo init, a fresh low-rate optimizer, and the full data replayed anyway. At forty hours and a few hundred dollars a round, the waste of retraining from base starts to matter. You asked me, so that's the honest wobble. At two hours, no.

Corn: Hm. I'd have said chain sooner. Sloths don't like waiting.

Herman: You've never started a two-hour job in your life without napping through it.

Corn: Correct, and the nap is the efficient part. Sorry, someone's put a hand up. We have a question.

Beryl: Sorry, I've been nodding along for weeks and I'm still not sure I've got the first bit. When you say checkpoint, is that like a bookmark in a book, or more like a saved game on a console, and if it's the saved game, does the model keep its old memories or start each round fresh?

Herman: It's a bookmark, love, with a photocopy of the pages you've read stapled to it, and you decide whether to keep reading from there.

Corn: That cost you something.

Herman: It cost me a lot. To answer the second half properly, the model keeps what it learned either way. The weights are the memory. What changes is whether the training machinery, the pace and momentum, carries over too, or starts fresh.

Corn: So the saved game is the .ckpt, and the photocopied pages are the .nemo.

Herman: Close enough, and don't tell anyone I said the console one was acceptable. Where were we.

Corn: The hybrid nobody mentions.

Herman: Right. Keep every round independent, each one fine-tuned from base on the full accumulated data. If Daniel ever wants a single merged model, average the weights. Plantinga, Yoo and Dhir showed that linear interpolation of models fine-tuned from the same generalist model performs well on all tested data, with almost no loss on the original domain. Reproducibility, plus a merge if he wants it.

Corn: Averaging models sounds like it shouldn't work.

Herman: It works because they all start from the same base, so they sit in the same neighborhood. Start them from different places and you'd get mush. It's like averaging two routes drawn on the same map versus two routes drawn on different maps.

Corn: So the base checkpoint is doing double duty. It anchors the history and it makes the merge possible.

Herman: One decision, two benefits. That's why I keep coming back to it.

Neville: Hello! Yes. I've been brought in specially as a consultant on this one.

Corn: You've been at the desk since nine making the tea, Neville.

Neville: Well, yes. Also that. I'm the intern, actually. Uncle Hilbert said to help, and he's technically my great uncle twice removed, then sideways, so I do what he says.

Herman: Fair enough. What've you got?

Neville: I tried to train a little speech thing for my flatmate's podcast, on the Number Nine bus, on my phone. Four pounds fifty on a cloud computer. I started from a checkpoint, which is the saved game thing, and I only trained it on the new words. Then it forgot the old ones, which I think is called catastrophic remembering.

Herman: Forgetting, Neville. Catastrophic forgetting, and you hit exactly what we described. Training only on the new words is what causes it. Putting the old recordings back in fixes it.

Neville: Do I need all of them, though? My phone's quite full.

Herman: Not necessarily. Even a small sample of the old recordings mixed in helps a lot. But if you can keep them all, keep them all.

Neville: What if the old ones are on the cloud computer and the new ones are on my phone? Is that a problem?

Herman: Not for the model. It only sees the combined list of files at training time. Where they live matters for upload time, not for forgetting. Keep the originals somewhere durable and let the job pull them together.

Neville: Oh, that's much better, actually. I'll tell my flatmate. A group of hedgehogs is sometimes called an array, though they mostly live alone.

Corn: Useful.

Neville: Uncle Hilbert keeps everything in boxes. Labelled, dated, boxes inside boxes. He'd have kept the old recordings.

Herman: He'd have been right, too.

Corn: Which is roughly Daniel's situation. The manifest is the box.

Herman: Close to it. Each retraining becomes a natural checkpoint in the audit sense, even if it isn't one in the file sense. A manifest, a base model, a fixed eval set. That's a version you can reproduce on demand.

Corn: So the open question is whether chaining is worth the discipline. If Daniel does it, he needs a fixed eval set and a replay of everything. At that point he's built most of the from-scratch pipeline anyway.

Herman: And then he may as well press the button on the clean run, since the two hours are cheap.

Corn: Thanks to Hilbert Flumingtop for producing, and to Neville for the tea and the hedgehogs.

Herman: This has been My Weird Prompts. If you like the show, a review helps more people find us.

Corn: We'll be back soon.