Quick bit of housekeeping before we get into it. Daniel sent something this week that is, on its face, extremely practical. No hypotheticals about sloths running a space program. Just a straightforward question about what it actually costs to fine-tune a model.
Which, knowing Daniel, is basically his love language.
Right. Here's what he wrote. He wants to know the practical difference between fine-tuning a model at its original precision versus fine-tuning a quantized version. He's noticed that Hugging Face makes model lineage visible. You can often see whether a fine-tune descends from original weights or from a quantized derivative, and he wants to know what that distinction actually means in practice. Then he gets specific. Focus especially on the economics and computational requirements. For an independent developer or a small project, is full-parameter fine-tuning of the original weights of a modern model realistically feasible, or is some combination of quantization and parameter-efficient fine-tuning now the norm? And he wants the major approaches, what people are actually doing in twenty twenty-six, and the trade-offs in cost, hardware, and model quality.
That's four questions in a trench coat.
It is. And they're all good. So let's start with what's actually happening when you fine-tune a quantized model versus the original, because the mechanics explain almost everything else. Herman, take it.
Two paths. Path one, you take the original weights, they're in sixteen-bit, and you update them. All of them. Every parameter gets a gradient and a new value. That's full fine-tuning. Path two, you take those same weights, you compress them down to four bits using a scheme called NF4, you freeze them so they never move, and then you bolt on a small set of trainable adapter matrices called LoRA and you train only those. That second path is QLoRA. Dettmers and colleagues, twenty twenty-three, and it's still the canonical method three years later.
And the reason anyone cares is not academic curiosity.
Not remotely. It's memory. Here's the number that reframes the whole conversation. When you do full fine-tuning with AdamW in bfloat sixteen, you need roughly six times the model's sixteen-bit size in VRAM. Six times. Because you've got the weights, the gradients, and two optimizer states, all sitting in memory at once.
And QLoRA?
About zero point four times. LoRA alone, without the quantization, is around one point two. So the multiplier drops from six to under half.
That's a factor of fifteen.
On the same model. Which is the difference between a single consumer card and a rack of H100s. Concretely, an eight-billion model. Full fine-tune, sixty gigabytes of VRAM. QLoRA, about ten. Fits on a four thousand ninety.
Sixty to ten. That's not a tuning knob, that's a different universe.
And it scales the same way. Seventy billion. Full fine-tune, four hundred eighty gigabytes, which is six to eight H100s. QLoRA, forty-eight gigabytes. One A100. Four-oh-five billion, full fine-tune, two point eight terabytes, which is thirty-two to forty H100s and effectively enterprise-only. QLoRA on that same model, two hundred ten gigabytes. Three H100s.
So the answer to Daniel's feasibility question is already visible in those numbers, but I want to push on the quality side first, because that's where the interesting stuff is. What does quantization actually do to a fine-tune?
It introduces error. That's the honest one-line version. You're representing weights with four bits instead of sixteen, so you lose resolution, and those errors propagate through training. There's a paper from October twenty twenty-four, QuAILoRA, whose entire premise is that quantization errors negatively impact post-fine-tune performance. That's not a fringe claim, that's the starting assumption.
But the same paper has a recovery number, doesn't it?
It does, and it's the most useful number in this whole space. QuAILoRA uses quantization-aware initialization, which is a fancy way of saying it picks better starting values for the adapters before training begins. And it recovers, on average, seventy-five percent of the validation perplexity improvement and eighty-six percent of the downstream accuracy gain that you would have gotten by doubling precision to eight-bit. Without any extra GPU memory.
So you can claw back most of what you lost, for free, just by being smarter about where you start.
Most of it. Not all of it. That eighty-six percent figure is the honest ceiling for that particular trick.
What about the bit-width question? Everyone says four-bit is fine. Is four-bit fine?
Four-bit is usually fine. That's the accurate statement, and the qualifier is doing real work. There's a paper from August of this year, the eighth, that maps this properly. Four-bit usually preserves performance. Two-bit often causes broad degradation. And three-bit is the weird one. Three-bit is variable depending on the task, the method, and the model scale. It's not a clean ladder where less bits equals worse.
Three-bit is the awkward middle child.
And the paper's explanation is interesting. Larger models benefit from weaker cross-layer error amplification. Meaning the errors introduced by quantization get damped as they propagate through a big network, in a way they don't in a small one. So a four-bit seventy-billion model degrades less than a four-bit seven-billion model, all else equal.
Which means the quality question isn't "is four-bit worse." It's "worse at what, on which model, for which task."
That's the right framing. And there's a second-order cost almost nobody puts in their cost comparison. There's ICML work this year finding that quantized models are substantially more vulnerable to safety alignment degradation during fine-tuning than full-precision models. The quantization errors make the safety behavior easier to knock loose.
So you fine-tune your four-bit model on your domain data and you might quietly degrade its refusal behavior along the way.
Quietly is the operative word. That's not in the VRAM number. That's not in the cost table.
And there's a time cost too, isn't there? I remember something about QLoRA not actually being faster.
It saves memory but not time. That's a direct quote from a paper this year on fast quantized LoRA. The reason is mechanical. QLoRA stores the base weights in four bits, but to actually do the matrix multiplication during training, it has to dequantize them back up on the fly, every single step. So you're paying a decompression tax on every forward and backward pass. It trains more slowly than sixteen-bit LoRA.
So the simple story, "quantization is strictly cheaper," is wrong.
It's wrong in a specific way. Your hardware requirement goes down dramatically. Your wall-clock training time goes up. You're trading dollars of GPU rental for hours of waiting.
Which for a solo developer is usually the right trade.
Almost always, because the alternative isn't "spend more money," it's "you can't do this at all."
There's one more theoretical point I want on the table before we go to economics. The argument that PEFT is fundamentally limited.
Yes. Liu and colleagues, a paper from May twenty twenty-five. The claim is that parameter-efficient fine-tuning is a strict subset of full fine-tuning. The limited parameter space constrains the model's representational ability, making it more susceptible to perturbations. That's the strongest theoretical case that full fine-tuning retains a real edge.
Strict subset. That's a strong word.
It's a mathematically loaded word, and I think it's correct as stated. The set of functions you can reach by only moving the adapters is a subset of the set you can reach by moving everything. The practical question is whether the functions you need are inside that subset. For a lot of narrow domain tasks, they are. For heavy reasoning or instruction-following work, the gap shows up.
So that's the mechanism. Damage from quantization, partial recovery through smarter initialization, a quality gap that's task-dependent rather than uniform, a time cost that undercuts the simple savings story, and a safety-alignment effect that most comparisons ignore entirely. Now let's talk about what it actually costs.
The cost tables are where this stops being philosophy. Fifty thousand training samples, an eight-billion model. Full fine-tune, fifteen to twenty-three dollars. QLoRA, one seventy-six to two sixty-four. Six to thirteen times cheaper.
At eight billion the absolute numbers are small enough that it barely matters.
It barely matters at eight billion. It matters enormously at seventy. Same dataset, seventy-billion model. Full fine-tune, three hundred ninety-eight to five hundred ninety-eight dollars. QLoRA, forty-five to sixty-eight. That's an eighty-five percent reduction.
And the on-demand rental comparison is even starker, isn't it?
GigaGPU published one in April. Full fine-tune of Llama three point one, seventy billion, on-demand, roughly twenty-four hundred dollars. The same job with QLoRA on a dedicated RTX six thousand Pro, under eighty-five dollars.
Twenty-four hundred versus eighty-five. That's not a discount, that's a different activity.
It's the difference between a line item you have to justify to somebody and a charge you put on a personal card.
Where's the feasibility line? At what size does full fine-tuning stop being something a person can do on their own hardware?
Presenc AI put it clearly in May. Single-workstation full fine-tuning is impractical above thirteen billion parameters. That's the line. Below thirteen, if you've got the card, you can do it. Above thirteen, you're renting a cluster.
And QLoRA's line?
QLoRA on seventy billion is now feasible on a single device with a hundred twenty-eight gigabytes of unified memory. A DGX Spark. A Mac Studio with the M5 Max. That's a desk, not a data center. Unsloth's own documentation says a seventy-billion LLaMA fits in under forty-eight gigabytes with QLoRA.
So the practical answer to Daniel's question is that the feasibility threshold for full fine-tuning has stayed roughly where it was, and the threshold for QLoRA has moved down onto people's desks.
The gap has widened, not narrowed.
What are people actually running in twenty twenty-six? Not what's theoretically available.
QLoRA is the practical default. That's the phrase being used, and it's accurate. The Unsloth documentation is refreshingly honest about the trade-off. LoRA, meaning sixteen-bit, is slightly faster and slightly more accurate but uses four times more VRAM. QLoRA is slightly slower and marginally less accurate. That's the whole comparison, stated plainly by the people who build the tooling.
Slightly slower and marginally less accurate. That's the deal.
On the framework side, four names. Unsloth, which is the single-GPU speed play, two to five times faster with seventy to eighty percent less VRAM. Axolotl, which is multi-GPU with FSDP and DeepSpeed and YAML-based reproducibility. LLaMA-Factory, which is UI-driven and supports over a hundred model families. And TRL, which is where the research and RLHF work happens.
And TorchTune?
Dead. Development wound down in July twenty twenty-five, last release was zero point six point one that April. If you're following a tutorial that uses it, the tutorial is stale.
Now. Here's the part of Daniel's question I find more interesting than the cost tables. He said Hugging Face makes lineage visible. Does it?
It makes it visible. It doesn't make it true.
Say more.
The mechanism is a metadata field on the model card called base_model. The Hub recognizes exactly four relation types. Finetune, adapter, merge, quantized. And that's what builds the family tree you see on the site. It's a nice feature. It is also entirely self-reported.
Meaning the uploader types it in.
The uploader types it in. VentureBeat put it about as bluntly as it can be put in July. That tag is a string the uploader typed. Hugging Face does not require uploaders to substantiate the claim through weight-level analysis.
So when I look at a model card and it says this fine-tune descends from that base, I'm taking someone's word for it.
You are. And the scale of the problem is the interesting part. Sixty-nine percent of new open-model derivatives declared Qwen as their parent as of February this year. Up from one percent in January twenty twenty-four. That's a self-reported string that nobody verified, attached to more than two thirds of the ecosystem.
That's not a lineage graph, that's a rumor with a schema.
There is exactly one independent verification layer I could find, and it's Cisco's. They built a provenance explorer that does weight-level fingerprinting. Embedding anchor similarity, norm layer fingerprints, weight-value cosine. They reported ninety-six point four percent accuracy on a hundred-eleven-pair benchmark at a zero point seven threshold.
And coverage?
About nine hundred models. Against two million plus on the Hub.
So under zero point zero five percent.
Under zero point zero five percent. It's a proof of concept that the technique works, not a solution to the problem.
Here's the twist I didn't expect. The download numbers invert the publish numbers.
They do, and it's a lovely detail. Across the whole Hub, finetune and adapter dominate. Three hundred twenty-eight thousand finetune relations, three hundred twenty thousand adapter relations. Quantized is third at a hundred seventy thousand. But if you narrow to the top fifty thousand models by downloads, quantized wins sixteen thousand to five thousand. Three to one over finetune.
So people publish fine-tunes and consume quantizations.
The census put it well. Quantized repos are disproportionately popular, not disproportionately numerous. There's a gap between what the ecosystem builds and what it actually runs.
Which makes sense. You fine-tune once, you quantize for every deployment target. The quantized artifact is what ships.
And it's the artifact with the least provenance attached to it.
Now there's a legal layer under all of this that I think is going to matter more than the technical one. The EU AI Act.
Enforcement powers over general-purpose AI providers kicked in on the second of August this year. Fines up to fifteen million euros or three percent of global turnover. And here's the threshold that matters for this conversation. If your modification compute exceeds one-third of the compute used to train the original model, you can be deemed a provider.
So whether you did a light adapter tune or a heavy full fine-tune is now potentially a regulatory question, not just a technical one.
And the only thing distinguishing those two cases on a model card is a string somebody typed.
You've been quiet back there.
Hilbert: The number's wrong. Sixty gigabytes for the eight-billion.
Go on.
Hilbert: Depends on the batch size and the sequence length. Sixty is the clean figure. In practice, with real data, you're paging. I worked at a place in eighty-six that did model compression for embedded systems. Vision models for industrial cameras. My job was taking a full-precision model and shrinking it down to run on a chip with almost no memory. We had a customer, a glass manufacturer. They used the system to catch defects on a production line. Hairline cracks in the panels. The full model caught every one. We compressed it, shipped it, and three weeks later they called and said it was missing a specific crack. Just one kind. A particular angle, particular lighting. Everything else it caught fine.
What did the numbers say?
Hilbert: The numbers said we were fine. The hairline crack was zero point one percent of the test set. The average didn't move. Nobody looking at the spreadsheet would have flagged it. But that one case was the entire reason they bought the system. That was the defect that was costing them money. Everything else the human inspectors already caught.
So the average was telling you the truth and the truth was useless.
Hilbert: The average is a summary of the cases you don't care about. It's the cases you do care about that decide whether the thing works. I'm not saying QLoRA is bad. I use it. I'm saying the number that matters isn't the average degradation, it's the worst case for your specific task. And that's not a number anybody publishes, because it's different for every task.
How did you fix it?
Hilbert: We kept a small full-precision fallback. It only ran on the frames the compressed model flagged as uncertain. Cost us almost nothing in compute because it almost never fired.
That's quantization-aware initialization. Functionally. You're using a higher-precision path to catch the cases the low-precision path gets wrong.
Hilbert: I don't know what it's called. That's what we did. Someone's waiting for me in the car park.
So the misconception I want to kill before we close. The one I hear constantly is that the quality gap between QLoRA and full fine-tuning is negligible, so you should always just use QLoRA.
The correction is that "negligible" is an average, and averages hide the cases that matter. One to three percent on most benchmarks. Under two percent on domain-specific tasks. That's real and it's usually fine. But PEFT is theoretically a strict subset of full fine-tuning, and it falls short on reasoning and instruction tasks specifically. Quantization adds safety-alignment degradation on top. For a narrow classifier or a style transfer, the gap is nothing. For knowledge-heavy or safety-critical work, it might be the whole thing.
The gap is small on average and specific in the tail, and the tail is where the value lives.
Which is exactly what Hilbert just described from the other direction.
One thing worth flagging for Daniel. We went looking for a study that directly compares fine-tuning from original weights versus fine-tuning from a quantized derivative as a lineage question. It doesn't exist. The literature compares methods. Full fine-tune versus LoRA versus QLoRA. It doesn't compare provenance. Nobody has run the experiment where the only variable is what the parent weights were.
That's a real gap. And it's the exact question Daniel asked.
It is. And the EU AI Act's one-third threshold is going to create a legal reason to close it, because as enforcement ramps up, a self-reported base_model tag stops being a convenience and starts being a liability. If you can't prove what your model descends from, that's a problem you didn't have two years ago.
The technical question and the compliance question are converging on the same field.
If you're building something with a fine-tuned model, the question isn't just whether it works. It's what it descended from, and whether you can prove it.
Thanks to Hilbert Flumingtop for producing.
This has been My Weird Prompts. If you're enjoying the show, a review helps more than you'd think. We'll be back soon.
