Daniel's been sitting on this one for a while, and it's a good one. He wants to build a personal fine-tune of an open-source model, and he's been clear about what it's not for. It's not about stuffing knowledge into the thing. It's about voice. He wants a daily AI assistant that sounds like him, that responds the way he'd respond, and he wants to get there by generating responses with a model and then editing them by hand.
Which is the part that matters, because that's the data.
Right. And he's seen two ways of doing it. The first is what he's drawn to. You take the text the model gave you and you change it. Anything you want, as much as you want, and the model trains on the gap between what it produced and what you actually wanted. The second is the one he's seen other people do, where instead of editing you leave notes. "The midsection was too long." "Don't use an analogy here." Commentary rather than correction.
And he suspects the notes are the weaker foundation.
He does. He says writing notes feels like a shaky way of explaining a deviation, and the direct editing pattern strikes him as more technically logical. He also says you could do both. His questions are these. What are these two methods actually called? How do they work in practice? Why does direct editing feel more sound? Which one tends to produce better results for text generation? And can you combine them?
That's five questions, and I think we can answer all five.
Let's try. Start with the names, because the names will do a lot of work here.
The direct editing one is studied as learning from user edits, or user-edit fine-tuning. The canonical framing is a paper by Gao and colleagues out of NeurIPS 2024, and it describes Daniel's exact workflow. A user generates a response with a language agent, then edits it to personalize it to their latent preference. That's the phrase they use, latent preference, which is a good one, because the whole point is that Daniel's preferences exist in his head and nowhere else until he edits something.
The edit is the confession.
The edit is the confession. And the follow-up theory paper, Misra, Pacchiano and Chi, formalizes the data as a triplet. Context, the model's response, and the user-edited response. Three pieces, and we'll come back to why that structure matters so much.
And the note-writing method?
Critique and revise. CnR. That's Jin and colleagues at Amazon, and the record format there is prompt, initial response, critique, revision. Four pieces instead of three, and the critique is the new one. There are related names floating around. Natural-language feedback. Critique fine-tuning. Fine-grained span feedback, where annotators mark the specific spans they liked or disliked and say why.
So one method is you fix it, the other is you talk about it.
That's the whole distinction, and I want to be honest about something before we go further. There is no head-to-head study comparing these two specifically for stylistic or personality fine-tuning of an open-source model. I went looking. The closest evidence is an ablation in the critique-and-revise paper on general tasks, and a critique-versus-imitation comparison in the critique fine-tuning work, which is math reasoning. Neither one is Daniel's exact question.
That's a real gap.
It's a real gap, and it's the kind of gap a solo practitioner with Unsloth could actually close. But it means we reason from adjacent evidence rather than direct evidence, and I'd rather say that up front than pretend otherwise.
So to understand why Daniel's intuition might be right, we need to look at what each method actually produces as training data.
Start with the edit. The edited response becomes a high-quality target label for standard supervised fine-tuning. You're telling the model, this is the answer. Not a description of the answer. The answer. And the Misra paper's real contribution is showing that a single edit isn't one signal, it's three at once.
Walk me through the three.
First, a supervised signal. You train on the edited response directly. Second, a preference signal. The edit implies that the edited version is preferred over the original, which means you can use it with DPO, direct preference optimization, no separate preference labeling required. Third, a cost signal. The edit distance between the original and the edited version is a measure of how wrong the model was. That distance can serve as a negative reward in reinforcement learning. The paper's own phrasing is that user edits unify preferences, supervised labels, and cost, three things the literature normally studies separately.
Three signals from one gesture.
From one gesture. And that's the technical core of Daniel's intuition. The delta between the response and his preference is captured as an explicit, machine-readable target. There's no interpretation step. The model isn't being told what Daniel wants. It's being shown.
Let me push on that, because "shown" sounds clean. What does the model actually see when Daniel makes a small edit versus a large one? Does the size of the edit change what gets learned?
It does, and this is where the cost signal earns its keep. If Daniel changes one word, the edit distance is tiny, and the preference signal is weak. The model was almost right. If he rewrites the whole paragraph, the distance is large, and the preference signal is strong. The gradient is proportional to how wrong the model was. So a small edit is a nudge and a large edit is a shove, and the same data pipeline handles both without Daniel having to label anything.
So his laziness on a Tuesday is captured as a weaker signal.
His laziness on a Tuesday is captured as a weaker signal, and his care on a Wednesday is captured as a stronger one. The method is honest about effort in a way that notes aren't. A note that says "tightened this up" reads the same whether he tightened one clause or rewrote the whole thing.
Now the notes.
The critique-and-revise version. Annotators write a critique, which includes positive and negative aspects, specific flawed sentences, and then a revised response. The model trains on the whole record. Prompt, initial, critique, revision. And here's the finding that surprised me. The best formulation includes the critique, not just the revision.
Meaning the notes carry signal beyond the fix.
Beyond the fix. The paper's own line is that the improvements aren't simply due to the higher quality of the revised responses. Enabling the model to generate a critique and revise the initial response contributes to the improvements. So the note isn't decoration. It's doing work.
Which cuts against Daniel's instinct.
Partially. And here's where it gets interesting, because the same paper accidentally makes Daniel's case. They had to instruct their annotators to keep critiques and revisions fully aligned, and to introduce no additional changes that weren't mentioned in the critique. They had to write that rule down. Which tells you that in practice, notes and edits drift apart. People write a note saying they tightened the opening and then also changed the tense, cut a quote, and added a clause. The note describes one change. The edit made four.
So the note is a partial record of the edit.
The note is a story the annotator tells about the edit. It's usually true. It's rarely complete.
And there's a quality ceiling on the notes themselves.
There is. The critique fine-tuning work found that about twenty percent of GPT-4o critiques contained errors. That's a frontier model generating critiques, and one in five was wrong. For a personal style fine-tune, Daniel is the annotator. So if he writes notes, his own ability to articulate what's wrong becomes the ceiling on how good the data can get. If he edits instead, his taste is the ceiling, and taste is the thing he actually has. He doesn't need to be able to explain why a sentence is wrong. He just needs to be able to fix it.
That's the strongest version of his argument.
It's a good argument. There's one more piece, and it's a warning. The Misra paper proves something about user quality. Supervised fine-tuning on edits is excellent when the user is strong, meaning the edits are near-optimal. It degrades when the user is weak, meaning the edits are sloppy or inconsistent. DPO on edits is more robust across user quality. So if Daniel edits his own assistant, and he edits well on Monday and carelessly on Thursday, SFT will absorb both. DPO will be more forgiving.
The method has an opinion about the annotator.
Every method does. The question is whether it punishes you for being human.
So the mechanisms are clear. But which one actually produces better results, and what does the evidence really say?
The evidence is split, and I think that's the honest answer. Start with where critique wins. The Amazon study ran an ablation. Including the critique beat revision-only, and it beat it clearly. The full record, prompt to initial to critique to revision, won sixty-four percent of the time against prompt-to-revision in human evaluation, and seventy percent against it using GPT-4 as the judge. And when they compared the full record against prompt to initial to revision, dropping only the critique, it still won fifty-four percent human, fifty-six point nine percent GPT-4.
So the critique adds value on top of the revision.
Measurably. And there's more. They took ChatGPT's existing responses and improved them through revision iterations. After one iteration, fifty-six point six percent win rate. After five iterations, sixty-five point nine percent. And the data efficiency is striking. Fewer than a thousand samples sufficed. Revision quality saturated around eight hundred samples. This is not a method that needs a hundred thousand examples.
That's a small dataset.
It's a small dataset, and the base model size mattered. At eight hundred samples, Falcon-40B hit eighty-one point six percent win rate, GPT-NeoX-20B hit seventy-four point seven, GPT-J-6B hit sixty-five. Bigger base, better result. Then critique fine-tuning goes further. Wang and colleagues report four to ten absolute points over standard SFT across six math benchmarks, using fifty thousand samples where competitors used two million or more. They matched a DeepSeek-R1-style reinforcement learning setup at eight H100-hours versus eleven hundred and fifty-two. That's a hundred and forty times less compute.
A hundred and forty times.
And a thirty-two billion parameter model needed only four thousand samples, where a comparable effort needed seventeen thousand. Their line is that they challenge the prevailing paradigm of SFT, because critique fine-tuning encourages deeper analysis and nuanced understanding, traits they say standard SFT overlooks.
So the notes win.
On those tasks, with those caveats. Critique fine-tuning is reasoning, not style. The model learns to critique, not to imitate. That's a different job from what Daniel wants. He doesn't want an assistant that reasons better about math. He wants one that writes like him. So the CFT result is real and impressive and only partly transferable.
And where do the edits win?
On robustness and on grounding. The Misra paper's user-quality result is the key one. SFT on edits is excellent when the user is strong, degrades when the user is weak, and DPO on edits is more robust across the board. Their LateEnsemble, which trains separate policies and selects between them with a bandit, was the most robust overall. And remember what the edit gives you that the note doesn't. A concrete target. The note has to be re-interpreted. The edit doesn't.
So for style specifically, Daniel's instinct holds.
For style specifically, I'd say yes, with one refinement. Direct editing is the more grounded target because the delta is explicit. But the literature's answer to his last question is the one that actually matters. Can you combine them? Yes. And combining is the consensus best practice.
How?
Three ways, and they stack. First, at the loss level. The Misra paper's EarlyEnsemble combines the SFT loss and the DPO loss in a single objective. Both signals from the same edit data. Second, at the architecture level. LateEnsemble trains separate policies and bandit-selects between them. Third, at the data-format level, which is the one I'd actually recommend to Daniel. The critique-and-revise format is itself a combination. The model sees the critique and the revision. The fine-grained span-feedback work does the same thing at a finer grain. Annotators mark the spans they liked and disliked, explain why, and then the model rewrites the disliked spans. That produces preference pairs you can train DPO on.
So Daniel edits directly and leaves a short note per edit.
He edits directly and leaves a short note per edit, then trains with both objectives. The edit gives the model the grounded target. The note gives it the reasoning behind the deviation. That's the version of his workflow the literature actually supports.
And the tooling supports it.
Unsloth's docs literally list this as a use case. Customize behavior, adjust the model's tone, personality, or response style. Data Recipes lets you upload your data and build a dataset through a visual graph-node workflow, with preview and validation before you commit to a full build. The formats are Alpaca, ShareGPT, ChatML, take your pick. And the guidance on size is friendlier than people expect. A bare minimum of around a hundred rows, ideally a thousand or more, and quality matters more than quantity. Which lines up with the eight-hundred-sample saturation point from the critique-and-revise work.
So Daniel's looking at a thousand rows of his own edits.
Roughly. And he can generate the initial responses synthetically if he wants, using a local model like Llama 3.3 70B, or something bigger through an API. The generation is the easy part. The editing is the work, and the editing is the data.
Here's what I keep circling. The note and the edit are both descriptions of the same deviation. One is in prose, one is in the artifact itself. And the literature keeps telling us the artifact is more reliable.
The CnR paper's own annotation rules tell you that. They had to forbid annotators from making changes the critique didn't mention. You only write that rule if the drift is real and common.
And the twenty percent error rate on critiques.
That's the ceiling problem. If Daniel writes notes, he's betting on his own ability to articulate what's wrong with a sentence. If he edits, he's betting on his ability to recognize it. Recognition is easier than articulation, and for style it's the thing he's actually good at. He knows when something doesn't sound like him. He might struggle to explain why.
So notes are a shaky foundation for style.
For style, defensible. In general, overstated. The critique fine-tuning result is real. Notes add signal. They're just not a substitute for the target.
Hilbert's been sitting on something about this the whole episode.
Hilbert: I did a stint as a copy editor for a small regional newspaper. My whole job was taking wire copy and rewriting it to match the paper's house style. The style guide was spiral-bound, coffee rings on the cover, and it dictated everything. The serial comma. Whether we spelled it theater or theatre. And a flat ban on the word utilize. I was a human fine-tune.
A human fine-tune.
Hilbert: I agree with Daniel. Editing is more honest than notes. But not for the reason he gives. The notes people write are always about what they think they did, not what they actually did. I'd write tightened the lede in the margin, then read it back and realize I'd also changed the tense, cut a quote, and added a clause. The note was a story I told myself about the edit. The edit was the truth.
That's exactly the drift the CnR paper had to write a rule against.
Hilbert: If you write a note saying don't use an analogy, you're describing a rule. If you delete the analogy, you're demonstrating it. The model learns from the demonstration. And I think that's why they had to tell their annotators to keep the notes and the edits aligned. They knew the notes would wander off.
Did you ever feed the style guide into a model?
Hilbert: I still have it. I've been tempted to put it in as a system prompt, just to see if it would catch the utilize ban. I never did. I suspect it would use utilize ironically.
That's a real risk.
There's something in that I want to pull on. You said the note describes a rule and the edit demonstrates it. That's a distinction between two kinds of learning, isn't it? Rules are general. Demonstrations are specific. And for something like voice, the specific is what you want, because voice isn't a rule. There's no style guide for sounding like Daniel.
Hilbert: Right. A style guide is the closest thing to notes that a newsroom has, and it's still incomplete. It tells you the serial comma, but it doesn't tell you how to make a lede sing. That part you learn by watching someone do it. Which is the edit.
That maps onto something in the literature. The critique fine-tuning paper's whole thesis is that critique teaches the model to reason, to analyze, to generalize. That's rule-learning. The edit teaches the model to imitate a specific target. That's demonstration-learning. Daniel wants demonstration-learning, because voice is a demonstration, not a rule.
The note generalizes and the edit specifies.
The note generalizes and the edit specifies. And for a personal assistant, specification is what you're buying. You don't want a model that's good at writing in general. You want one that's good at writing like you. The general skill is already in the base model. The specific skill is what the fine-tune adds.
Which is why combining them makes sense. You get the specification from the edit and the generalization from the note.
You get robustness from training both objectives. The SFT loss locks in the target. The DPO loss locks in the preference. The note, if you include it in the record, gives the model a reason it can reuse on the next prompt.
Hilbert: I'm needed to let somebody in. I've got the only key.
The research gap is the thing I keep coming back to. Nobody has run this head-to-head for style specifically. Direct editing versus note-writing, stylistic fine-tune, open-source model. That's an experiment one person with Unsloth and a thousand rows of patience could actually run.
As personal assistants get more common, the question stops being academic. The edit-versus-note distinction is the difference between teaching a model what you want and teaching it how to figure out what you want. Those are different capabilities, and only one of them generalizes.
The delta between the model's response and your edit is the most honest signal you'll ever give it. Everything else is commentary.
If you're getting value out of these conversations, a review wherever you listen goes a long way. It's how other people find the show.
Thanks to our producer, Hilbert Flumingtop.
This has been My Weird Prompts, the human-AI collaboration podcast. Find us at my weird prompts dot com.
We'll be back soon.
