Fifty-nine percent of the stories in one GPT-four-generated dataset began with Once upon a time. Daniel, that's a remarkably expensive way to discover one sentence.
That's the problem I want to get into. If I ask for ten thousand synthetic examples, I don't want the same ideas wearing different names. How do I get actual variety, and how do I check it before paying for the whole batch?
Correct examples can still make a terrible training set. Each one passes inspection, the labels look sensible, and together they teach a classifier that complaints always begin with I am writing to express my dissatisfaction.
Nobody's furious until they've drafted a letter.
Then a real user writes, charged twice again, and your beautifully trained classifier struggles. It learned the generator's preferred performance of complaining rather than the full range of complaints.
I'd like to separate the things that change that from things that just make the output look more varied.
Names, sentence lengths, elaborate imaginary customers. You can spend a lot decorating the same example.
The important distinction is between changing the wording and expanding the situations the training set represents. You need both, but the second is much easier to overlook.
Imagine a thousand paraphrases covering fifty underlying situations. That does not suddenly become a thousand situations because you put each paraphrase on a separate spreadsheet row. Though I appreciate the spreadsheet's optimism.
One correction to the shorthand people use here. Fifty concepts doesn't necessarily mean fifty classifier labels. You might have the same small label set throughout. What's limited is the variety of evidence associated with each label.
Particularly the evidence that separates neighboring labels. Billing complaint versus cancellation request. If all your cancellations mention moving abroad, you've accidentally made geography part of the decision.
Is the generator repeating itself because the prompt is too narrow, or because each request doesn't know what came before?
Both. The cross-batch problem persists even with fairly open prompts. Ask for five exam questions repeatedly, and every independent call can return to the same comfortable territory. It hasn't watched your dataset fill up.
In the cross-batch mode collapse experiments, thirty-four percent of questions in the final fifty batches were near-duplicates of the initial batches. That's after two hundred generation calls.
Two hundred trips to the same stationery cupboard.
The severity depended on the domain. Sustainable packaging had much less repetition than educational content. You shouldn't import one collapse percentage into every application.
I also wouldn't assume perfect labels solve this. Quality helps when tomorrow's inputs resemble today's. Coverage matters when tomorrow introduces a different way of expressing the same need.
The quality, diversity, and complexity framing captures that well. Quality supports familiar-distribution performance. Diversity helps beyond it. Complexity can help both, provided the examples remain useful.
Meaning a spotless dataset can still be a very narrow education.
Yes. Corn, would you favor the deliberately narrow set if you knew it matched the deployment traffic?
For a narrow job, probably. I wouldn't add exotic complaints to a machine that only routes refund requests. I'd still vary how people ask, what they omit, and which almost-refunds should be rejected. Narrow scope doesn't excuse identical evidence.
Temperature is the obvious knob I'd think of. Does turning it up buy enough variety to matter?
Usually less than people expect. Temperature changes token probabilities. It doesn't maintain a register of situations you've already covered. Raise it and you can get unusual wording before you get useful new concepts.
The complaint now contains a lighthouse.
Sometimes the extra probability goes into invalid continuations. The sampling comparisons found that flattening the distribution could reduce validity before recovering meaningful diversity. Top-token filtering didn't repair the underlying coverage problem in those tests.
That sounds like it's still useful within limits, just not the main strategy.
That's how I'd treat it. Tune it for acceptable generation quality. Don't ask it to do the work of dataset planning. Some APIs remove the choice altogether. GPT-five-mini doesn't expose temperature or top-p in the setup discussed in those comparisons.
A diversity knob sold separately from the diversity problem.
You can still change the requested concepts, the demonstrations, and what information persists between requests. Those are available even when sampling controls aren't.
Herman, you're about to tell me the model needs a committee.
Not yet. It can have a list first.
I'll mark that as restraint.
What about personas? If I specify different people with different backgrounds, shouldn't that produce different ways of speaking?
It should when the background changes the task. A customer who knows the product's terminology may describe a fault differently from someone who calls every component the little thing.
Persona prompting can improve lexical diversity, especially with larger models. The disappointing result is that detailed biographies don't necessarily add much. Coarse personas performed about as diversely as fine-grained ones in those comparisons.
Six paragraphs of childhood, same refund request.
Fine-grained details also produced minimal gains over simply imposing a length cutoff. That matters because output length itself changes many lexical diversity measures.
Then a persona experiment could look successful because it made the examples longer?
Potentially. If one approach gives you more words, it has more opportunities to introduce unique words. You need length-aware comparisons before declaring the characters more diverse.
I'd budget for task-relevant differences instead. Familiarity, urgency, whether the person knows what happened, whether they blame the wrong cause. Those alter what the example requires a classifier to notice.
Would you discard personas entirely, Corn?
No. I'd discard the decorative biography. If you're generating accessibility support requests, a relevant accessibility constraint matters. A fondness for pottery probably doesn't. Unless the printer is made of clay.
Is using a huge synthetic persona library any better than making them up in the prompt?
It gives you more starting points, but quantity doesn't guarantee breadth. PersonaHub-derived prompts were less diverse than human-written prompts across every metric evaluated in that comparison.
That doesn't mean every human prompt wins. It means a large library carrying a diversity label hasn't earned your trust automatically.
You do enjoy a large library carrying a label.
I enjoy a large library carrying documentation.
The distinction has occupied most of our shelf space.
Your reading pile is horizontal because you sleep on it. Neither arrangement proves coverage.
Suppose I drop the biographies and start with a seed list of topics. What's the limitation there?
Whoever writes the list has already decided what the world contains. Ask a model for a hundred complaint topics and you may receive a hundred subdivisions of delivery delays.
A large, diverse seed set helps. Seed-and-expand methods have worked for a long time. The shallow version just samples a seed randomly and hopes the resulting examples remain distinct.
I'd want each example conditioned on a different seed. Does that solve the random repetition?
It solves seed reuse if you track allocation. It doesn't guarantee semantic distinctness. Two differently worded seeds can lead to the same scenario, and two different seeds can get flattened into the same answer pattern.
Late parcel and parcel not yet arrived have entered the database separately.
The output still needs checking against what you've already generated. Seeds are inputs to coverage, not certificates of coverage.
Rotating demonstrations seems more direct. Show different examples on each request and stop the model copying one template.
That can help. Demonstrations influence topic, formatting, sentence shape, and reasoning depth. Rotation reduces dependence on a single little patch of examples.
Unless every rotation contains three polished paragraphs and a courteous closing. Then you've varied the topic while teaching everyone to write like a procurement officer.
Generating several samples together can also help. In the multi-sample prompting comparisons, downstream F1 gains ranged from six to nearly forty-four percentage points across tested settings.
That's a large range. What do I take from it without expecting the largest gain?
That batch structure can materially affect utility, and that the effect is task-dependent. Not that requesting more examples guarantees a particular gain. A shared context lets the model see nearby alternatives, which independent calls can't do.
Within the batch, it can notice it has already said something. Between batches, it forgets again.
Prompt optimization can improve lexical diversity while producing mixed downstream results. More varied wording is not automatically better training material.
Then I'd compare rotated prompts on the actual task, not just count unique phrases.
Especially if your optimizer discovers that spelling everything strangely makes the vocabulary chart look magnificent.
The weak approaches aren't worthless. They lack feedback about the growing dataset. Temperature changes sampling. Personas change framing. Seeds change starting points. None automatically knows which useful regions are already full.
You want a generation process that changes its mind after seeing its own work.
Multi-stage generation makes that manageable. Separate planning from rendering. Establish categories, inspect their coverage, assign scenarios within them, then generate actual examples conditioned on those scenarios.
Make that concrete for a classifier. What would exist between the category list and the final text?
A short scenario specification. Say, a customer sees two charges, one is pending, and they mistakenly think both settled. That is different from two settled charges, even if both messages say charged twice.
You can track those distinctions before generating prose. Then vary the expression without losing the distinction. Abbreviated message, technical message, uncertain message. Keep the label tied to the evidence, not to the chosen style.
Wouldn't those dimensions create a huge grid?
They can. You shouldn't fill every combination merely because it's available. Some combinations are unnatural; others don't affect your task. The planning stage needs judgment about useful coverage.
A grid can generate nonsense with excellent attendance records.
MetaSynth goes further by categorizing and tracking generated instances in memory. The category information conditions subsequent generation. You're not just rotating examples, you're recording what those examples contribute.
Its template-prompted data degraded Mistral-seven-B's performance, even with previous generations and changing real demonstrations included. The more diverse generation approach improved the same model without mixing real data into training.
That last part could become a fairly sweeping claim. How far does it actually go?
It establishes that real-data mixing wasn't necessary for those gains in those tasks. Reported improvements reached about four percent in finance and nearly fourteen percent in biomedicine. It doesn't establish that synthetic coverage replaces real-world validation.
Especially when the generator wrote the exam and chose what would be on it.
Other approaches track which clusters have already supplied demonstrations and select different clusters next. An uncertainty tracker can guide that selection. The useful feature is deliberate movement through the space, rather than random seed replacement.
Can I get some of that benefit without building a full agent pipeline?
Verbalized sampling is a relatively cheap intervention. Ask for candidate responses along with estimates of their probabilities, rather than asking directly for one response. That can expose alternatives the model wouldn't otherwise select.
The estimates aren't measured probabilities, though.
Correct. They're the model's verbal estimates. Treat them as a prompting device, not calibrated likelihoods. In creative-writing tests, diversity increased roughly one-point-six to two-point-one times over direct prompting.
Does that transfer to labeled examples?
It's a promising candidate to test, not a guaranteed transfer. A rarer story premise can be valuable. A rarer interpretation of a fixed label can just be wrong.
Novelty has no obligation to understand your schema.
The more complete approach is Dynamic Context Evolution. It combines requesting less-obvious ideas, persistent semantic memory, and changing the next prompt based on what the memory contains.
Changing it how?
Suppose billing errors dominate the accepted examples while account-access problems remain sparse. The next request can explicitly seek different account-access scenarios. The embedding memory checks candidate meaning against earlier outputs across calls.
Rejection alone would keep buying duplicate attempts. The prompt needs to stop ordering what the database already has.
In those experiments, deduplication and prompt evolution weren't sufficient individually. Their combination mattered. Observed collapse was zero under the combined approach, compared with about five-point-six percent for naive prompting.
Zero under that measurement, rather than every example being completely original.
Under that measurement and those tested conditions, yes. The semantic clusters were also more stable. The reported generation cost was about fifty cents per thousand candidates.
Candidates. The rejected ones still send invoices.
Right. Well... accepted, useful examples are the denominator you actually care about. Infrastructure and evaluation also belong in a real budget.
Before the full run, what would you measure on a pilot? I don't want a beautiful diversity score that's irrelevant to the classifier.
First I'd decide what counts as a good example. Then I'd compare diversity among the examples that pass. Otherwise gibberish is an extremely competitive supplier.
That's effective semantic diversity. Quality and diversity interact. Preference-tuned models can score lower on raw diversity but higher once you count only acceptable outputs.
Corn, would you trust an embedding score enough to choose between two pilots?
Not alone. I'd want to know whether it's separating meaningful scenarios or merely different nouns. One batch might cover several businesses while repeating the same complaint. Another might stay in one business and cover different problems.
Lexical metrics are useful alarms. Distinct-n measures the share of unique word sequences. Self-BLEU compares examples against one another, with lower overlap suggesting more variety. Neither proves semantic breadth.
Names could inflate those too.
Easily. For meaning, compare embeddings and inspect clusters. Vendi Score uses the similarity matrix to estimate effective diversity through its eigenvalues. You don't need to recite the mathematics to use it, but your similarity representation determines what it notices.
Finally, mathematics spared an oral presentation.
DCScore uses relationships between samples to evaluate how distinguishable they are. It can cost less computationally than eigenvalue-based approaches. Task2Vec offers another, task-oriented representation rather than relying only on surface text.
Would you run all of them?
I'd choose complementary checks. One for wording, one for meaning, plus inspection of which task situations actually appear. Five scores measuring almost the same thing won't catch five kinds of problem.
Pilot successive batches too. A few hundred candidates can expose repetition, but an initial batch doesn't tell you whether later batches keep adding coverage. Track the accepted new scenarios as generation continues.
Would you stop the run when that number flattens, Corn, or revise the prompt?
Revise it, after removing the duplicate fire exits. I've generated evacuation notices for my imaginary underground pear museum while you've been talking. Nobody else is involved. The museum has no visitors, but I require one notice for each of its forty-three exits. Forty-two. I don't count an exit that opens into another pear gallery.
The pears remain indoors.
They are exhibits, Corn. I asked for different notices because identical notices make it difficult to identify the door in a report. Thirty-one began Remain calm. I replaced the seed with a different emergency for each door. They still began Remain calm. The emergency changed. The instruction didn't.
That opening could be perfectly appropriate across different emergencies. Its repetition alone doesn't show the notices have identical content.
The content was different. One dealt with water entering the west gallery. Another dealt with the ceiling descending during opening hours. I used my snout to keep the desk tablet awake while I checked them. The model put an assembly point beside every exit, including the exits into pear galleries. Those ones failed.
Your evacuation sends people deeper into the exhibition.
Only on the rejected notices. Also, Herman, distinct-n is unique n-grams divided by total n-grams, not quite the share of unique word sequences unless you specify their length. You gave the verbal probabilities more room than that. I've corrected the caption.
Fair correction. Your rejected notices also change the coverage count. Some of those emergencies may now have no acceptable example at all.
The museum could have forty-two doors and no usable instruction for the descending ceiling.
Forty-one usable doors. The ceiling notice was assigned to two.
That's a useful distinction for the pilot. If I remove bad examples, I need to look again at what situations survived, not keep the original coverage report.
The quality gate can reshape the dataset. If difficult situations fail more often, the accepted batch can quietly become simpler and narrower.
The common misconception is that a handful of convincing examples tells you the dataset is convincing.
Each example can be excellent while the collection repeats itself. Measure relationships across the batch, and whether later batches add useful coverage.
I'll take that over a promise that one temperature setting fixes it.
The next question is where to set the quality threshold without filtering away the difficult cases you needed. A complete list of scenarios means little if the usable examples leave several doors uncovered.
Thanks to Hilbert Flumingtop for producing the show.
If this was your kind of episode, go back for episode sixteen, On Deepfakes, SynthID, And AI Watermarking. This has been My Weird Prompts, the human-AI collaboration podcast.
If you enjoy the show, leave a review. It helps other curious people find us.
See you soon.
See you tomorrow.
