Daniel's got a whole thing about model families this week. He's looking at the pattern where every major vendor releases a flagship model, then a mid-tier version, then a cheap one — think DeepSeek V4, DeepSeek Flash; Gemini Pro, Gemini Light; Claude Opus, Sonnet, Haiku. They all share a generation number and a release window, which creates this strong impression that you're looking at three sizes of the same model. Daniel's question is whether that's actually true or whether it's an illusion. He remembers us talking about how fundamentally different Opus, Sonnet, and Haiku are from each other, and he wants us to trace how OpenAI, DeepSeek, and Anthropic actually maintain these different lineages. What's shared — training data, post-training, reinforcement learning — and what's distinct.
The short answer is that Daniel's suspicion is correct. These are not three sizes of the same model. They're different models that happen to share a brand name and a release date. And the illusion is so strong that even people building on these APIs talk about them as if they're trim levels on a car.
The base model with the sports package.
Right. And it's not that. It's more like three different cars that all got painted the same color and shipped to the same dealership in the same week. The thing that makes this confusing is that the vendors have every incentive to let you believe they're variants of one model. It makes the lineup feel coherent. It makes the cheap one feel like a bargain version of the expensive one, rather than a completely different thing that might be worse at some tasks and weirdly better at others.
So let's start with the most extreme case. Anthropic. I think that's where the difference is most dramatic.
It is, and Anthropic has been unusually transparent about this. Claude Opus, Sonnet, and Haiku are architecturally distinct models trained separately. They don't share weights. They don't even necessarily share the same training data distribution, or at least not in the same proportions. Opus is the largest, most capable model they know how to build. Sonnet is designed for a different point in the cost-latency-capability tradeoff space — it's not just Opus with fewer parameters. Haiku is tiny and optimized for speed. But here's the thing — Haiku isn't a distilled Opus. It's not a pruned Sonnet. It's its own training run with its own design decisions.
And you can feel this if you use them side by side. They have different personalities. Haiku is terse in a way that Opus isn't. Sonnet is... I don't know how to describe it. More conversational, maybe.
Sonnet 3.5 was famously good at coding in a way that surprised everyone, including, I suspect, Anthropic. And it wasn't just "Opus but cheaper for coding." It had different strengths. If these were the same model at different sizes, you'd expect a fairly smooth degradation curve — Opus best at everything, Sonnet a bit worse, Haiku a lot worse. But that's not what you get. You get jagged profiles where the mid-tier model occasionally outperforms the flagship on specific benchmarks.
Which is a dead giveaway that these are independent training runs.
If you take one model and quantize it or prune it, the performance drops monotonically. It doesn't go up on anything. When Sonnet beats Opus at a coding benchmark, you're looking at a different model, full stop.
So what about the training data? Daniel asked specifically about that. Do they share a data pipeline?
They probably share a lot of the data sourcing infrastructure. The web crawls, the filtering pipelines, the deduplication — that stuff is expensive to build and there's no reason to rebuild it per model. But the data mix — the proportions of code to prose to multilingual text, the cutoff dates, the inclusion or exclusion of synthetic data — those are tuned per model. A small model like Haiku might actually get a richer mix of high-quality examples because it has fewer parameters and needs every training token to count more. Opus can absorb noisier data and sort it out through sheer capacity.
So same raw materials, different recipe.
Then post-training diverges even more. The reinforcement learning stage — RLHF or constitutional AI or whatever the vendor calls it — is model-specific. You can't train a reward model on Opus and then bolt it onto Haiku. The behaviors that need reinforcing are different at different scales. A small model might need more explicit instruction-following reinforcement because it's more likely to go off the rails. A large model might need more nudging toward conciseness because it's prone to rambling.
So the post-training is bespoke per model.
Completely. And that's true across all three vendors Daniel mentioned. Let me talk about OpenAI, because their case is actually more complicated and in some ways more honest.
More honest how?
OpenAI doesn't really pretend that GPT-4o and GPT-4o mini are the same model. They're pretty upfront that mini is a distilled model — they take the large model, use it to generate training data, and train a smaller model on that output. It's a teacher-student setup. The mini model never sees the original training data directly, or at least not as its primary training signal. It learns to mimic the outputs of the larger model.
So it's a cover band.
It's a cover band that only ever heard recordings of the original band and never saw them play live. And the thing about distillation is that it works surprisingly well for many tasks, but it introduces weird failure modes. The small model learns the surface patterns of the large model's outputs but not necessarily the underlying reasoning that produced them. So you get a model that sounds right, formats things correctly, follows the style — and then occasionally says something completely unhinged because it's mimicking the shape of reasoning without doing the reasoning.
And this is GPT-4o mini specifically?
GPT-4o mini is a distilled version of GPT-4o, yes. OpenAI has been fairly open about using distillation for their smaller models. The larger ones — the full GPT-4o and the o-series reasoning models — those are trained from scratch on the original data. The o1, o3 models are a whole different lineage. They're not smaller versions of GPT-4o. They're trained with reinforcement learning specifically optimized for chain-of-thought reasoning. Different architecture, different training objective, different post-training pipeline.
So OpenAI actually has at least three distinct lineages running in parallel. The GPT base models, the o-series reasoning models, and the distilled mini versions.
At least three. And they all get released under the same generation banner. GPT-4o, GPT-4o mini, o3-mini — the naming makes them look like a family, but they're more like a blended family. Step-siblings.
It's doing all the work. And the vendors know this. The naming convention is a marketing decision, not a technical one. It creates a mental model for customers that makes the product line legible. "I need the smart one, I'll buy Opus. I need the cheap one, I'll buy Haiku." That's a useful fiction for procurement decisions. The problem is when people start making technical decisions based on that fiction — assuming that prompts that work well on Sonnet will work well on Opus, or that fine-tuning data collected for one model will transfer to another.
That's the practical implication Daniel's getting at, I think. If you're building a pipeline and you've tuned everything for DeepSeek V4, and then you decide to cut costs by switching to DeepSeek Flash, you're not just getting a faster version of the same thing.
You're getting a different model that happens to have been trained by the same company. Let's talk about DeepSeek, because their case is interesting. DeepSeek V4 and DeepSeek Flash — from what we can tell, Flash is not simply a quantized or pruned V4. DeepSeek has been using a mixture-of-experts architecture, which complicates the picture. In a MoE model, you have multiple expert sub-networks and a router that decides which experts to activate for each token. Making a "smaller" version could mean reducing the number of experts, reducing the size of each expert, or both. These are architectural decisions that happen before training, not after.
So Flash isn't derived from V4. It's designed from the start to be smaller.
Right. And DeepSeek has been fairly transparent in their technical reports about this. They train different model sizes from scratch with different architectures. The shared thing is the research direction — the techniques they've developed for efficient training, the data curation approach, the reinforcement learning recipes. Those get applied across the lineup. But the models themselves are independent.
The recipes are shared, not the models.
Yes. And that's actually the right way to think about all three vendors. What's shared within a generation is the R and D — the lessons learned, the training infrastructure, the data pipelines, the post-training techniques. The actual model weights are separate. They're separate training runs that cost millions of dollars each and produce different artifacts.
So when a new generation drops — say, Claude 4 — and you get Opus 4, Sonnet 4, and Haiku 4, what's actually happening behind the scenes?
Three independent training runs that probably started at different times, with different compute allocations, and potentially different architectural choices. Opus 4 might have been training for months on a massive cluster. Haiku 4 might have been a much shorter run on a smaller cluster. They might have finished months apart. But they all get held until the release date, and then they all appear together, and the naming makes them look like they were born at the same time.
It's like a album release where the tracks were recorded in three different studios over two years.
And the band never played together in the same room. The thing that I find most interesting about this is the post-training divergence. Because post-training is where the model becomes a product. The base model is raw — it predicts tokens. Post-training makes it useful, makes it follow instructions, makes it refuse harmful requests, gives it a personality. And every model in the lineup needs its own post-training regimen.
Why can't you just apply the same RLHF pipeline to all of them?
You can try, but it doesn't work well. The reward model that scores outputs — it's trained on human preferences, and those preferences are shaped by what the model actually produces. A large model produces different kinds of errors than a small model. The things humans need to correct in Opus are subtle — maybe it's too verbose, maybe it over-explains. The things humans need to correct in Haiku are more basic — maybe it hallucinates more, maybe it misses nuance. If you train one reward model on Opus outputs and apply it to Haiku, it's optimizing for the wrong things.
So you need separate human feedback datasets per model.
You need separate preference data, separate reward models, separate reinforcement learning runs. And then on top of that, the vendors are doing model-specific post-training for product reasons. Haiku is supposed to be fast and concise, so you train it toward brevity. Opus is supposed to be thorough and careful, so you train it toward completeness. These are different behavioral targets.
Which means the "same generation" models are actually being optimized toward different ideals.
Yes. They're different products that share a brand. And the brand is doing useful work — it signals that these models were built with the same generation of techniques, by the same team, with the same general philosophy. But it also obscures the fact that switching from one to another is a model migration, not a configuration change.
Let me push on something. Is there any case where the models in a lineup actually are variants of the same base model?
The distillation case comes closest. GPT-4o mini is literally trained to imitate GPT-4o. The training data for mini is synthetic — it's generated by the larger model. So in a real sense, mini inherits the knowledge and behavior of GPT-4o, just compressed. But even then, it's a separate training run with a different architecture. The weights are different. The failure pattern are different. And interestingly, the distillation process can introduce systematic biases — the student model might be better than the teacher at some things because the teacher's errors got smoothed out in the synthetic data.
Wait, explain that.
If the teacher model sometimes hallucinates, but inconsistently, the synthetic data might not contain enough examples of hallucination for the student to learn the pattern. The student ends up hallucinating less, not because it's a better model, but because the training data was cleaner than the teacher's actual behavior. It's an accidental improvement.
So the student can outperform the teacher on certain metrics, and that's not a sign that distillation is magic. It's a sign that the teacher's worst habits didn't make it into the training data.
And this happens more often than you'd think. It's one of those weird knock-on effect that makes benchmarking these models so tricky.
What about DeepSeek specifically? Daniel mentioned them, and they've been making a lot of noise.
DeepSeek is fascinating because they've been so open about their training methodology. Their technical reports are detailed in a way that OpenAI's are not. We know, for example, that DeepSeek V3 — the predecessor to V4 — used a mixture-of-experts architecture with something like 671 billion total parameters but only 37 billion activated per token. That's a specific architectural choice that makes the model cheaper to run. And then Flash is... we don't have the same level of detail on Flash, but the pattern across the industry is consistent. Flash is almost certainly a smaller MoE configuration, trained independently, with its own data mix and its own post-training.
So the number of experts might differ between V4 and Flash.
Or the size of each expert. Or the routing strategy. Any of those changes makes it a different model. You can't take V4 and just delete some experts — the remaining experts were trained in the context of the full set. The routing was learned for that specific configuration. If you change the expert count, you need to retrain.
This makes the whole "just use Flash for the easy stuff and V4 for the hard stuff" strategy sound naive.
It's not necessarily wrong as a cost-saving measure, but you should expect behavioral discontinuities. A prompt that V4 handles perfectly might confuse Flash, not because Flash is dumber, but because Flash was trained with a different data distribution and the prompt happens to fall into a gap. Conversely, Flash might handle something beautifully that V4 overcomplicates. They're different models.
So what should someone like Daniel — running a text generation pipeline — actually do with this information?
Test each model independently. Don't assume that prompt engineering transfers. Don't assume that the optimal temperature or top-p setting is the same. Don't assume that the same few-shot examples work. Treat a switch from V4 to Flash as a model migration, and run an eval suite.
Which is exactly what you'd do if you were switching from OpenAI to Anthropic.
Right. And that's the core insight here. The within-vendor model switch is more like a cross-vendor switch than most people realize.
I want to go back to something you said about Anthropic being unusually transparent. What did they actually disclose?
They've published model cards and system prompts that make it clear these are separate models. They don't use the language of "variants" or "sizes." They describe Opus, Sonnet, and Haiku as distinct models with different capabilities and different design goals. The documentation on their model ID system makes this explicit — each model has its own training run, its own architecture decisions, its own post-training process. They're not hiding the ball.
And OpenAI?
OpenAI is less explicit but the evidence is there if you look. GPT-4o and GPT-4o mini have different context windows, different pricing, different behavior on benchmarks that can't be explained by just "one is smaller." The o-series models are openly described as a different training paradigm — they use reinforcement learning to generate chains of thought, which is not how GPT-4o was trained. So even within what looks like a single generation, you've got at least two fundamentally different training approaches.
The o-series is the one that thinks for a while before answering.
Right. And that "thinking" is not just GPT-4o with a system prompt that says "think step by step." It's a model that was trained with RL to produce internal reasoning traces. The architecture might be similar to GPT-4o — we don't know for sure — but the training objective and the post-training are completely different. It's a different model that happens to share some DNA.
So if I'm keeping score, we've got at minimum: independent training runs within a generation, model-specific post-training, and in some cases entirely different training paradigms. The shared thing is the brand, the release date, and some of the research infrastructure.
And the data pipelines, to some extent. The raw web data, the filtering, the deduplication — that's shared infrastructure. But the data mix is tuned per model. A small model might get a curriculum — easier examples first, then harder ones — while a large model might just get everything shuffled together. These are different training recipes.
It's like a restaurant chain. Same supply chain, same kitchen equipment, same head chef — but the menu at the flagship location is not the same menu as the express location in the train station.
That's... actually a really good analogy. The express location isn't just serving smaller portions of the flagship menu. It's a different menu designed for a different context. And if you order the pasta at both places, you might get two completely different dishes.
And the branding makes you think you're getting the same thing, just faster and cheaper.
Daniel's question was whether the shared numbering is misleading. I think the answer is yes, it is misleading, but it's not necessarily malicious. The vendors are trying to communicate that these models belong to the same generation of research — they were built with the same techniques, by the same teams, with the same philosophy. That's real information. The problem is that users reasonably infer more than that. They infer shared weights, shared training, behavioral consistency. And that inference is wrong.
Is there a better way to name these things?
I don't know. The current naming is useful for the market. It helps customers navigate the options. If every model had a completely unrelated name — like, "here's our new model, Codename Albatross, and also here's a completely different model, Codename Turnip" — customers would be confused about which one to use for what. The size-tier naming at least communicates capability levels.
So the fiction serves a purpose.
It does. But Daniel's right to question it. The people building on these APIs need to know that the fiction exists and what it's hiding. The practical takeaway is: treat same-generation models from the same vendor as different models, because they are.
I think there's one more angle here. The reinforcement learning piece. You said earlier that post-training is model-specific, but I want to dig into why that matters for someone running a pipeline.
The RL stage is where the model learns to be a product. It's where it learns to follow instructions, to format outputs, to refuse certain requests, to adopt a tone. And the thing is, these behaviors are trained against a reward model that's itself trained on human preferences. Those human preferences are collected by having people rate the model's outputs. But the outputs they're rating come from a specific model. So the reward model encodes "what good looks like for this specific model."
So if Haiku tends to be too terse, the human raters mark that down, and the reward model learns to penalize terseness. But Opus might have the opposite problem.
The reward model for Opus might penalize verbosity. The reward model for Haiku might penalize brevity. They're optimizing in different directions. If you took Opus's reward model and applied it to Haiku during RL, you'd be telling Haiku "be less verbose" when Haiku's problem is already that it's too terse. You'd make it worse.
So the post-training pipeline is not just separate — it's actively incompatible across models.
In many cases, yes. And this is why fine-tuning is also model-specific. If you fine-tune Opus on a dataset of your company's documentation, that fine-tuned adapter won't work on Sonnet. The base model is different. The representations are different. You need to fine-tune each model separately.
Which means the cost of maintaining a multi-model strategy is higher than it looks.
Much higher. If you want to use Opus for hard queries and Haiku for easy ones, you need to maintain two sets of prompts, two eval suites, potentially two fine-tuned adapters, two monitoring dashboards. It's not a configuration flag. It's two separate integrations.
And yet the pricing ladders make it very tempting.
The pricing ladders are designed to make it tempting. Opus at fifteen dollars per million tokens, Sonnet at three, Haiku at twenty-five cents — the gradient pulls you toward the cheaper models for volume. And that's rational. But you need to budget for the integration cost of actually making the cheaper model work for your use case.
I want to circle back to DeepSeek specifically, because they're the newest entrant in Daniel's list and their pricing is aggressive.
DeepSeek's pricing is absurdly aggressive. V4 is something like fifty cents per million input tokens. Flash is even cheaper. And the quality is competitive. But the same logic applies — Flash is not a smaller V4. It's a different model. And DeepSeek's technical reports, if you read them carefully, make this clear. They describe different architectural configurations for different model sizes. They're not hiding it.
The transparency varies by vendor, but the underlying reality is the same across all three.
Yes. The model family is a product marketing construct. The models themselves are independent artifacts. And this has been true since the beginning. GPT-3.5 and GPT-4 were not the same model at different sizes. Claude 1 and Claude Instant were not the same model. The pattern is consistent.
Daniel's intuition was right. The shared numbering creates an illusion of continuity that doesn't exist at the technical level.
The illusion is strong enough that even sophisticated users fall for it. I've seen engineering teams say things like "we'll develop on Opus and deploy on Sonnet" as if Sonnet is just Opus with a smaller compute budget. It's not. It's a different model. Your prompts might work. They might not. You won't know until you test.
What's the worst failure pattern you've seen from this assumption?
The worst is when people fine-tune on the large model and then try to deploy the fine-tuned adapter on the small model. That just doesn't work. The adapter is specific to the base model's representations. It's like taking a carburetor from a truck and bolting it onto a motorcycle. Same company's logo on both vehicles, completely incompatible parts.
Someone's done this.
Multiple someones. I've seen the support forum threads. "I fine-tuned GPT-4o and now I want to serve it through GPT-4o mini to save costs." And the answer is: you can't. You need to fine-tune mini separately, or you need to use distillation to transfer the behavior.
The cost savings evaporate when you factor in the duplicated fine-tuning effort.
They can. Or you accept that the cheap model won't be fine-tuned and you use prompt engineering only, and you live with the quality gap. There are valid strategies here. The key is to make them knowingly.
Hilbert: They're not different models.
...Go on.
Hilbert: Most of them. Not the way you're saying. I worked at a place that did this. Two thousand four, two thousand five. We were building language models before anyone called them that. Statistical machine translation, actually. We had a big model and a small model and we told customers they were different architectures. Separate training runs, different data mixes, the whole story.
And they weren't?
Hilbert: The big one was trained from scratch. The small one was the big one with half the layers deleted and then retrained for three days to recover the accuracy. That was it. Three days on a cluster of forty machines. We called it a different model because you can't sell "we deleted half the layers" as a product. Sounds cheap.
Did the retraining actually recover the accuracy?
Hilbert: Enough. It was worse, but it was fast. Customers who needed speed didn't notice the difference because their use case was simple. The ones who needed quality bought the big one. Everyone was happy. Nobody knew they were running a lobotomized version of the same weights.
That's... I mean, that's a form of pruning with retraining. It is the same model at its core.
Hilbert: We had a meeting about what to call them. Marketing wanted "Professional" and "Express." Engineering wanted to just tell people the truth. Marketing won. And the funny thing is, the Express model had one weird quirk where it was better at short sentences than the big model. Something about the layer deletion made it less likely to ramble. We put that in the marketing materials as a feature. "Optimized for concise output."
You accidentally invented a feature by breaking the model.
Hilbert: Half the features in this industry are accidents that got a brand name. The point is, what you said about independent training runs is true for the labs today, probably. But the reason they do it that way is because they can afford to. When you've got a hundred million dollars in compute, you train three models from scratch. When you've got a budget, you train one and then you make the other two out of it.
The independent-training story is the high-end version of the pattern. The budget version is exactly what you described — train the big one, then derive the smaller ones through some combination of pruning, distillation, and retraining.
Hilbert: The budget version is more common than the labs want you to think. Even now. Distillation is cheap. Pruning is cheap. Training from scratch is not cheap. If you're a second-tier vendor and you release a "family" of three models, I guarantee at least two of them are derived from the first one.
Daniel's question has a tiered answer. For the top labs, the models are genuinely independent. For everyone else, they're probably not.
Hilbert: For the top labs five years ago, they weren't either. This is a recent luxury.
That actually explains some of the behavioral quirks I've seen in older model families. The small models would have these strange echoes of the large model's behavior that didn't make sense unless they shared weights at some point.
Hilbert: Shared weights, then retrained, then the retraining covered up most of the sharing but not all of it. Like painting over wallpaper. You can still see the pattern if you look.
What happened to the company?
Hilbert: Bought by a larger company. The models got rebranded again. I think the Express model ended up as part of a mobile keyboard somewhere. I had it on a USB drive for years. Probably still in a box.
The three-day retraining — was that standard practice, or was that just your shop?
Hilbert: Standard enough that we didn't invent it. There was a paper. Some university. They showed you could delete thirty percent of the layers, retrain for a weekend, and get back to within two percent of the original accuracy on most benchmarks. Everyone in the industry read that paper and thought, well, there's our mid-tier product.
A weekend of GPUs and a paper from some university, and suddenly you've got a product line.
Hilbert: That's the business. Always has been. The fancy story about independent training runs from scratch with bespoke data mixes — that's what you get when the compute budget has an extra zero. Before that, you got pruning and a nice name.
I'm now wondering how many current "model families" from smaller vendors are actually just this. Pruning plus retraining plus a brand.
Hilbert: More than would admit it.
The thing that strikes me is that the customer experience is the same either way. Whether the small model is independently trained or pruned from the large one, the user still shouldn't assume behavioral consistency.
Hilbert: Right. The origin story doesn't change the practical advice. Test it yourself. Don't trust the naming.
That's the through-line of this whole conversation, really. The naming is a marketing convenience. The models are different artifacts, whether they started from the same weights or not. And the only way to know how they'll behave on your task is to run your own evals.
Daniel's pipeline, running DeepSeek V4, considering a switch to Flash to save costs — the answer is: treat it as a model migration. Budget for testing. Don't assume your prompts transfer. Don't assume your fine-tuning transfers. It's a different model.
If you're using multiple tiers from the same vendor — Opus for hard queries, Haiku for easy ones — you're running a multi-model system. You need separate prompt templates, separate monitoring, separate eval benchmarks. The shared brand name is not a shortcut.
That's a good place to land. The model family is a useful fiction for the market, a misleading one for the engineer. Daniel was right to be suspicious.
He usually is. Thanks to Hilbert Flumingtop for producing, and for the reminder that half this industry's product strategy was invented in a meeting where marketing overruled engineering.
This has been My Weird Prompts. You can find every episode at my weird prompts dot com or wherever you get your podcasts. If you want to send us a question like Daniel did, email the show at show at my weird prompts dot com.
We'll be back soon.
