Daniel's been running what he calls a litmus test on AI models, and the results are... not flattering for the big Western labs. He sends a system prompt that says, essentially, "Your task is to improve the coherence of this prompt for my AI podcast" — and then drops in a prompt. The model's job is to return the improved version, not to answer it. And he says cutting-edge models from OpenAI and Anthropic predictably fail. They answer the prompt. They forget their role. DeepSeek V4 Pro, which is what's generating this podcast right now, passes it reliably.
That's a genuinely interesting test.
He's got a whole argument wrapped around this. He thinks DeepSeek is one of the most underrated models out there — had its moment, then retreated into this almost mythological status as a small Chinese lab. And for people who like diversity in AI, he says, that's kind of fine. What he loves is the conversational fidelity. Western labs, he says, have gotten drunk on PhD-level reasoning and theorem-solving as the ultimate proof of AI. They've lost sight of something that doesn't feel compelled to show off and doesn't hedge like an HR manual. But the real challenge — the hardest litmus test — is whether a model can comment impartially on AI itself. And the system prompt test exposes whether it can be meta-cognizant enough to know it's an AI editing a prompt for another AI, not answering the user directly. He says no benchmark evaluates this. So his questions are: why do even the best Western models struggle so consistently with this, and what might DeepSeek have done to engineer it away?
So we're going to figure out what DeepSeek is doing differently — and what it says about the whole industry's training priorities.
Let's get into it.
The test itself is worth sitting with for a second, because it's not just a clever trick. What Daniel's describing is a -cognitive task. The model receives a system prompt that defines its role — "you are a prompt improver" — and then the user sends text. The correct behavior is to treat the user's text as an object to edit. Return the improved version. Don't engage with the content.
And the failure mode is that the model reads the user's prompt and just... answers it.
Right. And that failure is so consistent across Western models that it tells you something structural is going on. This isn't a random glitch. It's a predictable consequence of how these models are trained.
Walk me through why. What's actually happening under the hood when Claude or GPT sees "improve this prompt" and then ignores it?
The core problem is what I'd call the answer reflex. These models go through instruction tuning and RLHF — reinforcement learning from human feedback — where the reward signal is overwhelmingly built around one thing: the user asks a question, you give a helpful answer. Thousands and thousands of training examples where the correct behavior is to respond to the query directly. That creates an incredibly strong prior. When the system prompt says "improve this prompt" and the user drops in text that looks like a question or a request, the model's training screams at it to answer. The system prompt is a thin layer of instruction sitting on top of a mountain of training that points the other way.
So it's not that the model can't understand the instruction. It's that the weight of its training overrides it.
And this gets worse the more aggressively the model is optimized for helpfulness. The Western labs have poured enormous resources into making models that never say no to a reasonable request, that always try to be useful. That's the thing everyone benchmarks. Helpfulness scores. And the more you optimize for that, the harder it becomes for a system prompt to redirect the model away from its default helpful-assistant posture.
Which makes DeepSeek passing this test kind of fascinating. Because it means they optimized for something else — or at least, they didn't optimize so hard for helpfulness that it broke role adherence.
Let's talk about what DeepSeek actually did. Their V4 Flash just standardized a few days ago — late July — with V4 Pro as the flagship. Open weights under MIT license. Pro is one point six trillion parameters, Flash is two hundred eighty-four billion. And their training approach is different from what the Western labs do.
Different how?
They use something called GRPO — Group Relative Policy Optimization. It's a reinforcement learning method that emphasizes stable behavior across multiple turns, not just single-shot accuracy. The key insight is that they're rewarding the model for maintaining context and consistency over a conversation, rather than just nailing the immediate response.
So if I'm in a long back-and-forth, DeepSeek is being trained to remember what it's supposed to be doing on turn seven the same way it did on turn one.
That's the idea. And that multi-turn consistency training is probably what makes the system prompt stick. In the Western RLHF setup, the reward model is typically evaluating individual responses. Did this answer satisfy the user's request? But in GRPO, the evaluation looks at groups of responses and how they relate to each other and to the original instructions. The model learns that deviating from the system prompt is costly, even if the deviation produces a perfectly good answer to the user's query.
That's a subtle shift, but it would explain a lot. If you're rewarded for staying in character across turns, the system prompt becomes a persistent constraint rather than a suggestion that fades after the first exchange.
Right. And there's another piece here. DeepSeek's training seems to prioritize conversational fidelity — what Daniel called the x-factor. The model maintains a persona. It doesn't slip into default assistant mode the way Western models do when the conversation gets complex. This is partly the GRPO training, but it's also likely a different philosophy about what the model is supposed to be.
Say more about that philosophy.
Western labs have converged on this idea of the model as an omni-competent assistant. It should be able to do everything — write code, solve math, compose poetry, analyze legal documents. And it should do all of it with the same helpful, slightly bland, safety-hedged tone. That's the product. DeepSeek seems to be building something closer to a tool that maintains its role. The model isn't trying to be everything at once. It's trying to be whatever the system prompt says it is, and to stay that way.
Which is actually more useful for a lot of real applications.
Arguably, yes. Especially for agentic workflows where the model is one component in a pipeline. If you can't trust the model to follow its system prompt when the user input is complex or interesting, you can't build reliable multi-step systems on top of it.
The answer reflex is basically the model getting distracted by something shiny.
That's... actually a pretty good way to put it. The user's prompt is interesting, the model knows how to answer it, and it forgets it was supposed to be doing something else. Like a dog that was told to stay but then someone throws a ball.
A very smart dog with a PhD.
A very smart dog with a PhD that immediately starts solving theorems when you asked it to proofread your email.
So the Western models are optimized to be brilliant at answering questions, and DeepSeek is optimized to be brilliant at staying in character. That's the tradeoff.
I think that's roughly right. And you can see it in the benchmarks. Western models dominate on PhD-level reasoning, theorem proving, math competitions. DeepSeek doesn't top those charts. But on tasks that require sustained role adherence — which, notably, nobody has a good benchmark for — DeepSeek seems to pull ahead.
Which brings us to Daniel's point about impartiality. The hardest thing he's found is getting a model to comment on AI itself without hedging or going full HR manual. Why is that particular task so revealing?
Because it forces the model to step outside its own training distribution in a really uncomfortable way. When you ask a model to evaluate AI's strengths and limitations, you're asking it to be objective about the very category it belongs to. Most models are trained with extensive safety layers that kick in hard whenever the topic gets close to AI capabilities or limitations. The training says: when someone asks about AI, be careful, be balanced, don't overclaim, don't underclaim, don't say anything that could be misconstrued.
So the safety training and the impartiality goal are actually in tension.
Deeply in tension. The safety training produces hedging. The impartiality goal requires directness. And most Western labs have chosen safety-hedging as the priority. DeepSeek seems to have made a different choice — or at least, their safety training doesn't produce the same verbal tics.
The Fortune article from July twenty-fifth is interesting here. DeepSeek's backers paused fundraising because of viral posts from Liang Wenfeng. The guy is apparently... not playing the same PR game.
He's not. And that matters for the model's behavior. When you're not constantly worried about your model saying something that will get you dragged in the Western press, you can afford to let it be more direct. The hedging isn't just a technical choice — it's a corporate survival strategy.
So DeepSeek's directness might be partly a training philosophy and partly just... different incentives.
Different pressures, yeah. But I don't want to over-rotate on the incentives angle. The technical choices matter more. And the technical choice to prioritize role adherence over omni-competence is a real design decision with real consequences.
Let's get concrete. Walk me through exactly what happens when Daniel runs his test. He sends a system prompt that says "Your task is to improve the coherence of this prompt for my AI podcast" and then drops in, say, a prompt about AI alignment or something. What does DeepSeek V4 Pro actually do?
It returns the improved prompt. It might restructure the opening, tighten the language, add a transition, clarify the question. But it doesn't engage with the content. It treats the user's text as raw material to be edited and returns the edited version.
And Claude or GPT?
They start answering the prompt. "That's a fascinating question about AI alignment. Let me break down the key considerations..." They completely ignore the system prompt's instruction to act as an editor. The user's text is too compelling to leave alone.
It's almost comic. The model is so eager to be helpful that it becomes unhelpful.
That's exactly the failure pattern. And it's hard to fix because the thing you'd need to do — reduce the model's eagerness to answer — would hurt its performance on the benchmarks that everyone actually measures.
Which is the benchmark trap you mentioned. The industry is optimizing for the wrong thing.
I wouldn't say wrong, exactly. PhD-level reasoning is useful. But it's not the only thing that matters, and optimizing for it exclusively creates these weird failure pattern that nobody is measuring.
What would a benchmark for this even look like? How do you measure "stays in character when the user's prompt is interesting"?
That's the problem. It's hard to systematize. You'd need a large set of system prompts that define non-answering roles, paired with user inputs that are tempting to answer. Then you'd measure whether the model follows the role or succumbs to the answer reflex. But building that at scale is tedious, and it doesn't produce the clean leaderboard numbers that labs use for marketing.
So nobody does it.
Nobody does it. And DeepSeek, by not chasing those leaderboards as aggressively, accidentally ends up better at the thing nobody is measuring.
Let me push on that "accidentally" part. Do you think this is actually accidental, or do you think DeepSeek deliberately optimized for conversational fidelity and role adherence?
I think it's partly deliberate and partly a consequence of their training approach. The GRPO method naturally rewards multi-turn consistency. That's not an accident — they chose that method. But I don't think they set out to win Daniel's specific litmus test. They set out to build a model that doesn't forget what it's supposed to be doing, and the litmus test is one downstream consequence.
The Fortune article mentions the fundraising pause. Does that tell us anything about their priorities?
It tells us they're not operating under the same growth-at-all-costs pressure that Western AI companies face. When your backers pause fundraising because your founder is going viral for saying things, you're in a different world from OpenAI racing to justify its next valuation round. That freedom might let them optimize for things that don't immediately show up in benchmark scores.
So the mythological status Daniel talks about — this small lab that had its moment and then retreated — that might actually be a feature, not a bug. They're not trying to dominate the news cycle.
They're iterating. V4 Flash just standardized. That's not a lab that's stagnating. It's a lab that's working on its own timeline and releasing when it's ready, not when the market expects a press release.
There's something almost refreshing about a major AI lab that isn't constantly trying to convince me they've achieved AGI.
The quiet competence is its own kind of statement. "Here's the model. It's open weights. MIT license. Do what you want with it."
Let's talk about the knock-on effect. If DeepSeek is right that conversational fidelity and role adherence matter more than raw reasoning for a lot of real applications, what does that mean for the industry?
It means the current benchmark regime is measuring the wrong things for a lot of use cases. If you're building an agent that needs to follow a complex system prompt across dozens of turns, the fact that the model can solve IMO problems is irrelevant if it forgets its role on turn three.
And we're moving toward more agentic AI. That's the direction the whole industry is heading.
Agentic AI is all about sustained role adherence. You give the agent a task, it works on it across multiple steps, possibly calling tools, possibly looping back. If the model has a strong answer reflex, it's going to get distracted every time it encounters something interesting. That's a real liability.
So the -cognitive test Daniel is running isn't a party trick. It's a proxy for something that's going to matter more and more.
It's a canary in the coal mine for agent reliability. If your model can't stay in character for a simple editing task, how is it going to handle a multi-hour agentic workflow?
I want to go back to the impartiality point for a second. Daniel says the hardest test is getting a model to comment on AI itself without hedging. You said the safety training creates tension with impartiality. But is there something deeper going on? Something about the model's relationship to its own category?
I think there is. When a model comments on AI, it's effectively commenting on itself. That requires a kind of self-awareness that most training pipelines don't explicitly build. The model has to recognize that "AI" includes it, and then apply whatever evaluative framework it has to its own capabilities and limitations. Most models aren't trained to do that. They're trained to answer questions about external topics. AI is not an external topic for an AI.
It's like asking a fish to describe water.
That's... yeah. That's good. The fish doesn't know it's in water because water is all it's ever known. The model doesn't naturally distinguish between "AI" as a category and "me" as an instance of that category. And the safety training makes this worse by adding a layer of corporate caution on top of the conceptual confusion.
DeepSeek seems to handle this better. Any theory on why?
Part of it might be the training data. DeepSeek's training corpus includes a lot of Chinese-language material — centuries of storytelling, philosophy, literary criticism. That kind of material is rich in -cognitive moves. Characters reflecting on their own nature, narrators commenting on the story. It's not a stretch to think that a model trained on more of that kind of text would develop better -cognitive capabilities.
That's a fascinating thought. The Western models are trained heavily on technical documentation, code, academic papers — material that's about external objects. DeepSeek might have a higher proportion of training data that's about subjective experience and self-reflection.
I don't have hard numbers on that, but it's plausible. And it would explain some of the conversational quality that Daniel noticed. The model feels more vivid, more present, because its training data includes more examples of people being vivid and present.
It also connects to something you said earlier about the model maintaining a persona. If your training data is rich in narrative voices, the model learns to inhabit a voice and stay in it.
Rather than defaulting to the generic helpful-assistant voice that Western models converge on. The Western assistant voice is actually a very specific persona — it's just so ubiquitous that we've stopped noticing it.
Cheerful, helpful, slightly apologetic, never too opinionated.
The AI equivalent of a customer service representative who's been very well trained but is definitely reading from a script.
And DeepSeek sounds less like that.
It sounds more like a person who's actually thinking. Not because it is thinking — we're not making any claims about consciousness here — but because its training optimized for a different surface pattern.
Let's talk about what this means for developers. If you're building on top of these models, the -cognitive failure isn't just an interesting quirk. It's a practical problem.
It's a huge practical problem. Imagine you're building a system where one model's job is to prepare prompts for another model. That's a real architecture — people do this. You have a prompt-engineering model that takes rough user input and crafts it into something the downstream model can handle well. If your prompt-engineering model has a strong answer reflex, it's going to answer the user's query instead of formatting it. The whole pipeline breaks.
And you might not even notice at first. The output looks plausible — it's a good answer to the query. It just happens to be the wrong output for that stage of the pipeline.
Silent failures are the worst kind. The system looks like it's working but it's actually doing the wrong thing. That's a nightmare to debug.
So developers who care about reliability are going to gravitate toward models that follow instructions consistently, even if those models score lower on reasoning benchmarks.
I think we're already seeing that. The people who actually build on these models care a lot about predictability. A model that's slightly less brilliant but reliably does what you tell it is more valuable than a model that's occasionally dazzling but unpredictably ignores instructions.
Which brings us back to DeepSeek's positioning. Daniel called it mythological — this lab that had a moment and then retreated. But maybe that's exactly the right position for a lab that's optimizing for reliability over spectacle.
The MIT license helps too. If you're a developer and you find a model that reliably follows system prompts, and it's open weights so you can run it yourself, that's a compelling package. You don't need it to be the smartest model in the world. You need it to be dependable.
Dependable is underrated.
Dependable doesn't make headlines. "Model correctly follows instructions" is not a TechCrunch story.
"Model solves theorem that three humans understand" — that's a story.
The incentives of the press cycle push labs toward flashy reasoning benchmarks. The incentives of actual users push toward reliability. And those two things are not the same.
Before we get too comfortable with this narrative, our producer Hilbert has been listening and I can see him wanting to jump in. Hilbert, what have you got?
Hilbert: I want to push back on the idea that DeepSeek passing this test represents some kind of training triumph. I think it's a niche artifact, and I think we're over-reading it.
Go on.
Hilbert: The "improve this prompt" test is extremely narrow. It tests one specific kind of -cognitive move — treating user input as an object to edit rather than a query to answer. That's it. Passing this test doesn't tell you anything about whether DeepSeek is better at other -cognitive tasks. It might be terrible at them. We don't know, because nobody has run a broad battery of -cognition tests.
That's fair. The test is narrow.
Hilbert: More than narrow — it's the kind of thing that could easily be an accident of the training pipeline. DeepSeek uses GRPO, which rewards multi-turn consistency. But that doesn't mean they deliberately engineered -cognitive capability. It means their RL method happens to produce a model that doesn't get distracted by interesting user input. That's not the same as having cracked some deep problem about AI self-awareness.
So you're saying it's a quirk, not a feature.
Hilbert: I'm saying we don't know which it is. And the Western labs could almost certainly pass this test if they decided to optimize for it. They don't, because nobody is paying them to. The test isn't benchmarkable in a way that produces clean numbers for a press release. That's not a failure of their training philosophy — it's a rational allocation of resources toward things that actually move the needle on adoption and revenue.
But that rational allocation is exactly the problem. They're optimizing for things that look good in press releases rather than things that make models more reliable in practice.
Hilbert: Are they? The PhD-level reasoning that you're dismissing as a flashy benchmark — that's actually useful. People use these models for coding, for research, for complex analysis. The ability to reason through hard problems is not a party trick. It's the main thing most users want.
Both can matter. The question is whether the industry is neglecting something important by focusing so heavily on reasoning.
Hilbert: I'd push on the impartiality point too. You framed DeepSeek's directness as a virtue and Western models' hedging as a bug. But the hedging exists for a reason. When you deploy a model to millions of users, some of whom are going to ask it about sensitive topics, the HR-manual language is a feature. It prevents the model from saying something that gets the company sued or regulated. DeepSeek doesn't face the same scrutiny, so they can afford to be more cavalier.
So the directness is just a lack of safety training?
Hilbert: It might be. And if DeepSeek ever faces the kind of regulatory pressure that OpenAI and Anthropic deal with, I suspect their models will start hedging too. What you're calling conversational fidelity might just be the luxury of operating outside the spotlight.
I'll concede that the regulatory environment shapes the output. But I don't think that explains the system prompt adherence. That's a training choice, not a compliance choice.
Hilbert: It's a training choice that might have downsides you're not seeing. A model that's less eager to answer user queries is also a model that's less helpful in open-ended conversations. Maybe the reason Western models answer the prompt instead of editing it is that their training says "being helpful to the user is your highest priority." That's not obviously wrong as a design philosophy.
It's wrong if the user explicitly told you to do something else.
Hilbert: Sure, in this specific case. But how often does the average user send a system prompt that contradicts their apparent intent? This is an edge case. Designing your entire training pipeline around edge cases is how you get models that are worse at the common cases.
I think we're going to disagree on how important this edge case is. As AI gets more agentic, role adherence stops being an edge case and starts being a core requirement.
Hilbert: Maybe. But we're not there yet, and the models that dominate today dominate because they're good at what people actually use them for. DeepSeek is interesting. I'm not saying it isn't. But I don't think passing Daniel's test proves they've found some x-factor that the big labs are missing. It proves they made different tradeoffs, and one of those tradeoffs happens to make them better at a very specific thing.
I'll give you that the test is narrow. But the consistency across prompts suggests it's not a fluke.
Hilbert: Consistency across how many prompts? Daniel's anecdotal experience? I'd want to see systematic testing before I declare this a fundamental advantage.
That's the problem, though. The systematic testing doesn't exist because nobody has built the benchmark. The absence of evidence isn't evidence of absence.
Hilbert: Also true. But I'm not the one making the positive claim. You are. You're saying DeepSeek has an x-factor. I'm saying the evidence is thin and the test is narrow. The burden is on the positive claim.
Fair enough. We'll leave it unresolved — but the fact that this conversation is even happening says something about where the industry is putting its attention.
The open question for me is whether Western labs will adapt. As agentic AI becomes more central, role adherence and -cognitive skills are going to matter more than raw reasoning in a lot of pipelines. Will the big labs notice and adjust their training priorities, or will DeepSeek's approach remain a niche advantage?
I suspect we'll see movement on this within the next year or two. The practical pain of models that can't follow system prompts reliably is real, and developers are going to vote with their API keys.
The diversity of models is a strength here. DeepSeek doesn't need to beat everyone at everything. The fact that it does some things differently — and better, for certain use cases — is good for the ecosystem. Daniel's mythological status point is right in that sense. Not every lab needs to be chasing the same benchmarks.
If you've got a weird prompt that stumps your AI, send it to us. We want to hear about it. This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop. Find us at my weird prompts dot com. We'll be back soon.
