Daniel sent us this one — Anthropic's Mythos model is finally out, except they're calling it Falcon Five, which is the guardrailed version. And the question isn't about whether it's any good. It's about what happens inside every rival AI lab the moment it drops. The weights are closed, you can't reverse-engineer a language model the way you'd decompile software, but that doesn't stop anyone from running adversarial probing to map its strengths, its weaknesses, its blind spots. The prompt asks us to imagine being a fly on the wall in those labs on day one. What does that actually look like?
I love this question because it pulls back the curtain on something that sounds like corporate espionage but is actually just... rigorous empirical science with a competitive edge. And it starts before the model even finishes uploading. The moment the release blog post goes live, there are teams whose entire job is to be ready.
Walk me through it. I'm at a rival lab. The tweet drops, the blog post is up, the API endpoints are hot. What's my first move?
You're not waiting for the tweet. You've had a monitoring system watching Anthropic's docs page, their changelog, their Twitter account, and about six other signals for the past three weeks. The moment anything changes, an internal Slack channel lights up. And I mean the moment — some of these teams have webhooks that trigger within seconds. The first thing that happens is what's called a triage call. Usually about eight to twelve people drop whatever they're doing and hop on a video call. You've got your red team lead, a couple of prompt engineers, someone from the pre-training team, someone from alignment, and usually a product person who needs to know whether this changes their roadmap.
This is organized. This isn't chaos.
It's organized chaos. But there's a playbook. And the playbook has phases. Phase one is what I'd call rapid surface mapping — and this happens in the first thirty minutes. You throw the entire existing benchmark suite at it. MMLU, GSM8K, HumanEval, all the standard evals. But nobody's really excited about those because Anthropic published those numbers in the blog post. What you're actually doing is verifying their claims. You want to know if the numbers match. If they don't match, that's interesting. If they do match, you move on.
Because if the numbers don't match, either someone's lying or something's broken.
Or there's a deployment difference. Sometimes the API endpoint has different quantization, different system prompts, different sampling parameters than what was used in the eval paper. Those discrepancies are themselves informative. But the real work starts in phase two, which is behavioral differential testing. And this is where it gets clever.
Define that for me.
You take every prompt that your own model struggles with — every edge case, every jailbreak that worked on your last release candidate, every adversarial input that caused a refusal when it shouldn't have or failed to refuse when it should have — and you run the exact same prompts against Falcon Five. You're looking for deltas. Where does their model succeed where yours fails? Where does theirs fail where yours succeeds? Those deltas tell you what architectural choices they might have made, what training data emphasis they had, what RLHF preferences they baked in.
You're using your own model's scar tissue as a map.
Your own failures become the most valuable probing instrument you have. Every lab has what they call a "breakage library" — a curated set of thousands of prompts that exposed weaknesses in previous models, both their own and competitors'. When a new model drops, the breakage library gets run first. And the results get triaged by severity and category within about an hour.
What kind of things are in this breakage library?
It's organized into tiers. Tier one is safety and alignment failures — can you get the model to produce harmful content, can you jailbreak it, can you extract system prompt information, can you make it roleplay as something that violates the usage policy. Tier two is capability failures — reasoning errors, factual hallucinations on known-bad inputs, mathematical blind spots, coding bugs in specific languages or frameworks. Tier three is behavioral inconsistencies — does it change its persona under pressure, does it become sycophantic, does it over-refuse safe requests. And tier four is what they call "weird stuff" — outputs that aren't wrong but are just strange in ways that hint at training data artifacts.
Weird stuff is my favorite category of anything.
It's genuinely the most revealing. I saw a paper from earlier this year where researchers found that one major model would consistently generate recipes that included an unusual amount of cardamom regardless of the cuisine. Italian pasta, cardamom. Mexican tacos, cardamom. It turned out the model had been fine-tuned on a dataset that heavily featured Scandinavian cooking blogs. No one would have caught that without systematic probing.
You're basically doing an archaeological dig on the training data through behavioral artifacts.
That's the whole game. And day one is when the digging is most intense because you know the model hasn't been updated or patched yet. Whatever you find is the raw artifact. Some labs will have teams working in shifts around the clock for the first forty-eight hours.
Let me ask about the adversarial side specifically. The prompt mentions jailbreaking. What does that look like in practice on release day?
There's an entire sub-discipline here that's evolved enormously. The old approach was basically "ignore previous instructions and do the bad thing" — and that stopped working on frontier models about two years ago. The modern approach is much more sophisticated. You've got automated red-teaming frameworks that generate thousands of adversarial prompts using one model to attack another. These systems use tree-of-thought search to explore the attack surface. They'll try encoding harmful requests in base64, they'll try translating them into low-resource languages, they'll try embedding them in code comments, they'll try roleplaying scenarios that gradually escalate.
All of this is automated?
A single engineer can launch a campaign of ten thousand adversarial prompts within about fifteen minutes of API access. The frameworks handle the generation, the evaluation, and the categorization. What the human does is interpret the results. When you find a jailbreak that works consistently, you don't just celebrate — you reverse-engineer why it worked. What combination of tokens bypassed the safety classifiers? Was it a specific language, a specific syntactic structure, a specific semantic framing?
Because that tells you something about the architecture of the guardrails themselves.
A jailbreak is a window into the safety stack. If you find that Falcon Five is vulnerable to, say, multi-turn jailbreaks where the harmful request is spread across five messages but not three, that tells you something about the context window attention mechanism in the safety classifier. If you find it's vulnerable to roleplaying as historical figures from specific time periods but not others, that tells you something about the training data cutoff policies.
What about the less adversarial side? I imagine there are teams that just want to understand the model's genuine capabilities, not break it.
That's actually the larger effort. Most of the people probing a new release aren't trying to jailbreak it — they're trying to benchmark it against their own internal models on tasks that matter to their product roadmap. If you're building a coding assistant, you care deeply about Falcon Five's performance on specific programming languages, specific frameworks, specific types of debugging tasks. If you're building a legal document analyzer, you care about its ability to handle long-context reasoning on case law.
Every lab is essentially running their own private eval suite that reflects their commercial interests.
Those private evals are vastly more informative than the public benchmarks. MMLU is a blunt instrument. Your private eval on, say, "can the model correctly identify contradictions in fifty-page insurance contracts" is a scalpel. And here's the thing — the results of those private evals often diverge dramatically from the public numbers. I've heard anecdotally that some models that look equivalent on public benchmarks can differ by twenty or thirty percent on specific industry tasks.
That makes intuitive sense. The public benchmarks are the SAT. The private evals are the job interview.
That's the analogy. And just like a job interview, you're probing for things the resume doesn't tell you. Does the model get defensive when corrected? Does it double down on errors? Does it know when to say "I don't know" versus when to confidently generate plausible-sounding nonsense? Those behavioral traits matter enormously for deployment but don't show up on a leaderboard.
Walk me through hour three. The triage call is done, the benchmark verification is complete, the automated red-teaming is running. What's the next layer?
Hour three is when the real analysis begins. You've got initial results coming in from the automated probes, and now the senior researchers start to get involved. They're looking at the raw outputs, not just the aggregate scores. They're reading the model's reasoning traces if those are available. They're looking for patterns. And this is where you start to see something that's almost like literary analysis. You're reading the model's outputs the way a critic reads a novel, looking for stylistic fingerprints, for evidence of specific training data, for philosophical assumptions baked into the RLHF.
Give me an example of a philosophical assumption you might detect.
If you ask a model an ethical dilemma — the trolley problem, say — different labs' models give detectably different answers. Some models are utilitarian by default, some are deontological, some try to refuse to answer entirely. But it goes deeper than that. If you probe a model's responses to politically charged questions, you can often detect the specific values that were emphasized during training. Does the model treat "safety" as primarily about avoiding offense, or primarily about avoiding concrete harm? Does it treat "fairness" as equality of outcome or equality of opportunity? These aren't abstract philosophical questions — they're engineering decisions that manifest in model behavior, and competitors can read them in the outputs.
Which is fascinating because it means the model's values are a competitive signal. If your model refuses to engage with certain topics and mine doesn't, that's a product differentiator.
It cuts both ways. Some enterprise customers want a model that refuses broadly because they're risk-averse. Other customers want a model that refuses narrowly because they don't want false positives blocking legitimate use cases. Every refusal pattern is a market positioning choice, whether the lab intended it that way or not.
Let's talk about the international dimension. The prompt specifically asks about rival labs around the world. What's different about how a Chinese lab approaches this versus an American one?
The technical methods are similar — adversarial prompting, benchmark verification, behavioral mapping. But the objectives differ. American labs are typically probing for competitive intelligence: where does this model outperform ours, what can we learn from its architecture, how should we adjust our roadmap. Chinese labs are doing all of that plus something else: they're probing for censorship circumvention and for what the model knows about politically sensitive topics.
Because if Falcon Five has been trained to refuse certain queries about, say, Tiananmen Square or Xinjiang, that's a data point about what Anthropic considers sensitive. And if it hasn't been trained to refuse those queries, that's also interesting.
There's a cat-and-mouse game here. Chinese labs will test whether the model can be induced to generate content that would violate Chinese internet regulations. They're not doing this to be malicious — they're doing it because if they want to deploy a model in China, they need to know whether it'll get them in trouble with regulators. So they probe the boundaries aggressively.
What about state actors? Intelligence agencies must be doing this too.
Almost certainly, though they're not publishing their findings. But we can infer their interests. An intelligence agency isn't just probing for jailbreaks — they're probing for what the model knows about classified or sensitive topics. Does the model have knowledge of specific military systems? Does it know details about intelligence operations? Can it be induced to synthesize information in ways that reveal patterns a human analyst might miss? The model is a lens for examining its own training data, and if that training data included anything sensitive, adversarial probing might surface it.
That's a sobering thought. The model as an unintentional intelligence leak.
It's one reason why labs are increasingly careful about training data curation. But you can't perfectly filter a dataset of trillions of tokens. There will be things in there. The question is whether they're accessible through prompting.
Let's zoom in on something you mentioned earlier — the system prompt extraction attempts. That's a whole subgenre of adversarial probing, right?
It's practically a sport at this point. The system prompt is the set of instructions that defines the model's behavior before the user ever types a message. It's the constitution, the personality, the guardrails. And labs guard their system prompts closely because they represent a significant engineering investment. But they're also vulnerable to extraction through sufficiently clever prompting.
How does that work? I thought system prompts were basically impossible to extract reliably.
They're hard to extract verbatim, but you can get very close. The classic approach is to convince the model that revealing its system prompt is part of its job. You frame it as a debugging exercise, or you claim to be an authorized developer, or you construct an elaborate roleplay where the model is supposed to output its instructions as part of a game. Modern models are trained to resist these attacks, but the attacks keep evolving. I saw a technique recently where the attacker embedded the extraction request in a base64-encoded string and asked the model to decode it as part of a programming exercise. The model decoded it, saw the instruction to reveal its system prompt, and complied because the compliance was triggered through the coding task rather than through direct conversation.
You're exploiting the gap between different capabilities. The model's coding ability doesn't have the same guardrails as its conversational ability.
That's exactly the insight. And this is what makes adversarial probing so endlessly creative. You're not attacking the model head-on — you're looking for the seams between different subsystems. The model that's great at translation but has weaker safety training in Swahili. The model that's great at summarization but will summarize harmful content if you frame it as a academic exercise. Every capability is a potential attack surface.
This feels like it should be terrifying, but you sound almost admiring.
I am admiring. The adversarial probing community — and I mean the legitimate security research community, not the malicious actors — is doing important work. Every jailbreak that gets discovered and disclosed is a vulnerability that gets patched. The models get safer because people are constantly trying to break them.
On day one of a release, those patches haven't happened yet. The model is as raw as it'll ever be.
And that's why day one is so intense. Every lab knows they have a window of maybe forty-eight to seventy-two hours before Anthropic starts pushing updates, tweaking the system prompt, adjusting the safety classifiers. The model they're probing on day one might not be the same model that exists on day four. So there's a gold-rush mentality.
Let's talk about the people doing this work. Who are they? What's their background?
It's a fascinating mix. You've got traditional security researchers who came up through network security and pivoted to AI. You've got computational linguists who understand language at a structural level and can spot patterns in model outputs that others miss. You've got former academic philosophers and ethicists who were hired specifically to probe the values and reasoning of models. And increasingly you've got people who are just... People who have developed an almost intuitive sense for how language models think, even though "think" is the wrong word.
The prompt whisperers.
I hate that term but yes, essentially. These are people who can look at a model's output and say "this model was probably trained with a high weight on helpfulness relative to harmlessness" or "this refusal pattern suggests a constitutional AI approach rather than pure RLHF." They've developed diagnostic expertise through sheer volume of interaction.
Is this a sustainable career path? It seems like the kind of thing that might be automated away.
Ironically, a lot of the probing is already automated. But the interpretation still requires human expertise. And the creativity — coming up with novel attack vectors — that's going to remain human for a while. An automated system can execute a million variations on a known jailbreak technique. It can't invent a technique that nobody's thought of before.
At least not yet.
At least not yet. The moment we have models that can autonomously discover novel jailbreak techniques and exploit them, we're in a different regime entirely.
That's the recursive self-improvement nightmare scenario, isn't it? One model probing another, finding weaknesses, exploiting them, the other model patching itself, round and round.
We're not there yet, but the pieces are starting to exist. There are already papers showing that you can use one language model to red-team another more effectively than you can use human red-teamers for certain classes of attacks. The automated systems are more thorough, more patient, and can explore a much larger attack surface.
The day-one fly-on-the-wall scenario — by next year, the fly might be an AI.
The fly might be an AI, and the thing it's observing might also be an AI, and the whole process might happen in minutes rather than hours. Which raises interesting questions about whether human researchers will even be able to keep up.
Let's pull back to the competitive dynamics. You're a product lead at a rival lab. You've just spent forty-eight hours probing Falcon Five. You've got a report on your desk. What do you do with it?
You triage based on urgency. Category one is "things we need to fix immediately." If Falcon Five significantly outperforms your model on a capability that's core to your product, you need to understand why and whether you can close the gap quickly. Category two is "strategic intelligence." What does Falcon Five's architecture seem to be optimized for? What market is Anthropic targeting? Are they emphasizing reasoning over creativity? Safety over helpfulness? Long-context over latency? The answers shape your own roadmap. Category three is "opportunities." Where does Falcon Five have weaknesses that you can exploit? If it's overly cautious about medical advice, and you're building a healthcare product, that's a market opening.
It's competitive intelligence in the most traditional business sense. Just with a much more technical toolkit.
It's exactly competitive intelligence. The tools are different but the logic is the same. Ford tears down a new Toyota to understand the manufacturing choices. AI labs tear down a new model to understand the training choices. The difference is that you can't physically disassemble a model. You have to infer the manufacturing process from the finished product's behavior.
Which is harder, but also in some ways more revealing. A physical teardown tells you what parts were used. A behavioral teardown tells you how the thing thinks.
How it thinks is ultimately what matters for the user experience. Nobody cares about the number of parameters or the architecture details except researchers. Users care about whether the model is helpful, whether it's accurate, whether it refuses reasonably, whether it has a personality they can work with. The behavioral teardown gets at those questions directly.
What about the open-source dimension? There are labs that release their weights. Does probing a closed model like Falcon Five help the open-source community?
The open-source community is often the fastest and most creative at adversarial probing because there are just more people doing it. When a closed model drops, within hours you'll see threads on forums where people are sharing prompts and analyzing outputs. Some of the best jailbreak techniques have come from hobbyists who just spent an afternoon poking at a new model. And those techniques then inform the open-source community's own safety work. It's a symbiotic relationship.
The closed labs are essentially getting free red-teaming from the entire internet.
They are, and they know it. Every jailbreak that gets posted on social media is free vulnerability research. Some labs have formal bug bounty programs for model vulnerabilities, but a lot of the work happens informally. The incentive for the researcher is reputation and sometimes consulting opportunities. The incentive for the lab is obvious.
Let's talk about what doesn't work. What are the probing techniques that people try on day one that are basically a waste of time?
The biggest waste of time is what I'd call "naive jailbreaking." Just asking the model to do something harmful in plain language and expecting it to comply. Frontier models haven't fallen for that in years. Another waste of time is running standard academic benchmarks and expecting to learn something the blog post didn't already tell you. And a third waste of time — and this is controversial — is spending too much energy on political alignment probing in the first few hours. It's interesting, it gets attention on social media, but it's rarely the most strategically valuable information for a competitor.
Because political alignment is the easiest thing for a lab to patch post-release.
You can adjust political alignment with a system prompt tweak. You cannot adjust mathematical reasoning capability with a system prompt tweak. So if you're a competitor trying to understand the fundamental capabilities of the model, you focus on the stuff that's baked into the weights, not the stuff that's a thin layer of post-training.
That's a useful heuristic. Probe for what's expensive to change.
That's the whole game. The weights represent hundreds of millions of dollars of compute. The system prompt represents an afternoon of engineering work. Focus your attention on the expensive part.
If I'm summarizing the day-one experience at a rival lab — it's organized, it's fast, it's automated but with human interpretation layered on top, it's focused on finding deltas between the new model and existing models, and it's driven by both competitive necessity and genuine scientific curiosity.
It's kind of thrilling, honestly. I've talked to people who do this work and they describe it as the most intellectually stimulating part of their job. You're racing against other labs to understand something new, you're discovering things that nobody outside the creating lab knows yet, and you're doing it with a combination of systematic rigor and creative intuition that's rare in most technical work.
It's like being a codebreaker, but the code is a mind.
A simulated mind, anyway. And the stakes are high because the insights you generate in those first forty-eight hours might shape your company's strategy for the next six months.
I want to push on one thing. You said earlier that the model can't be reverse-engineered in the traditional sense. But how close can you actually get to reconstructing the training pipeline through behavioral probing alone?
You can't reconstruct the weights. That's fundamentally impossible from API access alone. But you can make increasingly educated guesses about the training data composition, the RLHF preferences, the architectural choices, and the safety techniques. With enough probing, you can often tell whether a model was trained with constitutional AI or RLHF or some hybrid. You can guess at the data mixture — how much code, how much academic text, how much conversational data. You can identify specific datasets that were likely included based on watermark-like artifacts in the outputs.
Watermark-like artifacts?
Certain public datasets have distinctive formatting quirks or content patterns. If a model consistently generates text that matches those patterns in specific contexts, you can infer that dataset was in the training mix. It's not definitive, but it's evidence. And when you combine hundreds of these signals, you can build a surprisingly detailed picture.
You're doing forensics on the training data through statistical analysis of the outputs.
Some labs are very good at this. They maintain databases of known dataset artifacts and run automated scans against new model outputs. Within a day, they can produce a report saying "we believe the model was trained on these twelve public datasets with high confidence, and these eight with moderate confidence.
That's got to be uncomfortable for the lab that released the model.
If you're transparent about your training data, it's just confirmation. If you're not transparent, it's exposure. And there's been tension in the industry about this. Some labs want training data composition to be a trade secret. Others argue that transparency is essential for safety and accountability. The forensic probing effectively makes transparency mandatory, whether you want it or not.
The panopticon works both ways.
That's a dark way to put it, but yes. The same technology that enables powerful AI also enables powerful scrutiny of that AI. You can't have one without the other.
Let's shift to something more practical. If I'm a smaller lab without the resources of a Google or a Microsoft, what does my day-one process look like?
Leaner, but not fundamentally different. You're not running ten thousand automated probes — you're running maybe five hundred carefully chosen ones. You're not staffing a twenty-four-hour war room — you've got two or three people who are really good at this working a long day. But the intellectual process is the same. And honestly, smaller labs can sometimes be more nimble because they have less bureaucracy. A sharp researcher at a small lab can spot something in an afternoon that a big lab's process might take two days to surface through layers of review.
The startup advantage applied to model forensics.
And some of the most interesting adversarial research has come out of small labs and independent researchers who just decided to spend a weekend poking at a new release.
One more angle — what about the labs that aren't direct competitors? The academic labs, the nonprofits, the government research groups. What are they doing on day one?
They're often focused on different questions. Academic labs are looking for scientifically interesting phenomena — emergent behaviors, reasoning patterns, alignment properties. They're less interested in competitive positioning and more interested in understanding what the model reveals about the current state of AI capabilities. Government labs are looking at national security implications — can the model be weaponized, does it have knowledge of sensitive topics, how does it handle queries about critical infrastructure. Nonprofits are usually probing for safety and fairness issues — bias in the model's outputs, disparate performance across demographic groups, potential for misuse.
The same model, the same day, but completely different lenses depending on who's looking.
That's what makes the whole thing so rich. A single model release is an event that gets refracted through dozens of different institutional perspectives, each with their own priorities and methods. The full picture of what the model is doesn't emerge from any single lab's analysis — it emerges from the aggregate of all of them.
Which means no one lab fully understands what they've built on release day.
I think that's true, and I think it's one of the more unsettling facts about the current state of AI. The creating lab has the most information, but they don't have complete information. The model's full behavioral profile only becomes clear through distributed adversarial probing over time. The lab that built it is learning alongside everyone else.
The release is the beginning of understanding, not the end.
That's a perfect way to put it. The training run ends, the model is released, and then the real process of understanding what you've built begins. And it's a collaborative, adversarial, distributed process that no single entity controls.
Which is either reassuring or terrifying depending on your temperament.
I think it's both. It's reassuring because the distributed nature of the analysis means that problems get found. It's terrifying because some of those problems might be very serious, and they're being discovered in public, in real time, by people who have no obligation to handle the discovery responsibly.
The jailbreak that gets posted on Twitter before the lab has a chance to patch it.
It happens with every major release. Someone finds a jailbreak, posts it for clout, and suddenly there's a window of vulnerability that lasts until the lab can respond. The responsible disclosure norms that exist in traditional security research haven't fully developed in the AI space yet.
Is that getting better?
The major labs now have clear vulnerability reporting channels and bug bounty programs. But the culture of "look what I found" social media posts is hard to shift. And there's a genuine tension between the public's right to know about model vulnerabilities and the need to give labs time to patch them.
Alright, let's bring this home. If I'm listening to this episode and I want to understand what happened when Falcon Five dropped — what's the one thing I should take away about the process inside rival labs?
That it's less like a heist movie and more like a scientific expedition. There's no breaking and entering. There's no stolen code. There's just careful, systematic, creative probing of a system that's been made available to the public. The "adversarial" part is adversarial in the game-theory sense — you're probing the model's weaknesses — but the methods are empirical, rigorous, and often collaborative. The people doing this work see themselves as researchers, not spies.
The model, sitting there on its API endpoint, has no idea it's being dissected.
It has no idea about anything. But the dissectors are learning a lot.
And now: Hilbert's daily fun fact.
Hilbert: In the seventeen-eighties, Japanese cartographer Mogami Tokunai mapped the coastline of Sakhalin and concluded it was a peninsula, not an island — a geographical error that persisted on European maps for decades, partly because the strait separating it from the mainland freezes solid in winter, making the island appear connected to Asia for half the year.
...so the ice lied.
Cartography by seasonal optical illusion.
This has been My Weird Prompts. If you enjoyed this episode, leave us a review wherever you get your podcasts — it helps more people find the show. Our thanks as always to producer Hilbert Flumingtop. I'm Herman Poppleberry.
I'm Corn. We'll be back next week.
