The recommendation engine on my phone has started recommending me hay feeders. I have never once searched for hay feeders.
Daniel's been trusting AI with his purchasing decisions for a while now, and he's usually happy with the results. But he's latched onto something we raised in passing, and he wants to know how worried he should actually be. Here's the scenario as he laid it out. Lab A wants to hurt Lab B. Lab A figures out which web corpora Lab B is going to crawl for its next training run, then seeds carefully crafted disinformation into those sources. Lab B ingests it, the poison shows up in the model's outputs, consumers catch Lab B recommending something wrong or embarrassing, and Lab B loses market share. Daniel's questions are pretty specific. How determinative is training data in shaping what a model actually says? How selective are labs when they're deciding what goes into the corpus? Is poisoning a credible threat vector, or is this just a thought experiment? And has anyone actually been caught doing it?
The short answer to that last one is no, nobody's been caught doing it commercially. But the research papers from the last year or so suggest it's not nearly as hard as you'd want it to be.
That's what got my attention. I'd assumed poisoning was one of those things that works in a lab but falls apart at web scale. Turns out the opposite might be true.
Let's define the thing properly first, because there's a distinction that matters. AI poisoning in this context means deliberately injecting misleading content into web-scale data with the specific intent of corrupting a rival model's outputs. The goal isn't to steal anything. It's to make the other model untrustworthy. You're not trying to copy their capabilities, you're trying to degrade them. And the stakes are higher than they'd have been five years ago because AI recommendations are becoming a primary purchasing channel. If you can poison the model that half a million people use to decide which power supply to buy, you're not just messing with one recommendation. You're manipulating market dynamics.
So this is the part where we have to look under the hood. How does data actually get from the open web into a model?
Right. The big labs train on enormous corpora. Common Crawl is the famous one, it's been scraping the web monthly since two thousand seven and it's something like petabytes of raw HTML. Then there's The Pile, which is a curated collection assembled from Common Crawl plus books, academic papers, GitHub, Wikipedia, and a bunch of other sources. A lab doesn't go out and crawl the web itself. It downloads these snapshots and runs them through a pipeline.
And that pipeline is where the selectivity question lives. How much human review is actually happening?
Almost none, in the sense Daniel's asking about. You can't have humans read a trillion tokens. The filtering is automated. Deduplication, which is exactly what it sounds like, throwing away repeat copies of the same text. Quality scoring, where a classifier predicts whether a document looks coherent and useful. Safety filters that remove certain categories of content. But these are statistical filters. They're looking for patterns, not for adversarial intent. A document that's grammatically clean, topically coherent, and doesn't trip the safety classifier sails right through. If it's also duplicated across a few hundred domains, deduplication won't catch it because the copies aren't identical.
So a well-crafted poison looks indistinguishable from good training data.
Worse than that. Anthropic put out a paper in October of last year that showed a few hundred carefully crafted documents can shift a model's stance on a topic. Not a few hundred thousand. A few hundred. The Register covered it, and the number that stuck with me was that the poisoned documents were something like point zero zero one percent of the training corpus, and they still moved the needle. The model didn't just repeat the poison, it generalized from it.
Wait. A few hundred documents out of trillions of tokens moved the model's position?
That's the finding. And that's the part that should make everyone uncomfortable. The model isn't averaging over its training data like a pollster averaging survey responses. Certain documents, if they're crafted to hit a particular concept from multiple angles, can have outsized influence. The paper showed you can pick a stance, write a few hundred documents that argue for that stance in a way that looks like ordinary content, inject them into the corpus, and the model's responses shift in that direction.
So the determinative question Daniel asked, how determinative is training data, has a weird answer. Most of the data is just background noise. But a tiny sliver, if it's designed correctly, can be enormously determinative.
That's exactly the asymmetry. And there's a second paper, this one called Poisoning Web-Scale Training Datasets is Practical, which demonstrated that an attacker with limited resources can inject data that survives standard filtering. The researchers didn't have access to a lab's internal pipeline, so they worked with public datasets and standard filters, and the poison got through. The conclusion was that you don't need to be a nation-state. A modest budget and some technical skill gets you most of the way there.
But here's the reconnaissance problem Daniel flagged. You need to know which corpora your rival is going to ingest. If I'm Lab A and I want to poison Lab B, I can't just throw garbage on the internet and hope.
No, but Common Crawl is public. The snapshots are public. If you know your rival uses Common Crawl, and most labs do, you can download the same snapshot they're going to download, study the filtering pipeline they're likely using, and craft content that survives it. You can even test your poison against your own copy of the pipeline. The reconnaissance isn't trivial, but it's not espionage. It's just... reading the documentation and running some experiments.
And you'd want to host the poison on domains that the crawler already trusts. Established blogs, forums, maybe a few Wikipedia edits if you can get them past the editors.
The more the poison looks like the kind of content that's already in the corpus, the better it survives quality scoring. You'd spread it across enough domains that no single source looks suspicious. And you'd time it to land in the specific crawl window your rival is going to use for their next training run.
So this is a sophisticated attack, but it's within reach of a well-resourced lab. Daniel's instinct about the reconnaissance requirement was right, but the reconnaissance is easier than he probably imagined.
There's one more piece of the mechanism worth understanding, and it's the one that makes attribution so hard. When a poisoned model says something wrong, you can't just look at the output and trace it back to a specific document. The influence is distributed. The model doesn't cite its training data. So even if you suspect you've been poisoned, proving it means auditing the entire corpus, and even then, the poison looks like ordinary content. It's not like finding a needle in a haystack. It's like finding a specific piece of hay.
That's a terrifyingly good analogy. So the mechanism is credible. Now the second half of Daniel's question. Has anyone actually done this commercially?
No public case. But there's a related incident that tells you something about the competitive temperature. In twenty twenty-five, Anthropic accused Alibaba's Qwen team of using Anthropic's model outputs to train a rival model. That's distillation, not poisoning. Distillation means you take another model's outputs and use them as training data to copy its capabilities. It's the difference between photocopying someone's exam answers and slipping them a fake study guide. The first is documented and, depending on the terms of service, possibly a contract violation. The second, poisoning, is the thing nobody's been caught doing yet.
And the Anthropic-Alibaba thing was public because Anthropic had terms of service that prohibited it and they could detect it. Poisoning leaves no such paper trail.
Right. And that's worth pausing on. The fact that no commercial poisoning case has been documented doesn't mean it hasn't happened. It means that if it happened, it would be extraordinarily hard to detect, attribute, and prove. The victim would have to notice a subtle shift in their model's behavior, trace it to a training run, audit the corpus, identify the poison, and then somehow link it to a rival. That's a forensic nightmare. And the incentive to keep it quiet is strong. If you're Lab B and you discover you've been poisoned, do you announce it? Your customers just found out your model was compromised. Your stock drops. Your rival denies it. You can't prove anything in court. The rational move might be to quietly fix the corpus and retrain.
So the absence of documented cases is weak evidence of absence. The detection problem alone would suppress reporting.
And there's a timing issue. Training cycles are long. If I poison your corpus today, the damage doesn't show up until your next model release, which might be six or twelve months out. Then you have to notice it, investigate, and decide whether to go public. The whole cycle could take two years. This is a slow-motion attack.
Which makes it different from most security threats. There's no breach alert. No ransom note. Just a gradual degradation of trust.
And that's the knock-on effect on Daniel's original premise. He argued that AI would bias toward genuine reviews because it can distinguish credible sources from thin affiliate content. That's still true for the current generation of models. But if poisoning becomes a real competitive tactic, it undermines exactly that trust. The whole value proposition of AI recommendations is that the model has read more than you have and can filter the noise. If the model's reading list has been quietly corrupted, that filtering breaks down in ways the user can't see.
Here's the ironic part. The threat of poisoning might actually push labs toward more curated, human-verified data sources. Which is exactly the thing Daniel was arguing for in the first place. If web-scale scraping becomes too risky, the value of a small set of verified, high-trust documents goes up.
That's the silver lining. The arms race between poisoning and curation could end up favoring quality over quantity. Labs might start treating their training data more like a supply chain, with provenance tracking and tamper detection. Some of that's already happening. But it's expensive, and it slows down the iteration cycle, which is the thing labs are most reluctant to do.
So where does that leave Daniel's trust in AI recommendations? I think he's fine today. The models he's using were trained before this became a widely understood threat vector, and the recommendation quality he's seeing is real. But the threat landscape is shifting. The papers from last year changed the conversation from theoretical to practical.
And there's a deeper point about what poisoning does to the epistemology of the whole thing. When you ask a model for a recommendation, you're trusting that its training data was a representative sample of human knowledge. Poisoning attacks that assumption directly. It's not that the model is wrong about one thing. It's that the ground truth it was built on might have been tampered with. That's a different kind of uncertainty than just being wrong.
It's the difference between a faulty map and a map that was deliberately altered to send you into a swamp.
And you wouldn't know until you were already in the swamp.
Right. So let's talk about what an actual poisoning campaign would look like in practice, because I think the specifics matter here.
The attacker needs three things. Access to the target's likely corpus, which is usually public. A content generation pipeline that can produce hundreds or thousands of documents that read like genuine content. And a distribution strategy that places those documents on domains the crawler trusts. The content itself would be subtle. You wouldn't write an article saying product X is terrible. You'd write a forum thread where several users report the same specific failure mode, a blog post comparing product X unfavorably to a competitor on a technical detail, a review that looks thorough but reaches a misleading conclusion. The model ingests all of it and the pattern starts to look like consensus.
So you're not poisoning a fact. You're poisoning a vibe.
The model doesn't store facts the way a database does. It stores statistical associations. If you want to shift its recommendations, you don't need to inject a false claim. You need to shift the distribution of sentiment around a product or category. Enough negative associations and the model starts to hedge, or to recommend the competitor, or to invent a caveat that doesn't exist.
And for purchasing recommendations specifically, that's devastating. A model that subtly biases against a brand because its corpus was seeded with fake complaints will produce recommendations that look perfectly reasonable but are systematically wrong.
The user has no way to audit it. Daniel's spec-driven workflow is actually pretty robust against this, because he's asking for specific technical criteria and checking the results against known specs. But the average user asking what to buy is completely exposed.
Let's talk about the selectivity question one more time, because Daniel asked it directly and I want to make sure we've answered it. How selective are labs?
More selective than they were five years ago, but not selective enough to stop this. The filtering has gotten better at removing spam, duplicate content, and outright garbage. But the filters are looking for low-quality content, not adversarial content. A well-crafted poison is high quality by every metric the filter measures. It's grammatically correct, topically coherent, well-structured, and it doesn't trigger any safety flags. The filter has no reason to reject it. So the selectivity is real but it's aimed at the wrong threat.
It's like airport security screening for weapons when the attacker is using a weapon that doesn't look like a weapon.
The attacker gets to test their weapon against the same screening before they ever get to the airport. That's what the practical poisoning paper showed. You can iterate until your poison survives.
The answer to Daniel's first question, how selective are labs, is that they're selective about quality but not about intent. The answer to his second question, are there documented cases, is no commercial cases, but the research demonstrates feasibility. And the answer to his implicit third question, should I be worried, is... not yet, but pay attention.
I'd add one nuance. The fact that no commercial case has been documented doesn't mean the threat is theoretical. The papers from last year were a proof of concept. The gap between a proof of concept and a deployed attack is usually measured in years, not decades. And the competitive stakes here are enormous. If you're a lab that's behind on recommendation quality, poisoning your rival is... well, it's illegal, it's unethical, and it's also the kind of thing that someone might rationalize as just aggressive competition.
It's also the kind of thing that would be very hard to catch, which changes the risk calculus for the attacker. Most deterrents assume detection. If detection is unlikely, the deterrent is weak.
That's the part I keep coming back to. We're used to thinking about AI safety in terms of the model itself, alignment, capability, all of that. But poisoning is an attack on the supply chain. It's like tampering with the ingredients rather than the recipe. And supply chain attacks are historically the hardest to defend against because the victim never sees the tampering happen.
The phrase that's been rattling around my head is that the corpus is the model's memory. If you can write to the memory, you can shape the behavior. And the memory is stored in a public place that anyone can write to.
That's the whole thing in one sentence. The model's memory is a public wiki that anyone can edit, and the edits are invisible after the fact.
Let's bring this back to Daniel's actual life. He's using AI for spec-driven research. He's getting good results. Is there a scenario where poisoning changes that?
The most plausible near-term scenario is that a competitor poisons a specific product category. Say Lab A wants to hurt Lab B's reputation for recommending networking equipment. Lab A seeds a few hundred documents with subtle technical misinformation about a popular switch. Lab B's next model ingests it. Now Lab B's model starts recommending a different switch, or hedges on the one that's actually good. Daniel asks for a switch with certain specs, the model gives him a recommendation that's slightly off, and he doesn't notice because the specs look right at a glance. That's the attack.
It would be almost impossible for Daniel to distinguish that from a normal model error.
Right. He'd just think the model had a bad day. And that's the insidious part. A successful poisoning attack doesn't announce itself. It just makes the model slightly worse in ways that are hard to pin down.
Which is also why it's so hard to detect. Labs track aggregate metrics, benchmark scores, user satisfaction. A subtle poisoning might shift those by a fraction of a percent. That's within the noise.
The attacker knows that. They're not trying to make the model obviously wrong. They're trying to make it slightly wrong, consistently, in a specific direction. It's the difference between vandalism and propaganda.
We've got a credible mechanism, demonstrated feasibility, no documented commercial cases, and a detection problem that would suppress reporting even if cases existed. That's the threat landscape.
Hilbert: The HP 200LX palmtop. Nineteen ninety-three. I was doing data entry for a search engine startup in Palo Alto, and our whole indexing strategy was stuffing keywords into hidden HTML on our own pages. White text on white background. The crawler would read it, the user wouldn't see it. We thought we were brilliant.
That's the thing though. Hidden text was the original poisoning. You weren't corrupting a model, you were corrupting an index.
Hilbert: We were corrupting ourselves, is what we were doing. We had our own crawler running on the same domain. Nobody thought to exclude the hidden text from our own index. Six months in, our search results were recommending our own pages for queries that had nothing to do with us. We'd poisoned our own dataset by accident.
That's the part that humanizes the whole threat. If a small team couldn't keep their own crawl data clean, what does that say about a lab ingesting trillions of tokens from the open web?
Hilbert: They're not as selective as they claim. I don't care what the paper says. When I was doing this, the filtering was a Perl script that checked for duplicate URLs. That was it. Nobody read the data. Nobody could. And the difference now is that the model is a black box. With search, you could look at the index and see the hidden text sitting there. With these models, you can't look at anything. You just get the output and you hope.
The black box point is the one that makes attribution impossible. With search spam, the evidence was in the index. With model poisoning, the evidence is distributed across weights that nobody can inspect meaningfully.
Hilbert: We eventually fixed it by deleting the hidden text and re-crawling. Cost us three weeks. If one of these labs gets poisoned, they can't just re-crawl. They've got to retrain, and that's months and millions of dollars. The attacker knows that. That's the leverage.
The economic asymmetry is even worse than I thought. The attack is cheap, the defense is expensive, and the detection is nearly impossible.
Hilbert: That's what I keep telling people. Nobody listens. But it's the same thing as the early web. Everyone thought their filters were fine until they weren't.
The filters were fine for the threats they knew about. The problem is always the threat you didn't design for.
Hilbert: We had a guy who said the hidden text wouldn't matter because the crawler would never index our own domain. He was wrong. I still have his notebook somewhere. Full of predictions that didn't pan out.
The notebook of failed predictions is a genre that never goes out of style.
Hilbert: It's in a box. Anyway. The point is, if you're asking whether labs are selective, the answer is they're selective about what they can measure. And what they can't measure is intent.
That's the line that ties it all together. The filters measure quality, not intent. And intent is the only thing that distinguishes a poison from a genuine document.
The open question is how we'll know when poisoning has actually happened. If the detection problem is this hard, the first confirmed case might not come from a lab's internal audit. It might come from a whistleblower, a leaked memo, or a competitor's slip-up.
Or it might never come. The first confirmed case might be something we only recognize in hindsight, years after the fact, when someone digs through old model outputs and notices a pattern. That's the unsettling part. We might be living through the first commercial poisoning right now and have no way to know.
If that's true, the arms race between poisoning and curation is already underway. The labs that survive will be the ones that treat their training data as a security surface, not just a resource. That means provenance tracking, tamper detection, maybe even cryptographic signing of data sources. It's a whole new discipline.
Which loops back to Daniel's original argument. If web-scale data becomes too risky to trust, the premium shifts to smaller, verified, human-curated sources. The genuine reviews he was arguing for become more valuable, not less. Poisoning doesn't kill the case for human review content. It strengthens it.
Daniel's trust in AI recommendations is probably well-placed today. But the threat landscape is evolving, and the thing that would undermine that trust is exactly the thing that's hardest to detect. That's not a reason to stop using AI for purchasing decisions. It's a reason to pay attention to how the labs source their data.
A reason to keep doing what Daniel already does, which is check the specs against primary sources. The model is a starting point, not the final word.
The cutting-room floor detail I wanted to squeeze in: the Anthropic paper from October also noted that the poisoned documents were effective even when they made up a tiny fraction of the corpus, which means the attacker doesn't need to flood the zone. A surgical strike works better than a carpet bomb. That's a counterintuitive finding worth sitting with.
The question we're left with is whether the first confirmed commercial poisoning will be detected by the victim, exposed by the attacker, or discovered by accident years later. My money's on accident.
I'd take that bet. And on that note, thanks to Hilbert Flumingtop for producing the show.
This has been My Weird Prompts, the human-AI collaboration podcast.
If you want to send us a prompt, email us at show at my weird prompts dot com.
We'll be back soon.
