Daniel's follow-up from the corpora episode. He says the poisoning research left him with a specific worry. Threat actors, nation-states, propagandists, all of them now have a documented technique for seeding content into training data. The safeguard he wants to understand is the filtering mechanism. His question is, what is this pipeline in general? Is any of it happening at the text corpus level, like Common Crawl itself, or is it all being done downstream by the providers when they preprocess for a training run? And when people talk about preventing poisoning, or keeping out inflammatory, low-quality, or safety-violating content, what are the actual tools and technologies in use?
So we went looking for the filters, and what we found is a pipeline that's both more sophisticated and more fragile than the word filtering suggests. The first thing to get straight is the scale. Common Crawl is over two hundred fifty billion pages as of last year. That's not something anyone reads. It's not even something anyone can meaningfully sample by hand. So every decision about what gets into a training run has to be automated, and every automated decision is a place where something can slip through.
Two hundred fifty billion pages. That's roughly thirty pages for every human alive, and we're supposed to believe a blocklist handles it.
A blocklist is maybe stage four of six. Let me map the whole thing, because the misconception Daniel's poking at is that filtering is a single toxicity check. It's not. The pipeline starts the moment a crawl lands. Raw scrape, then language identification, then deduplication, then quality filtering, then toxicity and safety filtering, then final curation and mixing. Six distinct stages, each with different tools, different failure modes, and different people making the calls.
And the first thing that happens to those two hundred fifty billion pages is that someone throws most of them away for being duplicate or unreadable.
Right. Deduplication is the least glamorous stage and probably the most important. There are two flavors. Exact dedup, where you hash each document and drop anything you've seen before. And fuzzy dedup, where you use something like MinHash to catch near-duplicates, pages that differ by a few words or a boilerplate footer. The reason this matters is that the web is astonishingly repetitive. News articles syndicated across forty outlets. The same Wikipedia paragraph mirrored on a thousand scraper sites. If you don't dedup, the model just memorizes the repetition and spends capacity on nothing.
And repetition is itself a poisoning vector, right? If I want a model to believe something, I don't need to inject one authoritative source. I just need to say it ten thousand times across ten thousand junk domains.
Volume is the oldest attack in the book. Dedup is the first real countermeasure, because it collapses my ten thousand junk pages into one. Or near enough. MinHash will catch pages that are ninety-five percent identical, so my strategy of spinning the same paragraph with a different headline mostly fails.
Mostly.
Mostly. The fuzzy threshold is a tradeoff. Set it too tight and you keep duplicates that differ by a sentence. Set it too loose and you start dropping distinct pages that happen to share boilerplate. Every parameter in this pipeline is a tradeoff, and that's the thing I want to keep coming back to. There is no clean setting.
So dedup first. Then language ID, which sounds trivial but isn't, because the web is full of mixed-language pages, and a page that's ninety percent English with a paragraph of Hindi can confuse a model if it's not labeled properly.
And language ID is usually a fast classifier, something like fastText. It scores each document and you set a confidence threshold. Below the threshold, the page goes in the bin. But here's the thing, fastText is itself a model. It has its own biases. It's better at some languages than others. Low-resource languages get misclassified more often, which means filtering for language is also filtering for the languages the filter happens to recognize.
That's the first of several places where the sieve has opinions.
And it gets more opinionated from there. After language ID and dedup, you hit quality filtering. This is where the real judgment starts. The common approach is to train a classifier on human-annotated examples. You pay annotators to label a few thousand pages as high quality or low quality, then you train a model to predict that label at scale. The FineWeb pipeline did exactly this. They took Common Crawl, applied URL filtering, dedup, and then a custom quality classifier trained on human judgments. The full crawl was ninety-six terabytes. After their filtering, the final dataset was forty-four terabytes.
So more than half the corpus got thrown out, and that was considered a good outcome.
A very good outcome. The FineWeb paper showed that training on that aggressively filtered forty-four terabytes beat training on much larger, less filtered corpora. Their models did better on benchmarks like MMLU. The quality classifier was doing real work. But, and this is the part that doesn't get quoted as often, over-filtering hurts. The same paper noted that aggressive filtering reduces diversity. It can hurt performance on tasks that require exposure to messy, real-world language. Hate speech detection is the canonical example. If you strip all the hate speech out of your training data, your model never learns what hate speech looks like, and then it's worse at identifying it.
So the safety filter makes the model less safe at the thing the safety filter was for.
In that specific way, yes. The model becomes safer at not generating hate speech, but worse at recognizing it in the wild. That's the tradeoff in one sentence. And it's not hypothetical, it's in the FineWeb results. Filtering is not a free lunch. Every stage removes something, and sometimes the thing it removes is the very capability you wanted.
Alright, so we've got language ID, dedup, quality classification. Then toxicity. What's the machinery there?
Toxicity filtering is usually a combination of blocklists and classifiers. The blocklist approach is straightforward. You maintain a list of domains, URLs, and sometimes specific terms that are known to be bad, and you drop anything that matches. Common Crawl itself does a version of this at the corpus level, which I want to get to in a minute. The classifier approach is more sophisticated. Tools like Detoxify, which is an open-source model, or the Perspective API from Google's side, score text for toxicity, threats, insults, identity attacks. Anything above a threshold gets removed or downweighted.
And the Dolma dataset from AI2 uses both, right? Blocklists plus Detoxify-style classifiers.
Right. Dolma's pipeline includes toxicity filtering with blocklists and classifiers. NeMo Curator, which is NVIDIA's data curation toolkit, offers modular steps for language ID, quality scoring, and toxicity detection. Datatrove is the open-source tool that was used to build FineWeb itself. And then there's DCLM, DataComp-LM, which is a benchmark and toolkit for data curation. It's built around the question of how you trade off filtering against diversity. Four different toolkits, four different philosophies, all solving the same problem in slightly different ways.
And none of them agree on the right threshold.
They can't, because the right threshold depends on what you're building. A model for customer service chatbots needs different filtering than a model for academic research. A model that will be fine-tuned for medical applications needs different filtering than a general assistant. The pipeline is always tuned to the destination.
Which is where Daniel's actual question lands. Where does this happen? Is Common Crawl doing any of it?
Common Crawl's own FAQ is explicit about this. They provide raw web data. They do not perform content filtering beyond basic spam removal and deduplication. That's it. No toxicity filtering. No quality scoring. No safety classification. They're an archive, not a curator. Their job is to crawl the web and store what they find, month after month, in a format researchers can download.
So if you're a nation-state and you want to poison the next generation of models, you don't need to hack Common Crawl. You just need to put your content on the open web and wait for the crawler.
That's the uncomfortable part. Common Crawl will happily archive propaganda, misinformation, hate speech, all of it. It's not their job to judge. The judgment happens downstream, when a provider or a research lab takes that raw crawl and runs it through their own pipeline. OpenAI has their own filtering. Anthropic has theirs. Google has theirs. Every lab re-processes the same raw material through different tools with different thresholds.
Which means the filtering is fragmented. There's no single chokepoint where bad content is stopped. It's a dozen different sieves, each with different hole sizes, and an attacker only needs to find the overlap where all of them let something through.
Or target a specific provider's pipeline. If I know that Lab A uses a particular quality classifier and Lab B uses a different one, I can craft content that passes A but gets caught by B. The fragmentation isn't just inefficient, it's a security vulnerability. The attack surface is every pipeline, not just the corpus.
And the pipelines are increasingly commercial. Scale AI, Surge AI, these companies now offer data curation as a service. Human-in-the-loop filtering. You send them your raw crawl, they send back a cleaned dataset with human annotators having reviewed the quality.
Which sounds like the gold standard until you think about it for ten seconds. Human annotators are the new bottleneck. They can be biased. They can be coerced. They can be underpaid and rushed, which means their judgments are noisy. And they can be targeted. If I'm a nation-state and I know that a particular annotation vendor is handling the quality filtering for a major lab, do I try to get my people hired as annotators? Do I try to influence the annotation guidelines? The human-in-the-loop is also the human-in-the-loop-that-can-be-social-engineered.
There's a paper from last year, the Persistent Pre-training Poisoning paper, that tested exactly how hard this is. What did they find?
They found it's disturbingly easy. They poisoned a training corpus with a tiny amount of data, around zero point one percent, crafted to make the model produce a specific harmful behavior. And the poisoning survived. It survived the quality filtering. It survived the fine-tuning. It survived multiple rounds of safety alignment. Zero point one percent. That's one page in a thousand. In a corpus of billions of pages, that's a rounding error, and it was enough to change model behavior.
And the poisoned data didn't look poisonous. That's the key. It was designed to look benign to the classifiers. Semantically harmful, but superficially clean.
Right. The attack works by exploiting the gap between what the filter measures and what the model learns. A quality classifier looks at surface features. Is the text coherent? Is it well-written? Does it come from a reputable-looking domain? A toxicity classifier looks for known bad words and patterns. But a poisoning attack can be coherent, well-written, free of slurs, and still encode a specific harmful behavior. The filter sees nothing wrong. The model, which is much better at extracting semantic content, learns the poison.
The filter and the model are looking at different things, and the attacker aims for the gap between them.
The filter measures quality, not intent. And intent is the only thing that distinguishes poison from genuine content. A well-written essay about the benefits of a particular political ideology could be genuine or could be propaganda. The filter can't tell. It can only tell that the essay is well-written.
Which brings us back to the tradeoff problem. If you filter more aggressively, you catch more poison but you lose more signal. If you filter less aggressively, you keep the signal but you let more poison through. There's no setting that gets you both.
The research on data provenance and watermarking is still nascent. The idea is that instead of trying to detect bad content, you track where content came from. If a document can be traced to a known-good source, you trust it more. If it comes from a domain that didn't exist six months ago, you trust it less. But provenance is hard on the open web. Anyone can register a domain. Anyone can copy content and repost it under a new URL. The web is built to make provenance hard.
There's something almost philosophical about it. The web is a system where anyone can say anything, and the filtering pipeline is an attempt to impose order on that chaos after the fact. It's always going to be reactive.
It's going to get harder, not easier. Multi-modal training is the next frontier. Once models are ingesting images and video at scale, the filtering problem multiplies. A text filter can scan for bad words. An image filter has to understand visual content. A video filter has to understand sequences of visual content. The attack surface expands enormously, and the tools are even less mature.
Where does that leave us? The pipeline is real, it's sophisticated, it's doing important work. But it's not a silver bullet. It's a sieve, not a wall. And the people who want to poison models know how the sieve works.
The thing I keep coming back to is that the filtering happens at the wrong level. Common Crawl is the natural chokepoint. It's the single archive that everyone downloads. If filtering happened there, comprehensively, every downstream provider would benefit. But Common Crawl has explicitly chosen not to be that chokepoint. They're an archive. Their mission is completeness, not cleanliness. So the filtering gets pushed downstream, where it's fragmented and inconsistent.
The providers are in a race. They want the best model, which means they want the most data, which means they're incentivized to filter as lightly as possible while still claiming safety. The incentive structure pulls against thorough filtering.
That's the second-order problem. The labs that filter most aggressively are also the labs that throw away the most data, which can hurt performance. So there's a competitive pressure to under-filter. And the labs that under-filter are the ones most vulnerable to poisoning. It's a structural tension that no individual lab can resolve on its own.
Meanwhile, the nation-state attacker doesn't have any of these constraints. They don't care about model quality. They don't care about diversity. They just want their poison in the training data, and they have infinite patience.
The evidence suggests they're succeeding. Or at least, that they could succeed if they tried. The Persistent Pre-training Poisoning paper wasn't a theoretical exercise. It was a demonstration that the current defenses are insufficient. Zero point one percent. That's all it took.
You know, this reminds me of something.
Hilbert: We had a blocklist for adult content at WebSift. This was the early two thousands. I was running a small crawling operation for a search engine that doesn't exist anymore. WebSift. We crawled maybe forty million pages a month, which at the time felt like the whole internet. And we had this blocklist. It was a text file with about eight thousand terms in it. Every page that came through got checked against the list. If it matched, it went in the quarantine folder. And every morning I'd sit there and go through the quarantine folder by hand, because the blocklist was wrong about a third of the time.
A third.
Hilbert: The list had one entry for gambling. Just the word. And it blocked an entire forum about probability theory. Six thousand posts about Bayesian inference, gone, because someone used the word gambling in the title once. I had to add an exception for that domain. The exception list ended up longer than the blocklist by the time we shut down.
You were doing quality filtering by hand, with a blocklist plus human review.
Hilbert: That was the whole job. The spammers figured out the blocklist in about a week. They'd misspell words. Gambling with two Bs. They'd put the text in an image so the crawler couldn't read it. They'd use Unicode characters that looked like letters but weren't. We'd add the misspelling to the list, they'd find a new misspelling. It never ended.
A cat-and-mouse game.
Hilbert: It's the same game now. The spammers just have better tools. Instead of misspelling words, they're generating text that looks clean to a classifier but teaches a model something wrong. The principle hasn't changed. The filter measures something, the attacker optimizes against the measurement. You can't win that game. You can only keep playing.
The hand-filtering part is what strikes me. You were the human in the loop. You caught the probability forum and fixed the error. But at scale, with two hundred fifty billion pages, there's no human in the loop. The loop is all classifiers, and the classifiers make the same kind of blunt errors you made, except nobody's there to add the exception.
Hilbert: The exception list was the real filter. The blocklist was just the first pass. The exceptions were where the judgment lived. And the exceptions took years to build. Every week I'd find some new thing the blocklist was wrong about and I'd add an exception. By the end, the system was mostly exceptions. The blocklist was just the thing that generated the queue for the exception list.
The modern equivalent of the exception list is what, the human annotation vendors? The Scale AIs of the world?
Hilbert: They're doing the same job I did, just with more people and less context. I knew my forty million pages. I knew which domains were legitimate. These annotation vendors, they're looking at pages in isolation. They don't know the domain's history. They don't know the context. They're making judgment calls with less information than I had, and they're doing it for a corpus that's a thousand times bigger.
Which is exactly the vulnerability. The human judgment that catches the probability forum error doesn't scale. So the modern pipeline either automates it, which reintroduces the blunt errors, or fragments it across thousands of annotators, which introduces noise and coercion risk.
Hilbert: I don't know how you solve it. We never solved it. We just kept the blocklist updated and the exception list growing and we shut down before it got worse. The web got too big for hand-filtering sometime around two thousand five. After that, it was all automated, and the automated systems made the same mistakes faster.
There's something almost comforting about that. The problem isn't new, and it isn't uniquely AI-shaped. It's the same arms race, just with better weapons on both sides.
Hilbert: The weapons are better. The fundamental problem is the same. You're trying to decide what's true and what's false at a scale where nobody can check the answers. We couldn't do it with eight thousand terms and forty million pages. They can't do it with whatever they're using now and two hundred fifty billion pages. The numbers just got bigger.
The pipeline is a necessary but insufficient defense. That's the honest summary. It catches the obvious stuff. It deduplicates, it filters for language, it scores for quality, it blocks known-bad domains. But it can't catch a determined adversary who's optimizing against the filter itself. And the more we rely on the filter, the more the adversary optimizes against it.
The open question is whether filtering can ever be truly robust, or whether it's always going to be a game of catch-up. My sense is the latter. The attacker has the advantage, because the attacker only needs to find one gap, and the defender has to close all of them.
The gap is widening as models go multi-modal. Text filtering is hard enough. Image and video filtering is a whole different problem, and the poisoning techniques for visual data are even less understood.
The real safeguard might have to come from the model level, not the data level. Alignment, safety training, interpretability. Things that happen after the data is already ingested. Because the data pipeline can't guarantee clean input. It can only reduce the probability of dirty input.
Which is a humbling place to land. The entire industrial apparatus of data curation, all the toolkits and classifiers and annotation vendors, and the honest conclusion is that it's a sieve, not a wall. It reduces the poison, it doesn't eliminate it.
The one thing I'd add is that the sieve is worth having anyway. Even if it's not perfect, it's doing real work. The difference between no filtering and the current pipeline is the difference between a model that's trivially poisonable and one that requires a sophisticated adversary. That's not nothing. It's just not everything.
Daniel's question was whether the filtering happens at the corpus level or downstream. The answer is both, but the important filtering is downstream. Common Crawl is a raw archive with minimal processing. The real work happens in the provider pipelines, and that's where the fragmentation and the vulnerability live.
The tools are datatrove, NeMo Curator, DCLM, Dolma's stack. Blocklists, quality classifiers, toxicity models like Detoxify. A whole ecosystem of machinery that most people never see, doing the unglamorous work of deciding what a model gets to learn from.
The question that stays with me is whether the next generation of defenses will be about filtering better, or about building models that are less fragile to poisoned data in the first place. Because the filtering arms race is one the defenders have been losing since the early two thousands, and I don't see that changing.
Thanks to Hilbert Flumingtop for producing, and for the blocklist war stories. This has been My Weird Prompts. If you got something out of this episode, leave us a review wherever you listen. We'll be back soon.
