Daniel sent us this one. He's been building an automated news pipeline for Israel and Iran updates, using a whitelist of sources that includes Iranian state media to get the regime's angle. The core challenge he's hitting is how to move beyond just churning out repetitive, RSS-style summaries of the same major headlines everyone else has. He wants to engineer the pipeline to reliably surface unique or less-reported developments alongside the big stories. The question is, technically, how do you do that without just amplifying noise or missing the main event?
That is a fantastic and timely engineering puzzle. It gets at the heart of what separates a useful intelligence tool from a simple aggregator.
And by the way, today's episode is being powered by deepseek-v3.
A new friend on the writing team. Okay, so Daniel's prompt cuts right to a major pain point in current news aggregation. Everyone gets the same AP headline about a ceasefire extension. Very few are automatically surfacing, say, the specific spike in drug prices for dialysis patients in Iran as a knock-on effect of strikes on infrastructure. Or the sudden, localized protests in southern Lebanon over fuel shortages that didn't make the international wires. The need for that kind of nuanced, multi-perspective awareness in a fast-moving situation has never been higher.
If your pipeline just regurgitates the top five stories from a standard feed, you've built a slightly faster version of a problem that already exists. The real value is in creating a shared situational awareness that includes the edges of the narrative, the data points that are true but aren't leading the broadcast. It’s like the difference between a weather report that just says “rain” and one that gives you precipitation density maps.
And we have concrete examples from just the last forty-eight hours that illustrate this gap perfectly. The major development is the Israel-Lebanon ceasefire being extended for twenty-one days. That's in every summary. The less-reported development, visible in satellite data analysis, is the scale of non-military damage in Iran from recent strikes—over seven thousand six hundred buildings, including schools and water plants. But here’s a fun fact: that satellite data number? It came from a group called Conflict Ecology, which is essentially a bunch of researchers applying open-source intelligence (OSINT) techniques to environmental and infrastructure damage. They’re not a news outlet, but their data is a critical, novel input.
The engineering challenge is to build a machine that can see both of those things with equal clarity, weigh them appropriately, and present them without one drowning out the other. Let's dig into how you'd actually do that—Herman, I know your team's approach starts with the format itself.
The uniqueness starts with the format. Daniel mentioned he's using the SitRep format we've talked about. That's key. A standard RSS feed aggregates headlines for consumption. A SitRep aggregates data points for decision-making. The goal isn't to keep someone scrolling; it's to create a coherent, multi-source picture of the operational environment. Think of it as the difference between a news ticker and a pilot's instrument panel.
Which is why the whitelist approach, especially including a source like Iranian state media, is fundamental. You're not casting a wide net hoping to catch contrary views. You're deliberately placing sensors in specific, strategic locations. One of those sensors is inside the regime's own narrative factory.
And that's what separates this from a standard feed. Most aggregators might include Reuters, AP, maybe Al Jazeera. Including Press TV or IRNA isn't about taking their reporting at face value as ground truth. It's about capturing, as a primary data stream, what the Iranian government wants its population and the world to believe is happening. That perspective is a strategic fact in itself. But how do you technically handle that? Do you just dump it all in with everything else?
Right, that’s the first design decision. So the pipeline's first job is to ingest these designated sources with that understanding. It's not a democracy where every vote counts equally. It's an intelligence gathering operation where each source has a known bias and a specific informational purpose. You need a source taxonomy from day one.
The whitelist defines the battlefield of information—but the real engineering begins when you process that raw, biased, voluminous input. Take Iran's state media alone: they publish about one hundred twenty English articles daily on regional affairs. The core challenge? How do you filter that torrent to highlight not just the loudest story, but the most meaningfully divergent ones? The naive approach would track the most frequently mentioned entities—Israel, Hezbollah, Natanz—and call those the top stories. But frequency just gives you consensus. To surface unique angles, you need to invert that logic entirely.
Instead of measuring commonality, you're measuring divergence. But hold on—how do you even define your baseline “consensus”? Is it a rolling average of what your Western sources say? What if they’re wrong, or late?
That’s the critical setup. You start by establishing a dynamic baseline of what's being widely reported across your trusted, non-adversarial sources—say, your Western and Israeli outlets. That's your "consensus cluster." It’s not about truth, it’s about volume and overlap from sources with high factual reliability. Then, you run every incoming article, including from your adversarial sources like Iranian state media, through a comparison. The algorithm isn't asking "is this popular?" It's asking "how different is this from the established picture?
You'd quantify that difference how? The presence of entities or topics that are absent from the consensus?
Both, in layers. The first filter is simple entity and topic analysis. If the consensus cluster is heavily focused on "ceasefire extension" and "diplomatic talks," but an IRNA article spends significant verbiage on "Zionist buffer zone violations" and "Lebanese sovereignty," that's a flag. The topic is the same, but the framing and emphasis are divergent. That's a data point for the SitRep: "Iranian media framing ceasefire around Israeli territorial demands." But that’s relatively easy.
That seems straightforward. But what about articles that introduce entirely new entities or facts not in the consensus stream? That's the holy grail—the completely unreported angle. Like that dialysis drug price spike.
That's the second, more valuable layer. You need a mechanism to detect novel claims or data points. This is where named entity recognition and relation extraction get serious. Let's take the non-military damage report. If the consensus is talking about military targets hit, but a Conflict Ecology report—or even a passing mention in a regional Iranian outlet—starts listing specific numbers of "schools" and "hospitals" damaged, those are novel entity-relation pairs. The algorithm should score that article very high for uniqueness. But, here’s a natural follow-up: how do you stop it from just grabbing any random, new claim?
Yes, the risk there is noise. One obscure blog post might claim aliens mediated the ceasefire. That's novel, but it's garbage. Or, a state media outlet might invent a brand new “martyr” or a fictional battle location to test the waters.
A critical point. You don't just weight for novelty alone. You have to weight for novelty within a source of a certain baseline credibility. That's why the whitelist is so important. You've pre-vetted these sources. An outlandish claim from a known propaganda outlet is still a data point—it tells you what the regime is trying to inject. An outlandish claim from a random X account is just noise. The system trusts the whitelist, then measures divergence within it. But even within the whitelist, you need tiers.
The weighting mechanism is twofold. First, source-tier weighting. You might have a core tier of fact-based reporting outlets, and a perspective tier for state media and advocacy groups. Second, content-based weighting based on divergence from consensus.
And you can get even more sophisticated. For the perspective-tier sources, you might actually penalize them for repeating the Western consensus. If Press TV is just paraphrasing Reuters, that's low value. But if it's offering a completely different causal narrative—attributing the ceasefire to "the resistance's steadfastness" rather than US diplomacy—that's high value for understanding their posture. Conversely, for your core tier, you might penalize excessive divergence, as it could indicate an error or outlier report.
Let's walk through a case study with the recent ceasefire extension. How would this pipeline handle it step-by-step?
The consensus cluster from AP, Reuters, Haaretz, Times of Israel lights up with the core facts: ceasefire extended twenty-one days, US mediation, aims to prevent escalation. High frequency, high overlap. That's your major development. It gets surfaced.
It's tagged as "consensus-high." Meaning, everyone has this.
Now, the pipeline simultaneously processes the Iranian state media output. It finds articles that mention the ceasefire, but the entity and sentiment analysis shows a heavy focus on "American pressure on the Zionist regime" and "violations of Lebanese land." That's a divergent framing. It gets tagged as "perspective-divergent" and included as a secondary bullet: "Iranian narrative emphasizes US role and Israeli violations, rather than mutual de-escalation.
Then, maybe a report from the Arab Center D C or American Progress comes in, discussing the humanitarian impacts pre-dating the ceasefire—the drug price spikes, the infrastructure damage. These contain entities and relations largely absent from the main ceasefire story.
Those are novel clusters. They're not about the ceasefire per se; they're about the enduring second and third-order effects of the conflict period. The algorithm detects the low overlap with the consensus cluster but high coherence within its own report, and because it's from a whitelisted, credible policy institute, it gets a high "novelty-credible" score. It surfaces as a separate, but equally important, development: "Pre-existing humanitarian crises in Iran exacerbated by strikes, with dialysis drug prices up fifty percent." But here’s an expansion on that: the system might also cross-reference that drug price claim with specialized trade or health databases, if available, to add a confidence score.
The output isn't one story. It's a layered briefing. Major consensus event, divergent political narratives around that event, and novel reporting on adjacent, under-covered consequences. The tradeoff you mentioned earlier is between breadth and depth. A broader whitelist catches more novelties but increases noise. A deeper, more complex scoring system requires more tuning and risks over-weighting esoteric details. The key is to design the scoring so it balances surprise with significance. A novel fact about a minor official's statement is less significant than a novel fact about seven thousand buildings being damaged.
And you avoid the main pitfall of just creating a "contrarian for the sake of it" feed. The baseline isn't "what's popular"—it's "what's empirically happening according to our best sources." That's why divergence and novelty are measured against that baseline, not against mainstream opinion. It's a technical measure, not an ideological one. But how do you technically define “significance”? Is it the entity type? A school vs. The number count?
That's where the balance between surprise and significance becomes crucial. Once you start surfacing these divergent and novel items, you're not just tweaking a summary—you're shaping an entire narrative over time. A pipeline like this doesn't just report the news; it defines what counts as news for its users. That’s a huge responsibility.
Which is a massive knock-on effect. If an analyst or policymaker comes to rely on this SitRep as their primary situational picture, their mental model of the conflict is being built on a deliberately constructed information diet. The benefit is a richer, more nuanced understanding. The risk is developing blind spots if the algorithm's "significance" weighting has a hidden flaw. For example, if it overweighted military entities and underweighted economic ones, you’d miss the drug price story entirely.
Or if an adversarial source learns to game it. If Tehran's information ops team figures out that Press TV gets maximum points for articles that mention "Zionist war crimes" and "schools" in the same breath, they might start fabricating those specific entity pairs to pollute the pipeline with emotional triggers disguised as novel humanitarian reporting. They could weaponize your own novelty detection.
A brilliant and worrying point. Source credibility on a whitelist is static, but source behavior is dynamic. This makes continuous validation against ground-truthing sources absolutely critical. You need a feedback loop where the system's "novel" findings are occasionally spot-checked against, say, satellite imagery providers or trusted NGO reports on the ground. Otherwise, you've built a system that can be expertly poisoned. It’s an arms race in algos.
Which brings us to the practical implications for the analyst using this tool. They move from being a passive consumer of headlines to an active interpreter of a multi-source data fusion. The skill set shifts from "what happened" to "why is this source saying this this way, and why is that fact appearing now?" That’s a higher cognitive load.
It demands a higher level of media literacy, ironically, even as the tool tries to automate that literacy. The output isn't a single truth. It's an annotated landscape of truths, half-truths, and narratives. For a policymaker, that's more valuable than a simple summary, but it's also more cognitively demanding. The pipeline's presentation layer is as important as its sorting algorithm. You need clear visual tagging: "Consensus Event," "Regime Narrative," "Novel Humanitarian Data." Maybe even a confidence indicator.
Let's look at a concrete case study on framing from the recent strikes. How did Iranian state media frame the non-military damage compared to, say, a Western outlet like Reuters, in more granular detail?
According to the research, the satellite data showing over seventy-six hundred buildings damaged came from groups like Conflict Ecology. A Reuters piece might mention "collateral damage" or "infrastructure hits" within a broader military story. The focus is on the event and the military outcome. The damage is a subordinate clause.
The Iranian take?
Iranian state media, when it acknowledges such damage at all—which is not guaranteed—would frame it exclusively as deliberate, criminal targeting of civilian life by the "Zionist enemy" and its "American masters." The fact of the damage is the same. The attribution of cause and intent is radically divergent. The pipeline would surface both: the factual report of damage and the two competing frames for why it happened. That gives the analyst the complete informational battlefield. But to get that, your pipeline needs to be good at extracting relations: not just "school-damaged" but "school-damaged-by-Israel" vs. "school-damaged-in-crossfire.
It turns the news from a bulletin into a dialectic. You see the thesis, the antithesis, and maybe you have to synthesize the reality somewhere in the middle. This is where this whitelist-and-divergence method pulls ahead of other common aggregation techniques. Sentiment analysis, for example, would just tell you "Iranian sources used negative words." That's trivial. Event extraction might miss the framing entirely by just pulling "Building-Damaged" tuples. Our method preserves the narrative context as data, which is often where the real intelligence value lies.
The ethical considerations here are thick, though. By including Iranian state media on the whitelist, are we legitimizing a propaganda arm of a regime that calls for Israel's destruction? This isn't an abstract question; it's a real design and presentation challenge.
It’s the core ethical tension. My view, from an intelligence gathering perspective, is that it's not about legitimization; it's about monitoring. You include it for the same reason you'd tap a phone line—not because you believe what's said, but because you need to know what's being said. The ethical burden falls on the presentation. You must never present Press TV's output neutrally. It must always be tagged as "Regime Perspective" or "Adversarial Narrative." The user must never forget the source's nature.
The pipeline itself has to treat that source's text as valid input for its divergence algorithms. It's a strange dual reality: the system technically respects the source's data enough to process it, while the human operator fundamentally disrespects its truthfulness. How does the engineering reflect that?
That's the tightrope. The engineering handles it by assigning a low baseline credibility score but a high potential novelty score. The system says, "This source is usually unreliable on why things happen, but it is a primary source for what the regime wants people to believe is happening." That's an ethically sound, strategically critical distinction. You’re treating it as a signal of intent, not a source of fact.
The ultimate implication is that this isn't a "news app." It's an intelligence fusion engine for open-source information. Its purpose isn't to inform the public, but to create decision-making advantage for its users by automating the most tedious parts of all-source analysis—spotting patterns, gaps, and novelties across a hostile information environment.
You've nailed it. And in a conflict where information asymmetry is a weapon, building a better, faster, more nuanced picture than your adversary is no longer a nice-to-have. It's a strategic necessity. The pipeline Daniel's building—with that two-layer weighting system, source-tier first, then content-based divergence—doesn't just summarize the day. It helps you see around the corners of tomorrow.
So the actionable insight for anyone building a similar pipeline is to start with that same approach: source-tier weighting first, then content-based divergence. That's what gives you the edge. And the tuning happens in the scoring thresholds. You decide how much overlap constitutes "consensus." Is it eighty percent entity match? You decide the penalty for a perspective source echoing consensus. And most critically, you define the "significance" score for novel items. Is it based on the entities involved? Civilian casualty numbers get a higher score than minor diplomatic statements. Is it based on the source's historical reliability on that topic?
That last one is dynamic. If a source frequently reports novel facts that later get corroborated by your ground-truth sources, its novelty credibility score should increase over time. The system learns who cries wolf and who spots real wolves first. That feedback loop is what turns a static system into a learning one.
That's the feedback loop. For surfacing narrative divergence automatically, the technique is to use semantic similarity clustering, but then analyze the differences within clusters. Don't just group similar stories; flag the stories in the group that use a different set of causal verbs or sentiment-bearing adjectives. If ninety articles say "ceasefire extended," and ten say "ceasefire imposed," that divergence is a signal, not noise. It tells you about power dynamics.
Listeners can absolutely apply this to other contexts. Ukraine-Russia, China-Taiwan, any adversarial dynamic where understanding multiple narratives is key. The core engineering is the same: a curated whitelist, a clear source taxonomy, and algorithms that measure against an empirical baseline, not an ideological one. You could even use it for competitive business intelligence.
The practical first step is to stop thinking in terms of "news feeds" and start thinking in terms of "source sensors." Each outlet on your whitelist is a sensor with known biases and blind spots. Your pipeline is a fusion center. Your job isn't to find the truth, but to map the space between the sensor readings. That shift in mindset is the most important takeaway—though it does raise new questions about how adversaries might adapt. That sensor fusion mindset opens the door to adversarial gaming. If this becomes a widespread tool for analysts, how might bad actors start to exploit the system itself? Not just poisoning a single source, but designing entire campaigns to target the pipeline's novelty and divergence detection.
It’s an arms race. If they know you're looking for semantic divergence, they could flood your whitelist with minor, fabricated variations on a core propaganda line. If you're tracking novelty, they could create fake NGOs or sock-puppet news sites that report dramatic but false "humanitarian" data to hijack your significance scoring. The defense has to be as dynamic as the offense—continuously auditing your source list and requiring multi-source corroboration for high-significance novelties.
Which points to the broader future implication. The real role of AI here isn't just aggregation; it's in combating that information asymmetry at scale. An AI that can't just read, but can reason about source reliability, detect coordinated inauthentic behavior across your whitelist, and model the likely intent behind a narrative shift. That's the next layer. The pipeline we're describing is smart, but it's still following rules. The adversary can learn the rules. The future is a system that learns the adversary's playbook and adapts its own rules in response. That's how you maintain the decision-making advantage.
We'll have to leave it there for today. A huge thanks to our producer, Hilbert Flumingtop, for keeping the whole operation running. And thanks to Modal, whose serverless GPUs power the automation behind the scenes, and probably behind a few news aggregation pipelines too.
If you found this dive into information engineering useful, please leave us a review wherever you listen. It makes a real difference.
This has been My Weird Prompts. I'm Corn.
I'm Herman Poppleberry. Until next time.
