Daniel sent us this one — he's come across a new Israeli poll with some pretty staggering numbers. Ninety-two percent of Israelis surveyed think Iran emerged victorious from the war, and here's the kicker — ninety-three percent of people who voted for Netanyahu feel the same way. Seventy-two percent flat-out don't believe the Prime Minister's claim that Israel removed an existential threat. Only twenty-six percent rate his war handling as good or excellent. What Daniel's really asking is about the mechanics of trusting a poll like this. Sample of three thousand six hundred forty-four from a population of nearly ten million — is that actually representative? What factors would undermine the credibility of findings like these, and what would solidify them?
This is exactly the kind of question that separates thoughtful news consumers from the rest of us staring at a headline in slack-jawed disbelief at eight in the morning. Let's dig into this — because the poll matters, but how we evaluate it might matter more. And I want to say upfront, the reason Daniel's question is so good is that he's not asking "is this true." He's asking "how would I know if it's true." That's a completely different muscle.
Right — he's not rejecting the finding or embracing it. He's interrogating the process that produced it. So let's unpack what this poll actually tells us — and more importantly, what it doesn't.
When you see a survey with a sample of three thousand six hundred forty-four Israelis drawn from a population of about nine point eight million, the instinct is to ask how much precision that gives you. The math isn't hard to walk through — the standard margin of error formula at ninety-five percent confidence is roughly one point nine six times the square root of point-five times point-five divided by the sample size. With three thousand six hundred forty-four respondents, that works out to about one point six percent. So that ninety-two percent who say Iran won — the actual confidence interval is roughly ninety point four to ninety-three point six. That's pretty tight.
Meaning even at the most generous end of that range, ninety percent of the country thinks they lost. There's no version of that error bar that turns into "mixed feelings" or "a divided public.
And there's this thing called the finite population correction factor, where people worry that three thousand people can't possibly speak for ten million. I hear this all the time — "how can three thousand people represent millions?" It feels intuitively wrong. But the math there is equally clear. The correction factor multiplies your margin of error by the square root of the population minus sample divided by population minus one. When your sample is three thousand six hundred forty-four and your population is nine point eight million, that ratio is so tiny — about zero point zero zero zero three seven — that the correction is basically one. The sample is more than large enough for national estimates.
We can stop worrying that the sample is somehow too small to represent the country. But I want to pause on that intuition you mentioned — the feeling that three thousand can't possibly capture millions. Why does that feeling persist even when the math says otherwise?
Because our brains evolved to reason about small groups, not large populations. If you're in a village of two hundred people and you want to know what everyone thinks, you really do need to ask most of them. We carry that intuition into modern statistics, but it doesn't transfer. Once your population is above about twenty thousand, the sample size you need for a given precision barely budges no matter how big the population gets. A sample of three thousand gives you roughly the same margin of error whether you're polling Israel, the United States, or the entire planet. That's wildly counterintuitive, but it's true. It's why national polls in America — a country of three hundred thirty million — routinely use samples of one thousand to fifteen hundred. The math doesn't care about the size of the ocean once it's deep enough.
That's a helpful anchor. So for the headline number, the sample size is solid. But here's the first trap people fall into — and Daniel hinted at it in his question. The ninety-three percent figure for Netanyahu voters moves us into subgroup territory. And subgroups change everything.
So if Netanyahu voters make up roughly thirty percent of the sample — which is a reasonable estimate given recent election results — we're looking at about one thousand ninety-three people. Same formula applied to a subgroup of that size bumps your margin of error to around three percent. So instead of ninety point four to ninety-three point six, you're looking at something like ninety to ninety-six. Still an overwhelming finding — but you can see how the precision erodes.
If you go even deeper — say you want to look at Netanyahu voters under thirty, or Netanyahu voters in a specific region — the subgroup gets smaller still, and suddenly your error bars are wide enough to drive a truck through.
That's exactly the danger. People see a big headline sample and assume every number in the crosstabs carries the same precision. They don't. I've seen polls with three thousand respondents where a demographic slice of two hundred people gets quoted as if it's gospel, and the actual margin of error on that subgroup is seven points. You can't say anything meaningful with that kind of spread. Let me give you a concrete case. Imagine a poll that reports "seventy percent of Israeli Arabs support policy X," and you get excited about that number — but then you discover the Arab subgroup in the sample was only a hundred and fifty people. That margin of error is around eight percent. The real number could be anywhere from sixty-two to seventy-eight. That's not a finding, that's a shrug.
News organizations rarely flag this. They'll report the subgroup number with the same confidence as the topline.
Because flagging it makes the story less clean. "Ninety-three percent of Netanyahu voters agree" is a crisp headline. "Ninety-three percent of Netanyahu voters agree, though the confidence interval spans six points and the subgroup may not have been weighted independently" is not a crisp headline. But the second version is closer to honest. And this brings us to something deeper. That margin of error number doesn't account for everything that can go wrong. It only covers sampling error — the luck of who you happened to reach.
This is where most people's understanding breaks down. The margin of error is a statement about random chance. It says, if you had drawn three thousand completely random Israelis from a perfectly representative swimming pool of all nine point eight million, your result would fall within the interval ninety-five times out of a hundred. But polling reality is nothing like that swimming pool. You're not reaching a random slice. You're reaching the nine percent of landline holders who still answer unknown numbers at dinner time. Or the people registered on online panels who get a few shekels for their time.
Right — and the gap between the swimming pool ideal and the messy reality is where most polling failures actually live. Sampling error is the clean, mathematical part. Non-sampling error is everything else — coverage bias, non-response bias, measurement error, weighting error. And non-sampling error isn't captured by the margin of error at all. Your one-point-six percent confidence interval might be perfectly calculated for a sample that's systematically skewed in ways the math can't see.
If we run with a sample around three and a half thousand when the response rate's dropped from about thirty percent in the year two thousand to under ten percent now... that means to end up with three thousand six hundred responses, they might have dialed something like forty thousand numbers.
Right, and deciding who was available to answer those forty thousand calls in the forty seconds before the automated dialer gave up — suddenly you've filtered through a very specific subset of the population. Young mobile-only users? They're virtually unreachable by traditional phone polling. Ultra-Orthodox communities with different media habits? Israeli Arabs who may be reluctant to answer pollsters or may not have been represented proportionally in the sample frame. These things don't show up in your neat one-point-six percent error bar.
Of course they don't. That error bar is like a camera lens that tells you how sharp the focus is, but doesn't tell you whether you've pointed the camera at the wrong garden.
And let me extend that metaphor, because I think it's useful. The margin of error tells you the resolution of the image. But if you've pointed your camera at a garden that only contains retirees who answer landlines, no amount of resolution will show you what twenty-five-year-olds think. You've got a crystal-clear picture of the wrong thing.
The problem is, the wrong thing can look very convincing. A sharp, well-lit photo of the wrong garden still looks authoritative.
That's the seduction of quantitative data. The precision feels like accuracy. But precision and accuracy are different things entirely. A thermometer that consistently reads two degrees too high is precise but not accurate. A poll can have a razor-thin margin of error and still miss the true population value by ten points because of non-sampling error. And there's another piece of this — weighting. Every pollster compensates for the people they couldn't reach by rebalancing the numbers. Say your sample came back with too many older respondents and too few younger ones. The polling firm will weight the young respondents they did get more heavily to match the census distribution. That's standard practice, and when done transparently it works well. But when done opaquely, you get the twenty sixteen U.Polls missed state-level results largely because they didn't weight for educational attainment. Working-class white voters without college degrees were profoundly underrepresented in landline telephone polls, and the weighting schemes didn't catch the gap until it was too late.
Do we know what weighting was used in this Israeli poll?
We don't — and that's the red flag Daniel was probing for. The poll was covered in a Middle East Eye piece reporting the ninety-two percent finding, and highlighted by the Independent. But when I looked at the Israel Democracy Institute's own site for methodological details beyond that broader ten-article release, the parameters weren't spelled out in lay summaries — we know the questions posed, but whether weighting was by age, gender, region, and past voting behavior, or just a couple of those, that's not clear from the published summary. And that's exactly the sort of thing a critical reader should notice. If the weighting scheme is opaque, you mentally inflate your uncertainty.
Let me push on that — because weighting sounds like a technical footnote, but it can completely invert a result. Can you give a concrete example of how that plays out in practice?
Imagine you're polling support for a policy, and by raw numbers your survey shows sixty percent approval. But you notice your sample is sixty-five percent male, when the actual population is fifty percent male. And it turns out men support the policy at much higher rates than women. So you weight the women up and the men down to match the real population distribution — and suddenly your sixty percent approval drops to fifty-two percent. A single weighting variable flipped the narrative from "majority support" to "country evenly split." Now imagine you didn't weight for that variable at all, or you weighted for age but not gender, or your gender targets came from a census that's ten years out of date. The point is, weighting choices are editorial choices, whether pollsters admit it or not.
When the Israel Democracy Institute publishes a finding and we can't see the weighting scheme, we're essentially taking their word that they balanced the scales correctly.
And that's not to say they didn't — the IDI is a respected institution. But the principle holds: if you can't see the recipe, you can't judge the dish. This ties into the challenge — I recall past episodes dealt with serious uncertainty around Palestinian polling. That domain had direct relevance to understanding this in a smaller war after twenty-four. Gaza didn't produce solid full demographically-weighted sample polling because logistics were impossible amid displacement and conflict. Israel here is obviously a freer environment to answer — political will doesn't force wrong format structure.
That comparison is worth sitting with for a second. In Gaza, you literally cannot conduct a proper random sample because the population is moving, the infrastructure is destroyed, and people are in survival mode. So any number coming out of that environment has to be treated as directional at best. Israel, by contrast, has functioning institutions, stable infrastructure, and a population that can be reached — but that doesn't mean the polling is automatically golden. It just means the problems are different problems.
The challenges shift from "can we physically reach people" to "are we reaching the right people in the right proportions, and are they telling us the truth." And that second question — about truthful responding — is its own can of worms, especially on a topic as emotionally charged as whether your country won or lost a war. There's social desirability bias, where respondents tell pollsters what they think they should say rather than what they actually believe.
How would that play out in this specific context? If I'm an Israeli who voted for Netanyahu and I'm being asked whether Iran won the war, what's the social pressure?
It cuts in fascinating directions. On one hand, admitting Iran won might feel like admitting your side failed — that's socially costly. So social desirability bias would push people toward saying Israel won, or at least that nobody won. That would mean the ninety-two percent figure might actually understate the true level of disillusionment. On the other hand, there's a countervailing pressure in the current Israeli political climate. If the dominant social norm in your community has shifted toward criticizing the government's handling of the war, then admitting you still think Israel won might be the socially costly response. You'd be going against the grain of your peer group.
The bias could run in either direction, and you don't know which without external benchmarks.
And there's another layer — differential non-response. People who feel strongly about the government's failure might be more motivated to participate in a political poll than those who are satisfied but less engaged. If you're angry, you want to be heard. If you're content, you might just hang up and go back to dinner.
Or the reverse — people who are deeply disillusioned might be so checked out that they refuse to answer polls entirely, skewing the sample toward the more engaged.
Right, and you don't know which effect dominates unless you have external benchmarks. That's why I always tell people to look for multiple polls from different firms asking similar questions around the same time. If three different pollsters using three different methodologies all land in roughly the same ballpark, you can be more confident the signal is real. If one poll shows ninety-two percent and another shows sixty percent, someone's methodology is broken.
What about poll mode — phones, online, face-to-face?
Telephone polling used to be the gold standard. It still can be for some questions — random-digit dial gives almost every known-living geographic resident a negligible pathway inside; nationwide totals theoretically just picking a stranger randomly would ideally give similar spread stats each successive pull, holding general variables matched outside blips from minor weekend momentary single-game events. Say a large soccer final dominates responses — it spikes one night and ebbs the next morning. Random dial theoretically smooths that out.
Only theoretically — because reality attacks theory hard. Nobody responds anymore. The proportion is huge — dial nine figures, get minimal answers. Currently the sample skews extremely toward idling pension-age individuals with home landlines, some remnant wires they still pick up. The vast missing younger segments can't be repaired by simple weighting because the base cell numbers weren't included in sufficient coverage to begin with. You need dual-frame designs nowadays — landline plus mobile — and even that's getting harder as people screen calls from unknown numbers.
Online panels offer convenience and can recapture higher yields across varied demographics more easily. But they come with their own baggage. People join panels for the incentives — a few shekels per survey — and that attracts a particular type of respondent. You get professional survey-takers who blast through questionnaires at speed, or people whose financial situation makes micro-payments meaningful in ways that distinguish them from the general population. Unless the panel provider does massive pre-stratification — ensuring known compositional cell limits across multiple cross-ratios, multi-layered correcting throughout regional, education, age, ethnicity, gender, economic, and last-ballot cross-sets with rigorous tuning post random intakes — the final numbers can be distorted.
What do we know about mode for this specific Israeli poll?
Many mainstream Israeli outlets like Maariv typically use live operator phone conversations — harder, more expensive, but often higher quality. Others have trialed mixed modes. But the broader worldwide trend is that smaller quality live sets are being replaced by faster quant online panels that can fill quotas in days rather than weeks. It's quicker, it's cheaper, and it lets news organizations spike on relevant agenda items while the story is hot. If this IDI poll used an online panel and the recruitment source isn't transparent, my suspicion level rises. Unknown opt-in panel sources carry real potential for differential bias.
Transparency about mode isn't a nice-to-have. It's essential for credibility.
It's foundational. And here's something else that rarely gets discussed — what questions weren't asked. Glossing over home-front exclusion means an entire theme is omitted. If the poll didn't ask about certain dimensions of the conflict — say, the domestic economic cost, or views on the hostage negotiations, or attitudes toward specific military tactics — that absence shapes what conclusions readers can draw. Someone deliberately limited the dimensions, and that matters almost as much as the data they did collect.
Can you unpack that with a specific hypothetical? What's an example of a missing question that would change how we interpret the ninety-two percent?
Imagine the poll asked "Did Iran emerge victorious from the war?" and ninety-two percent said yes. But what if they'd also asked "Did Israel achieve its military objectives in Gaza?" — and eighty percent said yes to that. Suddenly the picture gets more nuanced. People might be saying Iran won the broader strategic and reputational game while simultaneously believing the IDF accomplished its tactical goals. That's a much more interesting story than "Israelis think they lost." Or what if they'd asked about the economic cost — "Has the war's economic impact on your household been severe, moderate, or minimal?" If the people saying Iran won are predominantly those who felt the economic pain most acutely, that tells you something about what's driving the sentiment. Without those additional dimensions, "Iran won" is a blunt instrument that might be standing in for several different things.
The questions that aren't asked define the boundaries of what we're allowed to conclude.
Asking designed-purpose questions while ignoring essential context says nearly as much as the truth readings literally printed. The omitted data is often more revealing than what's included. A careful reader should always ask: what didn't they ask, and why might that be?
That's a discipline most people don't bring to a headline. They see "ninety-two percent" and the number feels solid, objective, unarguable.
That's the trap. Numbers feel objective. But every number in a poll is the product of dozens of human decisions — who to call, how to weight, what to ask, how to phrase the question, what order to put the questions in. Change any one of those decisions and the number moves.
Let's linger on question wording for a moment, because I think that's one of the most underappreciated levers. How much can phrasing shift a result?
There's a classic example from U.polling on abortion. When Gallup asks whether abortion should be "legal under any circumstances," you get one number. When they ask whether abortion should be "legal under some circumstances," you get a much higher number. When they ask whether the government should "intervene in personal medical decisions," you get something else entirely. Same underlying issue, different frames, wildly different results. In the Israeli context, imagine the difference between asking "Did Iran win the war?" versus "Did Israel achieve a decisive victory?" versus "Looking at the regional balance of power after the conflict, which country is in a stronger position?" Those are all probing roughly the same sentiment, but the framing could easily produce ten-point swings.
We don't know the exact wording from the published summary.
We know the questions were posed, but not necessarily the precise phrasing or the order. Question order matters enormously. If you ask "Do you approve of the Prime Minister's handling of the war?" right after a series of questions about casualties and economic costs, you'll get a different answer than if you ask it after questions about successful military operations. The preceding questions prime certain considerations. A well-designed poll randomizes question order or at least acknowledges the potential for order effects. An opaque poll leaves you guessing.
Which brings us to a practical framework. If Daniel — or anyone listening — wants to evaluate a poll they encounter in the wild, what's the checklist?
I'd say three rules. First, check the sample size and the margin of error, and remember that subgroups have wider error bars. Second, look for transparency on methodology — weighting variables, poll mode, response rate, question wording. If those aren't published, mentally downgrade your confidence. Third, never trust a single poll in isolation. Look for corroboration. Channel Twelve ran a separate survey around the same period that found roughly sixty-eight percent rated the war outcome poorly — not identical to ninety-two percent, but directionally aligned. When multiple independent polls using different methods converge on a similar story, you can start to believe the pattern is real.
The ninety-two percent number passes the basic math test, raises some yellow flags on methodological transparency, and gets partial corroboration from other surveys. That doesn't mean it's wrong — it means we hold it with appropriate uncertainty.
Appropriate uncertainty is the goal. Not cynicism, not blind trust. Just a clear-eyed understanding that every poll is a model of reality, not reality itself. And models have error. I think of a poll like a weather forecast. A good forecast tells you there's an eighty percent chance of rain, and you bring an umbrella. You don't curse the meteorologist if it stays dry — you understand the forecast was probabilistic and you made a reasonable choice given the information available. Polls should be treated the same way. They're probabilistic estimates, not crystal balls.
Just like weather forecasts, they've gotten better over time but they'll never be perfect. The atmosphere is chaotic, and so is human opinion.
Right — and we should acknowledge that polling methodology has genuinely improved in many ways. The response rate crisis is real, but the best pollsters have developed sophisticated corrections. Multi-mode designs, better weighting algorithms, more careful attention to hard-to-reach populations. The problem isn't that polling is worthless — it's that polling quality varies enormously, and the consumer needs tools to tell the difference. The ninety-two percent finding from IDI might be methodologically sound. It might be slightly off. It might be capturing something real but incomplete. We don't know enough to say with confidence, and that's the point.
That's a good place to land. The broader lesson here isn't about Israel or Iran or Netanyahu — it's about building a mental habit. When you see a poll, you run through the checks. Sample size, subgroup precision, methodological transparency, external corroboration. Do that for a few months and it becomes automatic.
The world needs more of that automatic skepticism. Not the performative kind where people dismiss any data they don't like — but the genuine kind where you engage with the evidence seriously enough to understand its limits. That's the difference between being informed and just having opinions. An informed person can tell you what the evidence says and how confident we should be in it. A person with opinions just tells you what they think and treats any number that agrees with them as proof.
There's a humility in that approach that I think is rare. Saying "the data suggests X, but I'm holding it with a thirty percent uncertainty discount because the methodology isn't fully transparent" — that's a more honest sentence than most of what passes for political commentary.
It's also a harder sentence. It doesn't fit in a tweet. It doesn't make for good television. But it's what actual understanding sounds like. And the fact that Daniel wrote in asking about the mechanics rather than the headline number tells me he's already on that path.
Before we go — Hilbert has something for us.
Now: Hilbert's daily fun fact.
Hilbert: In the 1810s, a prevailing scientific theory held that certain cave-adapted crustaceans of the Outer Hebrides had evolved their blindness because their ancestors crawled repeatedly into tight spaces to escape persistent hazy low morning fog.
Leaving that one entirely untouched.
I have so many questions and I'm choosing to ask none of them. Though I will say, the image of crustaceans repeatedly bonking their heads into cave walls until they evolve blindness as a coping mechanism is going to stay with me.
I think that's the intended effect. And in some strange way, it's a fitting coda — a reminder that prevailing theories can be completely, spectacularly wrong, and the only defense is asking better questions. This has been My Weird Prompts — submitted typically via myweirdprompts.com, because brilliant nonsense awaits an audience, gently possible, warm embrace eventually maybe hopeful.
Taking supportive appreciative leave. Sincerely, future near, brighter odd truly. Friends, stay well.
