If state poll results show ties between the Trump-Harris race, is it due to voters or pollsters?

Recent polls in the seven core states show a surprisingly tight presidential race, with 124 of the last 321 polls conducted in those states – nearly 39% – showing margins of 1 percentage point or less.

In fact, the state polls show not only an astonishingly tight race, but an impossibly tight race. Even in truly equal elections, the randomness inherent in polls would produce more varied and less clustered results—unless the state polls and polling averages are artificially close because of the decisions pollsters make.

The results of a poll depend on the opinions of voters and the decisions of pollsters. Decisions about how to weight the polls so that they reflect the expected composition of the electorate can move the results of a poll up to 8 points. This is true even if pollsters make perfectly reasonable decisions about how to weight their survey data, as survey researchers are forced to consider new methods and ideas for weighting and addressing the declining response rates following the poll misses of 2016 and 2020.

But the fact that so many polls are reporting the exact same ranges and results raises a troubling possibility: that some pollsters are making adjustments in such similar ways that those choices cause the results to coincide, creating a potential illusion of certainty – or that some pollsters even look to the results of others as a guide to their own results (i.e., “herding”).

If so, the artificial similarity of the polls may create a false impression that may not be realized on Election Day. We could very well be in for a very exciting election. But there is also a significant chance that one candidate or the other can sweep every swing state and win the presidency somewhat comfortably, at least compared to the balanced picture in the polls.

What should we see in a perfect election world due to randomness?

In a world perfect for polls—a researcher’s paradise in which every voter can be contacted and every contacted voter responds—we can use math to calculate how much variation there should be, due to the fact that voters are randomly selected to participate in a poll. to participate in an opinion poll.

If a race in this world were truly 50%-50% tied, the polls wouldn’t all produce results that were split 50%-50%. Imagine if pollsters in this world conducted 100 identical surveys of 863 randomly selected voters (that’s the average sample size of this year’s swing state polls). The results of 95 of those polls show the candidates receiving somewhere between 46.7% and 53.3% support – even though in this imaginary world we know the race is actually tied at 50%. The other five polls would show the candidates earning something even greater or less than that range.

This variation is known as the “margin of error” in a poll – i.e. how much randomly selected voters who always respond can influence a poll’s estimate for a candidate.

Because each candidate’s support varies randomly, these polls predict a margin in an even race that ranges from -6.6 to +6.6 for 95 of the 100 polls (with even wider margins for the other five).

It’s important to emphasize that the margins we can expect in a tie (and in a perfect polling world) are much larger than the margins in the swing states in 2020. Even in ideal conditions for pollsit is difficult, if not impossible, for a poll to be very informative about who is in the lead in a tight race. And this is perhaps a lower bound on what we should observe in the messier real world, where polls vary in the way respondents are selected, contacted, and weighted to match the voters pollsters predict will turn out in 2024.

We can also calculate what proportion of 863-person polls we might expect to see different margins in a true tie. Rounded to the nearest percentage point, about 11% of polls should show a tie.

That means that almost 9 out of 10 polls of a tie should not actually produce a tie result, due to randomness and the margin of error.

About 32% of polls should have a margin of 1 point or closer, 55% should have a margin of 2 points or closer, and 69% should have a margin of 3 points or closer. Even in a 50-50 race, roughly 10% of polls should have a margin of more than 5 points due to inherent randomness – almost the same percentage showing a (rounded) tie!

With enough polls, the predicted margin should also resemble a normal bell curve distribution – with a similar number of polls showing both candidates in the lead.

What do we see in the swing state polls?

Actual swing state polls show far less variation than the benchmarks we would expect in a perfect election world. Of the 321 polls in the seven swing states, only 9 polls (3%) report a margin of more than 5 points. Even if every race were equal – which it isn’t – we would still expect about 32 out of 321 polls with a margin of more than 5 points due to randomness.

Visualizing how the reported polling margins compare to what we would expect in a perfect polling world strongly suggests that the swing state polling margins are around the statewide polling averages. In these 321 state polls, 69 of them (21%) report an exact tie and 124 polls (39%) report a margin of 1 percentage point or less. Both figures are roughly double what we would expect in a perfect polling world where the only source of variation is the random selection of voters who respond.

Pennsylvania may be the most troubling state. More than 20 of the 59 polls there (34%) show an exact tie and 26 (44%) show a margin of 1 point or less. And even though there is a 15% chance that an actual tie could yield a poll with more than With a 5-point margin due to randomness, we only see 2 of 59 polls in Pennsylvania (3.3%) with a margin greater then 5 points.

Even where election results are not as tightly clustered, such as in Arizona, Michigan, and Wisconsin, there are still many more polls than we would expect around the polling average and too few polls by large margins.

What’s going on?

The concentrated margins we see in swing state polls likely reflect one of two possibilities.

One possibility is that pollsters sometimes adjust a poll result that looks “weird” by choosing a weighting scheme that produces results that are closer to the results of other polls. There appear to be strong incentives for risk-averse pollsters to do this. Unless a pollster conducts many polls and can be confident that the impact of randomness is moderate, they may fear reputational and financial costs if they get the wrong result due to randomness, since pollsters are judged on accuracy of their surveys.

A risk-averse pollster who is given a five-point margin in a race he thinks is tied may choose to “adjust” the results to something closer to what other polls show, to avoid that their outlier negatively affects their reputation relative to competitors.

Another, more likely possibility is that some of the tools pollsters are using in 2024 to address 2020’s polling problems, such as weighting by partisanship, past voting or other factors, could smooth out the gaps and reduce the variation in can reduce reported polls. The effect of such decisions is subtle, but important, because it means that the similarity of polls is determined by the decisions of pollsters, not voters.

And if these assumptions are wrong, something that won’t be known until after the election, the risk of a potentially significant election error increases as the variation in different polls decreases.

Why this matters

The fact that so many swing state polls report similarly close margins is a problem because it raises questions about whether the polls are tied in these races because of voters or pollsters. Will 2024 be as close as 2020 because our politics are stable, or will the 2024 polls only resemble 2020 results because of the decisions state pollsters make? The fact that the polls appear to be closer together than we would expect in a perfect election world raises serious questions about the second scenario.

The reported polls and polling averages are creating a consensus that the race will be very tight and that we will likely see an outcome similar to 2020. Perhaps that is true. It would be great if the 2024 polls could successfully address the concerns of 2016 and 2020.

But the fact that the polls all report such similar margins doesn’t necessarily make it more likely that those margins represent the final outcome. In fact, it raises the possibility that the outcome of the election could be unexpectedly different from the razor-sharp story that the cluster of state polls and the polling averages suggest.