When will they learn?

Newsbreak from The New York Times!  The Republicans are edging out the Democrats in next month’s Congressional elections!

The GOP has a “narrow but distinctive advantage” according to a poll from the Times and Siena College, confides Shane Goldmacher.  “The poll shows that 49 percent of likely voters said they planned to vote for a Republican to represent them in Congress on Nov. 8, compared with 45 percent who planned to vote for a Democrat.”

Next paragraph, a parenthetical comment: The poll’s “unrounded margin is closer to three points, not the four points that the rounded figures imply.”

And at the end of the story, in fine print: “The margin of sampling error is plus or minus 4.1 percentage points.” 

In other words, the race may well be a dead heat.

Why can’t the leading newspaper in the English language get a simple statistic straight?  A lead in a poll is not necessarily a lead in the population. Stuff happens.  Polltakers write down the wrong answers, people change their minds in mid-response.  The sampling error estimates the share of the poll responses that may mislead us.  In this case, the error in either direction is as at least as large as the “distinctive” advantage of Republicans over Democrats.

Landslide Donald

More technically, we cannot be 95% confident that the reported edge of the GOP exists outside of the sample.  To understand this, suppose instead that the edge had exceeded the sampling error.  Then, had we rerun the poll 100 times, we would have found that the Republicans led the Democrats at least 95 times. That would have been convincing evidence that the GOP was winning. Of course, we cannot actually rerun the poll a hundred times, so we rely on statistical theory to infer whether a 4% sampling lead is credible. Here, it’s not.

We could argue about whether a confidence interval of 95%, which is the usual one for political polls, is needlessly stringent. This standard means that we would not conclude that the Republicans were winning unless chances that this was wrong were less than 5%.  Is such rigor necessary?  What would be wrong with a 15% chance of an error in a recreational sport like political races?  How costly would such an error truly be for society?  If we allow for a 15% chance of a mistake (that is, we would use an 85% confidence interval), then we would be willing to live with a sampling error much larger than 4% and thus, in this case, with the conclusion that the GOP is winning.  

I grant these subtleties. What worries me is that the Times can't get even the obvious stuff right.  Goldmacher writes, twice, that “in a hypothetical 2024 rematch, Mr. Trump led Mr. Biden in the poll by one percentage point.” I don’t know of any national political poll with a margin of error below 1%. Anyway, it is common sense that a poll reporting a 1% lead connotes a race that is too close to call.  The Times should point that out to readers who know zilch about statistics.

If it helps, you can think about the problem generally. We want to know about a group of people; call it the "statistical population." For the Times, the population consists of all likely voters in the Congressional elections next month.  We can't canvass everybody, so we take a "sample" -- the Times's poll. No sample perfectly reflects the statistical population.  The problem in statistical inference is to decide whether you trust the sample enough to draw conclusions from it about the population anyway. In the poll, the Republicans surge ahead of the Democrats; can we conclude that this holds for likely voters in general?  The answer depends on how large of a probability of error you are willing to accept to use the sample to characterize the population. No, it doesn't do any good to say, "I will accept no chance of an error," because there is always a chance that the sample will mislead you. The usual maximum error that people are willing to accept is 5%, but there is no reason why that has to be the magic number.  In any event, the Times's mistake is to assume that all samples are perfect.    

The moral of the story, as usual: When the papers report a poll, flip down to the end of the story, and click on the link to the actual survey. Because you can be 95% confident that the news media, including in Central Asia, will screw it up.  --Leon Taylor, Baltimore tayloralmaty@gmail.com

 

Reference

Shane Goldmacher.  2022.  Republicans gain edge as voters worry about economy, Times/Siena poll finds.  The New York Times.  October 17.