Mistaking the mistake

                   Would you buy a used car from one of these men?  Photo source: Business Insider

The Times’s coverage of political surveys is improving. At least it seems to recognize what a margin of error is. If anything, it may recognize it too much.

The story cited below is an example. Senator Tim Scott of South Carolina has a “favorability rating among Iowa Republicans – 70 percent – [that] is on par with Mr. [Donald] Trump’s 72% and just behind Mr. [Ron] DeSantis’s 77 percent.”

This is not ridiculously wrong, but it’s still wrong. In fact, Ron DeSantis, the governor of Florida, probably has a good lead on Scott. The reporter’s gaffe is to treat the survey’s margin of error, which measures the potential errors in the conclusions, as if it were part of the estimates that led to the conclusions. Wait, I’ll explain.

The story is about a survey of Iowans who are likely to attend the Republican caucuses for President in January. We can’t survey all the likely caucus-goers, if only because we don’t even know who they are yet. So we must take a sample – of 432 respondents, in this case. But no sample is a perfect mirror of the population from which it is drawn. (In this case, the population is all likely caucus-goers in Iowa). So our conclusions from the sample – for example, about the size of Trump’s lead over DeSantis – won’t be exactly correct. The question is whether our error is likely to be large enough to matter.

The size of our error depends on the dispersion in the population. Suppose, for example, that all caucus-goers are virtually the same.  Then any sample of them is probably pretty good. On the other hand, if they are over the map in voting preferences, then we may easily draw an extreme sample. 

So: How can we measure the dispersion in the population? Unfortunately, since we can’t observe the exact population, we can’t directly measure its dispersion. But we can measure the dispersion in the sample. So we can base on it our estimate of the dispersion in the population. If the sample is highly dispersed, then the population probably is, too, and chances are good that any sample will mislead us. 

The “margin of error” measures the dispersion in the sample. When this value is large, we should be cautious about drawing conclusions from the sample.

In the survey at hand, the margin of error is 5.9%. Suppose that 50% of respondents like Smith and 49% like Wesson.  Then we cannot conclude that more Iowans like Smith than Wesson, because 1% is within the margin of error. On the other hand, if 55% like Smith and 45% Wesson, then we can reasonably conclude that more Iowans prefer Smith, since 10% is outside the margin of error. Statisticians usually pick the margin of error that ensures that chances of wrong conclusions from the sample are less than 5%.

In the survey, DeSantis’s favorability rating is 77% and Scott’s is 70%. This is a 7% difference, which is outside the margin of error. So, yes, Iowans probably view DeSantis more favorably than Scott. But 7% is close to 5.9%, the margin of error. Does this mean that DeSantis has only a bare advantage over Scott?

No!

The margin of error measures the confidence that we can put in a conclusion. It does not enter the conclusion directly. Our best estimate of DeSantis’s edge is 7%, not 1.1%.

A silly example may clear this up. Suppose that I would like to estimate the unemployment rate in New York City. I stand on 8^th Avenue and interview one person. He says he’s unemployed. I write it down and go home to my chicken noodles. End of survey.

What’s my best estimate of the unemployment rate? One hundred percent, because everyone in my sample (one person) was unemployed.  Can I put any confidence in this conclusion? Of course not. I need to survey more people.

The same principles apply to The Times survey. Probably Iowans prefer DeSantis, although it’s a close call. But the fact that it is a close call does not mean that DeSantis is barely leading Scott. It just means that we can’t be too certain of our conclusion, whatever our conclusion happens to be. Here, our conclusion is that DeSantis is ahead by a country mile. Well, maybe a half-mile. –Leon Taylor, Baltimore, tayloralmaty@gmail.com

Reference

Jonathan Weisman. Trump leads G.O.P. in Iowa, but his hold is less dominant.  The New York Times. August 4, 2023. Trump’s Lead in Iowa Is Less Dominant, Poll Shows - The New York Times (nytimes.com)