The games newspeople play

 

                                            A winner or just a statistic? Photo credit: Britannica

Is Kamala Harris winning? The Washington Post and The New York Times would love to tell you. What they won't tell you is that they are either hopelessly confused or lying. 

Let's start with today's Post. "Vice President Kamala Harris holds a narrow lead over former President Donald Trump in the presidential election, a notable improvement for Democrats in a contest that a little more than a month ago showed President Joe Biden and Trump in a dead heat, according to a Washington Post-ABC News-Ipsos poll....Given the margin of error in this poll, which tests only national support, Harris's lead among registered voters is not considered statistically significant."

Big news, sports fans! Harris is winning! Our poll says so! But...wait a minute...it's not statistically significant, which means...um...hmm.

Friends and neighbors, you can't have it both ways. Either Harris is winning, or she isn't. The rule of thumb is this: If the poll margin is within the margin of error, you cannot deduce with 95% confidence that either Harris or Trump is winning. The race is a dead heat.

But.

Digging into the story, we learn from a graph that the poll has a margin of error of plus or minus 2.5%. Harris's lead in the poll is 4%, or 49% to 45%. So Harris is winning. The poll margin, 4%, is larger than the margin of error, 2.5%. It is statistically significant: In other words, the result is very likely to hold as well outside of the sample, for the country in general. The Post is breathtakingly mucked up.

Its confusion probably arises from its misunderstanding of the plus-or-minus designation. The idea is this: We want to test the hypothesis that the race is a tie. That would happen if Harris is neither winning nor losing. Our usual criterion is that we will accept that the race is a dead heat unless we are 95% confident that Harris is either winning or losing. Well, if Harris's lead exceeds 2.5%, it is not a dead heat. And had Trump's lead exceeded 2.5%, that is, had Harris's lead been -2.5%, it would not have been a dead heat. It is not the case that Harris's margin must exceed 2 times 2.5% for us to conclude that the poll result is statistically significant, that is, that the race is not a dead heat.

The further conclusion, that Harris is winning, is easy to confirm once we see that the race cannot be a tie. The intuition is this: If Harris is so far in the lead that we can reject the possibility that the race is a tie, then we can also reject the possibility that Trump is winning.

In this case, The Post lucked out. Its headline correctly said the poll indicated that Harris was winning. But there is a more important point: The Post doesn't know what the hell it is doing.

 The error about errors

 Newspapers goof when reporting polls because they do not understand how polling errors occur. When you ask someone in newspapers why they don't take the margin of error seriously, you will get an answer like this: Well, the pollsters told us they were careful about polling. So we didn't worry about potential errors. They had to be small. We reported the margin of error only because everyone says we should.  

Grrr.  Now hear this: The margin of error is calculated under the assumption that the polling was perfect. Even when the polling sample is an accurate mirror of likely voters, the outcome of the poll may not be accurate. There is still a good chance that too many Harris supporters were interviewed. There is also a good chance that too many Trump supporters were interviewed.

A simple example will show what I mean. Suppose that we have a class of 100 students: half receive As, and half receive Bs. (Welcome to grade inflation.) We take a sample of 10 students. Even if the sampling is utterly fair, there is still a chance that at least 6 of the 10 students sampled received As. Based on the sample, we wrongly conclude that the majority of students received As. In reality, only half did.

Because of such possibilities, statisticians test the idea that of all likely voters, half favor Trump and half favor Harris. We can dismiss this hypothesis of a dead heat if a large-enough share of the sample favors either Harris or Trump. The "margin of error" reflects how large the share of respondents backing one candidate must be if we are to dismiss the possibility of a dead heat, assuming perfect polling. If the pollsters made mistakes, and they usually do, the actual margin of error is even larger than the one usually reported.

Moving right along…

 For years, The New York Times has reported poll survey results as if they were true for likely voters in general. Yesterday's story is an example. Yes, we can conclude that Harris is leading in Arizona, because her margin (barely) exceeds the margin of error (see the figure below). But contrary to The Times, we cannot conclude with 95% confidence that either Harris or Trump is leading in North Carolina, Nevada, or Georgia. Those survey margins are within the margins of error, which are shown in the graph in the article. By the usual definitions in political polling, those races are dead heats.

Remarks later in the story indicate what The Times is really up to:  "As the Harris and Trump campaigns rush to define each other in the remade race, voters see a choice between strength and compassion. Voters were about equally likely to see each candidate as qualified and change-makers, but significantly more voters viewed Mr. Trump as a strong leader....When voters were asked who 'cares about people like me,' Ms. Harris had a slight edge over Mr. Trump: 52 percent compared to 48 percent."

The Times seems to think that it can report results from a sample as if they held for the population, as long as it labels results that hold with 95% confidence as "significant." (The poll margin for the question about strong leaders was 8%, much larger than the margin of error.) The Old Grey Lady is playing word games. Most readers are not statisticians. When The Times says Harris has "a slight edge," they think that The Times is talking about the population, that is, the world outside of the sample. They view the word "significantly" as redundant. 

So The Times has it both ways. It tricks readers into believing that it is breaking news by declaring a winner. And, if challenged, it can always point out that, after all, it did refer to significance, sort of.

A simple example will show why this is a cardinal sin. Suppose that the president of General Motors misstated stock earnings for years. The Securities and Exchange Commission would investigate and probably force the president to resign. He might even face charges of fraud. Shouldn't we hold the nation's leading newspapers to standards at least as high? In a close race like this, newspaper reports of the polls affect donors. The money is sloshing towards Harris because of her perceived momentum. How real is that perception?

Another common dodge of newspapers: Well, do we really need 95% confidence that Candidate X is winning? Surely 90% is good enough. In that case, the margin of error is smaller, and the poll margin may now exceed it.  We can then conclude that X is winning.

It is true that 95% is an arbitrary standard. But it is also a universal one, especially in political polling. To apply 95% in some cases and (quietly) 90% to others, which The Post is fond of doing, amounts to moving the goalposts at halftime for some games but not for others. It makes it hard to compare poll results. Is Harris ahead in certain polls because she is winning the race, or because those polls adjusted the margin of error until she was "winning"?       

The Times and The Post say their polling staffs have graduate training in statistics. I do not see how a well-trained statistician can unknowingly make errors this basic. I will leave it at that.  

Noncardinal sins....

There's more, much more. The Times writes: "The polls show some risk for Ms. Harris as she rallies Democrats to her cause, including that more registered voters view her as too liberal (43 percent) than those who say Mr. Trump is too conservative (33 percent)." 

It is not clear why The Times says this shows "some risk" for Harris. Perhaps it means that voters are significantly more likely to view Harris as liberal than to view Trump as conservative. If that's what it means, it should provide evidence. The margin of error applies here, too. But even if it is true that voters are more likely to judge Harris as liberal than to judge Trump as conservative, why would this be a risk for Harris?

The Times continues, "For now, [Harris] is edging ahead of [Trump] among critical independent voters." Evidence?

The Times also writes, "Mr. Trump and Ms. Harris are tied at 48 percent across an average of the four Sun Belt states in surveys conducted Aug. 8 to 15. That marks a significant improvement for Democrats compared with May, when Mr. Trump led Mr. Biden 50 percent to 41 percent across Arizona, Georgia, and Nevada in the previous set of Times/Siena Sun Belt polls, which did not include North Carolina." 

The Times is comparing apples to oranges. One sample includes North Carolina, the other doesn't. And it is not clear why it bothers to make this top-level comparison. The polls differ from state to state, and The Times has specific information about each. Why not stick to reporting the results for each state poll rather than aggregate them into an inferior measure? 

If The Times simply must have a nation-level conclusion, it can view each state poll as an observation in a sample of all state polls and compute the probability that Harris is leading nationwide, given the sample of state polls. Do the same for May.

Finally, a puzzle. In Arizona, The Times/Siena poll reports Harris ahead in the sample by 5% among likely voters and 4% among registered voters. But the poll by the Competitiveness Coalition/Public Opinion Strategies about two weeks earlier reported Trump ahead by 5% among likely voters. And The Hill/Emerson College poll about three weeks before The Times/Siena poll reported Trump ahead among registered voters by 5%. Were these differences due to Harris's momentum? Or were they due to differences in how the polls were conducted?

 

                                           Source: The New York Times

 The key question: Are The Washington Post and The New York Times statistically significant? 

Anyway, these conundrums pertain to Central Asia because the political polling in the region, or at least the reporting of it, is anything but perfect. The margin of error is correspondingly higher.  In short, politics pervade political statistics. --Leon Taylor, Seymour, Indiana, tayloralmaty@gmail.com 

 

Notes             

For helpful comments, I thank but do not implicate Annabel Benson, Richard Green, and Mark Kennet. Parts of this post draw upon my earlier articles on my Facebook page.

 

References

Dan Balz, Scott Clement, and Emily Guskin.  Kamala Harris holds slight national lead over Donald Trump, Post-ABC-Ipsos poll finds - The Washington Post August 18, 2024.

Shane Goldmacher and Ruth Igielnik.  Kamala Harris Puts Four Sun Belt States Back in Play, Times/Siena Polls Find - The New York Times (nytimes.com)  August 17, 2024.

Lisa Lerer and Ruth Igielnik.  Harris Leads Trump in Three Key States, Times/Siena Polls Find - The New York Times (nytimes.com)  August 10, 2024.