We are in the midst of a massive #MeToo scandal surrounding Brett Kavanaugh with recent polls indicating that a majority of Americans do not want to see him confirmed. We are watching with tense breath to see if Rod Rosenstein still has a job by the end of the week in the wake of a massively irresponsible piece published by the New York Times. It is rumored that the Democrats could be looking at a blue wave that gives them control of the House of Representatives- though this is not to be taken for granted so go register to vote, make sure your registration is correct, and schedule yourself to vote on Election Day!! And, of course, the leader of the Republican Party is underwater in his own favorability.
And yet, in the midst of all this, numerous headlines are popping up declaring that, according to a recent Gallup poll, the Republican Party is, somehow, enjoying a 7-year high of favorability.
If 2016 proved anything, it’s that polls- and especially the reporting surrounding those polls- require more study and context than simple eye-catching percentage numbers. Unfortunately, too many reporters don’t take the time to do that, and too many readers simply check the headline and don’t read any further.
So let’s dig into this a little bit by looking at the data Gallop themselves provide, and follow along with this amateur as I identify my biggest issues with this claim.
Issue #1: The survey was conducted from September 4th to September 12th, 2018.
The very first sentence of Gallop’s data sheet gives us this very important point. A quick Google search revealed that The Washington Post (paywall), CNN, The Hill, and Gallop’s own press release all failed to mention this, while the conservative National Review is the only one that actually pointed it out and added context! So kudos to Mairead McArdle for having the best write-up of the bunch!
The timing of the survey has always been important, and often ignored, but it’s even more important in this 24-hour hyper-fueled political disaster that is the 45th administration. Can anybody even remember what the big stories were from September 4th to September 12th for that matter? Hold on, let me jump into my TARDIS… okay, there was Hurricane Florence, Kavanaugh’s possession of Leahy’s stolen emails, oh yeah that anonymous NYTimes op-ed about the administrative not-a-coup and the Bob Woodward book was coming out (wow, remember those?), the number of migrant children in detention had risen to over 12,000…
Anyway, the point is a lot of seemingly ground-shifting stuff happens on a daily basis nowadays, and a survey taken a measly three weeks ago doesn’t tell us much about today, and it’s certainly not going to tell us much about six weeks from now. Have you registered to vote yet?
Issue #2: The survey was conducted with a random sample of 1,035 adults and a 4-point margin of error.
This is an issue with all polls that purport to reveal the national mood. The United States of America has over 328 million people living here. It is impossible to get in touch with all of them to get a truly accurate picture, so pollsters have to rely on samples, and perform some massive extrapolation from there. In this particular poll, approximately 465 people having a favorable view of the Republican Party is 45% of the 1,035-respondent sample, and, after weighting, this gets extrapolated to 45% of America, or 147 million Americans. 465 people representing 147 million people.
Now, honest pollsters are aware of this weakness and they try to account for it, hence the industry’s emphasis on random, weighted, and large samples. To massively oversimplify it, randomness ensures you’re not cherry-picking to get the result you want; weighting uses a lot of math, proportions, and trends to try to make sure the result is an accurate representation; and large samples ensure that the data source is broad enough to be accurate. The greater number of people in your sample, the more confident you can be in the results. (This dynamic applies to all fields of research that require human interaction. This is why less-reputable studies, like ones that ridiculously claim chocolate helps you lose weight, tend to have a very low sample size. Always check the sample.)
However, regardless of how much effort pollsters put in making sure the sample is representative of the whole, they can never guarantee full accuracy, and this is where the Margin of Error comes in. The Margin of Error is an admission by the pollsters that they could be wrong, may be wrong, and if so, here’s the range of how wrong they could be. In standard practice by most reputable pollsters, the Margin of Error is usually four or five points in either direction.
In this particular poll, Gallup declares a margin of error of plus or minus four percentage points. What that means is, based on the data they accumulated in their survey, an accurate representation of the Republican Party’s nationwide favorability could be anywhere from 41-49%.
Forty-one to forty-nine percent.
See, the real-world limitations of the sample means Gallup (and every other pollster under the sun) cannot actually provide an accurate representation. Thus, the number they do give- in this case, 45%- is what they’ve deemed, based on their math and trends and expertise, to be the most likely to be accurate, not most accurate. Anyone could take Gallup’s data, apply different formulas and filters to it, and come out with a different number, but assuming integrity and honesty, that number would fall within the range of 41-49.
Ever since I took a statistics class in college and first learned about the Margin of Error, I have been skeptical of pollsters, and particularly the news releases surrounding them, that loudly declare in full confidence a certain percentage. I’ve become downright cynical in the aftermath of 2016. As a result, whenever I see a poll-based headline declaring a specific percentage (so, in other words, all of the poll-based headlines), I’ve started mentally applying the Margin of Error, and creating a range, rather than a dedicated point.
Thus, according to CNN, Donald Trump has an approval rating range of 32% to 40% (based on CNN claiming a Margin of Error of 3.8 in their data sheet).
According to the Gallup poll we’ve been discussing, the Democratic Party has an approval rating range of 40% to 48%, and an unfavorable range of 48% to 56%.
In that same poll, the Republican Party has a favorable rating range of 41% to 49%, and an unfavorable rating range of 48% to 56%. They match the Democratic Party in the “we don’t like you” department, but sure, Republicans, brag about your underwater favorable number.
Ricky of Kokiri 