Everyday Einstein tackles a topic everyone is thinking about this fall: Can you really trust those election polls? How do you design a reliable poll? How many people do you need to question to predict the results of a national election?
This method of weighting, however, is controversial and can be used to push the results toward a particular outcome if not done carefully. If weighted too heavily, the opinions of just a few respondents could play a disproportionately important roll in the poll’s outcome when they may not actually represent the majority beliefs of the demographics they were weighted to represent.
Others disagree on what demographics should be weighted. Weighting by gender or race can make sense, although of course all women or all Latino people do not vote as a monolith, but what about political affiliation? Weighting by political party could cause the derived poll results to be misrepresentative if, in a particular election, voters are crossing party lines more than usual.
Sometimes even the most carefully chosen methods cannot account for biases in respondents.
However, sometimes even the most carefully chosen methods cannot account for biases in respondents. For example, after the first debate between Obama and Romney before the 2012 election, polls shifted to favor Romney, a shift that in part reflected the fact that Democrats weren’t interested in providing responses after what was perceived as a huge loss for Obama in the debate. Republicans, feeling victorious, were naturally feeling more generous with their time.
Another important factor in designing a reliable polling experiment is methodological error. For example, are the questions clear? Even a sampling of the most accurately representative subset of the population will not do any good if particular questions can be easily misinterpreted. Pollsters will also often ask questions to help predict whether the respondent will actually vote, questions like “how interested are you in the election?” or “did you vote in the last election?”
When considering which polls to trust, one should also look to their margin of error which represents the statistical ability to accurately sample the distribution of opinions. For example, in a group of 1000 people who are equally divided between two candidates, a poll may talk to 490 of one group and 510 of another, suggesting a slight lead for one candidate when there really is none. Thus, if a particular poll suggests that a candidate is in the lead, but only by a percentage smaller than the margin of error, that lead cannot be claimed for certain.
The time at which the poll is conducted also affects its reliability. Back in May, the New York times examined how closely polls predicted, on average, the winner of a presidential election (in percentage points) back to 1980 as a function of the number of days before the election at which the poll was conducted. Polls were not with 5 percentage points of the actual outcome until ~80 days before the election. By ~4 weeks (our current distance from the 2016 election in the US), poll predictions were 3-4 points within the actual result, and as close as they ever got to reality.
Evidence from the same report also suggests that polls do not shift by more than a few percentage points based on events like debates. Thus, the debates matter only when the election is close. Party conventions typically give bigger boosts to support in polls for a particular party, and occasionally big news items will cause observable dips (like Mitt Romney’s 47% remark in the 2012 election).
Luckily, we can look to studies of polls that covered the last presidential election to see which had the most reliable algorithm. The National Council on Public Polls, for example, provides a report on the 2012 pre-election polls to give you an idea of who got it right last time. Pew Research highlights their own record on predicting the 2012 popular vote from that report on their website.
Polls are important not solely for predicting election outcomes, but also because they reflect what issues are important to the voting public. Candidates will base how much time and money they spend on particular issues or proposed policy changes on how much voters suggest they care in their polled responses. Hopefully our polling technology can continue to keep up with our lifestyles to improve polling response rates so that polls can continue to be a valuable resource.
Until next time, this is Sabrina Stierwalt with Everyday Einstein’s Quick and Dirty Tips for helping you make sense of science. You can become a fan of Everyday Einstein on Facebook or follow me on Twitter, where I’m @QDTeinstein. If you have a question that you’d like to see on a future episode, send me an email at firstname.lastname@example.org.