December 11, 2017

The Calgary mayoral election polling experience earlier this fall, where misleading polls played a significant role in the dialogue around the election, provides a compelling reason for reviewing why some polls are accurate and useful in public discourse, and some are not.

We need more responsibility from media and research companies to provide higher quality research and not just public relations for the research firms during elections – although the public relations value of providing wildly inaccurate results and then defending them in Trumpist style before the election, is dubious at best.

It has long been an acceptable strategy for market and public affairs research firms to publish results for the media to report on. The Gallup poll has been released since it was founded in 1935, and continues to be released to this day. But in 1935 releasing polls was quite unique; today there are thousands of polls published by hundreds of research organizations in North America.

Alberta seems particularly susceptible to dubious polling results. The Calgary mayoralty race was preceded by the surprise NDP victory in the provincial election. The failure of the accuracy of these public polls, not to mention the many that failed to predict the Trump victory in the last U.S. presidential election, leads people to wonder if accurate polling is even possible. The short answer? Of course it is. Allow me to explain…

There are six potential methodologies for collecting data for random, statistically representative market research: onsite collection (sometimes door to door), mail, live-operator telephone, telephone robo-dialing with recorded voice interviews, online panel research, and online sampling.

Onsite and mail research have their place for some types of research, but not for political polling. Robo-dialing polls are even worse: notoriously poor for getting representative samples, because they are so easy to hang up on, even after the first couple of questions have been answered. It sometimes takes hundreds of dials to get one complete interview making the quality of the sample suspect. We have seen reports of 100,000 dials to get 1,000 completed interviews. The primary strengths of robo-calls are that they are cheap and quick. Panels can be good if they are representative of the population – which is often, but not always the case. We advise using panels with caution, particularly for smaller areas where it is often difficult to get large enough populations. Sampling from general internet usage is limited, and is very poor for general public opinion research due to an inability to verify area or personal demographics.

Experience shows that the best approach is different than what often gets commissioned by the media, which is a one-off poll, and, at best, seldom more than a weekly sample, of what is viewed as the entire voting population. An ideal political polling process involves:

  • Sampling randomly from target ridings, not the general public,
  • Telephone calling into the targeted area – cell phones and land lines; with a supplementary sample for younger voters who are notoriously difficult to contact for interviews,
  • Calling over a period of three or four days so that numbers not answered can be called back three to five times on different evenings to try to find someone at home,
  • Tabulating results daily if possible, but at least three to four times a week, and
  • Weighting by age, gender, and likelihood to vote.

This process provides the best chance of getting a representative sample, allows for almost all interviews to be completed once they are started, and provides better feedback from the interviewers for issues encountered while completing the interviews.

In the heat of a campaign, politicians often say they have internal poll results that are different than ‘and more accurate’ than the published polls. And when that turns out to be true, as it did in Calgary, it’s often because their pollsters have employed a more painstaking and rigorous methodology than the polls commissioned by the media.

Dr. Brian Owen

CEO and Chairman