Angus Reid manages to stay in his chair for the first 30 minutes. But then, as the excitement over one of his company’s innovations overtakes him, he can’t help propelling himself from his buttery leather chair toward his desk. While crowds throng through the streets beneath his airport-sized office window in the Pivotal Building in Yaletown, he taps at his keyboard for a few seconds. And then, on the flat-screen TV attached to the office wall, an image that looks like a video game pops up. But it’s a video game that’s all about . . . shopping at the local supermarket.
Instead of progressing through levels where you have to destroy flaming fireballs or vicious trolls, you, the viewer, walk through the simulated aisles looking at the simulated boxes and cans and plastic-wrapped packages. And this is where, theoretically, you – the all-powerful, all-mysterious, all-knowing shopper – can provide a clue as to which box or can, which shelf position, which colour, which label attracts you the most. Businesses will pay thousands of dollars to know this, to discover the little trigger in your heart.
“This is the future,” says the 62-year-old Reid, CEO of Vision Critical, his missionary zeal unimpaired by the cold he’s fighting. “Eighty per cent of Canadians are online. The telephone polling industry will be largely dead and gone in five years. The telephone guys have got to realize that.”
Reid’s take-no-prisoners pronouncements about the future of polling and market research are making people in his industry edgy these days. “There is anger going around the industry between him and the others,” says one mildly exasperated local market researcher, Barb Justason, who is also the western representative for the Marketing Research and Intelligence Association, Canada’s industry group. “I think Angus’s work is great, but if there’s one thing that bugs me, it’s Angus going around saying the telephone is dead.”
Says another local researcher: “A lot of people are irked by him because he’s putting a lot of fear in our clients.”
[pagebreak]
To be accurate within the margin of error, Reid, with his mantra that “online is the only path to the future,” is making his peers edgier than they already were, as the polling and market-research business shifts under their feet. That shift doesn’t generate as many agonized headlines as the music, television or news businesses, but it’s still there. Market researchers have had to grapple for 15 years with people who are increasingly declining to participate in polling, who are moving wholesale to cellphones and iPhones, Facebook and Twitter to communicate, and who have been revealed – when you can reach them – as not that easy to get true opinions from in the increasingly complex science of public-opinion research. They lie, they say what they think you want them to say, they get tired, they are easily influenced by the question they were asked just before the one they’re answering now, and they will sign up to do polls just for the rewards.
On top of all that, the global recession of 2008-09 produced a decline never before seen in the industry. “Polling has only been alive for 50 years, and in every one of those years until last year there was seven to 12 per cent growth,” says Steve Mossop, the Vancouver-based president of market research for the western division of Ipsos Reid, which does $1 billion worth of business internationally every year, making it the second-largest market-research company in the world. (Angus Reid, founder of the Angus Reid Group, sold that company to France’s Ipsos SA in 2000.) “It’s only this past year that global research spending dropped. It shrank by three to five per cent. That has made us question our industry.”
It can be hard to keep a level head and to reassure yourself and your clients that the world hasn’t changed that much, that the telephone is not dead, that it’s worth spending a lot of money to get the opinions of 500 people and that the industry where you’ve spent 30 years isn’t about to disappear in a puff of smoke. Evi Mustel is breathing deeply and trying to stay very, very calm these days.
“For every project, telephone and online both have their advantages and disadvantages,” says the 54-year-old Mustel, co-owner of Mustel Group Market Research. She exudes tranquility as she sits in her charcoal-painted office at the entrance to Granville Island, its windows overlooking a small, still body of water. “I know that, for Angus, it’s his marketing strategy to go strictly Internet. That was, I think, a business decision, a marketing position for him.”
She has opted to do a mix, using the telephone and online surveys almost evenly, with a slight edge toward online. “Online has done very well in some elections and there have also been some big misses,” says Mustel, whose company continues to use the telephone for elections. But she now finds herself in the position of having to convince clients that online is not always the answer. So do others in her business because of growing concerns about the relevance of phone surveys, fed by the likes of Reid.
Mustel, like Reid, is one of the pillars of the small but thriving market-research sector in Vancouver. Her company, ranked ninth-biggest by Business in Vancouver among women-owned companies in B.C., has remained locally based since she launched in 1980, in contrast to Reid. After selling the Angus Reid Group in 2000, Reid was bound by a five-year non-compete clause, then in 2005 moved into the online world and created an international organization, Angus Reid Strategies. Recently, he merged that entity with Vision Critical as part of his drive to develop the visualization possibilities for online polling.
[pagebreak]
Tech warning: The Mustel Group's Evi Mustel says the argument that online polls are always best is misleading. Some jobs still need dialing
Other major players in the city include NRG Research Group, which is dominant in health-care research, Synovate and Ipsos. Then there are small companies such as Justason’s, which do almost no political polling. And there are companies such as Bob Penner’s Strategic Communications Inc., a crucial component of Mayor Gregor Robertson’s strategy team, that does no market research, only polling on public-opinion issues. Branches of eastern-based companies also operate here, with mixed success because of the language issues. In Vancouver, where 25 per cent of households speak Chinese and around six per cent speak Punjabi or Tagalog, companies that haven’t figured out how to connect with those groups struggle. B.C. resident Greg Lyle, whose company, Innovative Research Group, is Toronto-based, has formed partnerships with organizations such as the Vancouver-based Chinese immigrant-aid organization SUCCESS to help set up multilingual call centres. But others haven’t.
Many of the people now working in polling in Vancouver started in the 1980s, just as the polling industry was ramping up across Canada. Although polling had been around a long time – gaining fame through its ability to predict or to spectacularly flub in predicting elections – it really took off as companies, governments and even advocacy organizations started looking for more sophisticated ways to understand what their customers or taxpayers or potential members wanted from them. Now market research is the biggest part of the business, with Reid, Mustel and Mossop all reporting that political and public-opinion polling represents five per cent or less of their research spending.
But all the biggest companies do political and opinion polling as a branding strategy. Those public-opinion polls – how do people feel about the Olympics, who would they vote for tomorrow and how much sex they have – are what get them into newspapers, radio and television.
The political polling, in particular, is their big final exam. “Political polling would not even register on a percentage scale, but it’s the one that garners the most attention,” says Mossop. Pollsters have very few ways of checking to see if their public-opinion surveys are accurate. In very rare instances, they’re able to compare survey results with statistics. But election results are the real and very public measure everyone uses to verify that their methodology is working. Political polling has also tended to elevate those market researchers to near-omniscient status about politics, to the annoyance of those whose major focus is politics.
“The most confusing part is people think media pollsters are political experts. The emergence of market-research pollsters as political commentators has weakened the discussion,” says Bob Penner. “Since the onset of daily tracking polls in the media in 2004, there has been more information, but it’s kind of information out of context. The biggest single problem is that the media and pollsters have to attribute a lot of causality. If someone moves one point, they have to say it’s because someone did something yesterday. It’s almost never true.”
[pagebreak]
While pollsters’ political acumen may be overrated, their ability to help corporate clients understand the world isn’t. A few companies with big savvy marketing departments may use market researchers just to send out a questionnaire or an online poll with visuals, but the vast majority of them rely on the market-intelligence sector’s increasingly sophisticated understanding of both social psychology and how to reach people. Pollsters such as Greg Lyle say there has been a quantum leap forward in both of those facets of polling in the last 10 years, but that one is much more significant than the other.
“I’m not interested in how we collect the data. The social psychology is moving way faster than the data,” says Lyle. He maintains there are some kinds of research that can only be done by phone, while online is fine for others. “What I love about public-opinion polling is that we now have a way better understanding of how people think. Our memories are emotionally based, not fact based. If you want to change people’s minds, you need to appeal to them emotionally. That changes what you do with research.”
It was all so much simpler when the polling business got started for good in 1936, the year when George Gallup predicted that Franklin Roosevelt would defeat Alf Landon. Gallup’s new Institute of Public Opinion polled only 2,000 people before the 1936 election. But he chose them randomly, according to mathematical methods, and his surveyors interviewed them face to face. As a result, his prediction trumped the until-then omniscient Literary Digest poll, which mailed out millions of mail-in surveys to its members.
Forty years later, polling was still done strictly face to face and door to door, the way Margaret Atwood’s lead character did it in The Edible Woman. The telephone was regarded as somewhat suspect. Eventually, pollsters succumbed to its lure because it was so much easier and cheaper, especially as the cost of in-person surveying started to rise. But around 15 years ago, telephone polling started developing its own problems. More and more people refused to participate, which meant that polling companies had to do a lot more work to get the minimum numbers needed to make a poll valid. Questions were also raised about whether the people willing to answer were truly representative of the whole population.
As the Internet started rising in popularity, polling companies began experimenting with it. Some pollsters, including Reid and Mossop, say the debate over using online polling is a non-issue. “We’ve been polling online since 1994, and every year the amount we do online continues to go up,” says Mossop. “Back then, 50 per cent of people were online. Now it’s 90 per cent.” Mossop says the accuracy has been proven repeatedly over the years.
But when you read such journals as Public Opinion Quarterly or listen to other pollsters, it doesn’t sound as though it’s quite a non-issue yet.
[pagebreak]
“In the social-science world, no scientist would use online polling because they don’t have true probability,” says Kennedy Stewart, an associate professor at SFU’s public policy program, who specializes in research methods. “They’ve really sacrificed the laws of mathematics for convenience.” What Stewart and other critics emphasize is that online polls don’t allow polling companies to make a truly random selection among the population. With door-to-door or telephone surveys, pollsters will go to every 10th door or will do what’s called random-digit dialing to ensure that they get a group of respondents that represents the broadest cross-section possible.
No one has a registry of email addresses, so online polling can’t be verifiably random. Instead, what companies such as Ipsos, Vision Critical and even Mustel have done is build up what are called “panels.” These are huge groups of people recruited in various ways – as a request at the end of regular telephone surveys, through advertising, through links with commercial partners – who agree to be part of the survey pool. Ipsos has a panel of 200,000 across the country. Vision Critical’s panel includes 100,000 people. Then the polling company distributes surveys randomly to 500 or 1,000 or however many people are needed within that pool.
But, as the literature notes, there are potential problems with such panels. Pollsters end up getting people who like being online, and that can skew towards certain categories: wealthier, better educated, urban, male, people who happen to be interested in the topic you’re polling about, young (though there’s debate over that last one). As well, since people on the panels are usually enticed to stay on through rewards – cash or gifts or draw prizes – online polls can end up attracting people who are only there for the rewards and answer the question as quickly as possible, just to get through them.
A study published in August 2009 concludes that the ability to generate a random sample is crucial for accuracy, despite the claims from some of the online enthusiasts that randomness doesn’t seem to matter since their polls have been generating accurate results in elections (or that telephone polls weren’t really random anyway, since so many people were refusing to participate or couldn’t be reached).
“We find no support for these claims,” concludes the Stanford study, which compared the results of a telephone survey using random-digit dialing with an Internet survey where participants were also selected through random-digit dialing (a novel and somewhat cumbersome technique whereby people willing to participate are then asked to do an online survey and, if needed, are given a computer to do it on). Those results were then compared with the results of surveys done through seven panels assembled through the usual recruitment-and-reward methods.
The results from what are called probability surveys (the two polls where the participants were chosen randomly) and non-probability surveys (the seven polls where participants from panels get to opt in to surveys) were matched against what is considered high-quality data from federal surveys done face to face.
“Probability surveys, even ones without especially high response rates, yield quite accurate results. In contrast, non-probability samples are not as accurate and are sometimes strikingly inaccurate,” says Jon Krosnick, one of the study’s authors, who notes that was true even though the people who put together the online panels had made several efforts to screen participants in order to ensure a wide representation.
[pagebreak]
Or, as Stewart puts it, “You’re going to have some spectacular misses every once in a while.” That 19 times out of 20 they always talk about in reporting poll data – meaning that there’s a chance of a miss once out of every 20 times – could happen more frequently. That’s why the Market Research and Intelligence Association has specified in its guidelines that online polls should not state definite margin-of-error rates, the way telephone polls do.
Reid’s response is to point to his results: nine election predictions that were bang on, except for a notable 10-point miss in the last Alberta provincial election. And election results from his phone-polling friends that have also had spectacular misses – at least as many, if not more, than online polls.
In reality, the critics aren’t driving anyone away from online surveys. Phone polling is irreplaceable, say many market researchers, for small communities, for ethnic groups and for low-income or low-literacy respondents. But even the most diehard of phone loyalists among Vancouver’s market researchers say that online surveys serve many useful purposes and they’re here to stay. “They do work for some things: topics that are socially sensitive, or require visuals,” says Evi Mustel. She used her online panel to test advertisements for a recent Metro Vancouver message about reducing garbage to see how memorable it was. And she thinks that personal issues or difficult topics – things that people sometimes don’t want to talk about with someone real breathing on the other end of the phone – are good candidates for online.
Mustel and her team are working on refining the techniques that computers don’t offer. “A person will decide in the first five to 10 seconds whether they want to engage with a surveyor,” she points out. They can be turned off if someone is hesitant, if they sound unprofessional or if the respondent doesn’t like the sound of the topic they’re going to be asked about. And real telephone surveyors can tell when someone is getting tired or when they’re giving off-the-cuff answers, and can figure out how to compensate. But once the surveyors have negotiated all those potential hurdles, there’s a pleasant surprise waiting for them, says Mustel: “The one thing that I’m noticing about this business is that it’s amazing how it’s not that challenging to get people’s opinions. They are becoming more vocal and engaged. This generation is just a lot more vocal about what they want and don’t want.”
Ultimately, pollsters are grappling with what has been drummed into the heads of teachers for a few decades now. Everyone has a different way of learning, engaging and responding. Some will spill all on the telephone because a voice in their ear is powerful to them; some won’t trust anyone they can’t see. Some will respond better when they’re in focus groups than they will in a one-on-one conversation; some won’t respond to anything that doesn’t reach them on the electronic device in their pocket. And some, in this new graphic-novel world we live in, will only get turned on by that which hits their retinas in a dazzling way, like Avatar or a trip through Angus Reid’s simulated world of consumer choice.
The move to combine visualization with market research will make his company, Reid says expansively as he moves restlessly back in his chair, the “Google of the market-research world.” That would mean he could attract a lot of people. But not everybody.