A common reaction to Microsoft’s E3 briefing this year was “Where are the games?” Core gaming was conspicuously absent from the keynote, supplanted by a focus on the Xbox 360 as an entertainment hub.
This came as no surprise to any observer of the video game industry as the race to provide an indispensable media center for the living room has governed the later phases of the current console cycle. And this year, for the first time ever, the United States Presidential and Vice-Presidential debates featured interactive television polling, hosted on Xbox Live.
It was a clear demonstration of how Microsoft is trying to expand the definition and usage of the Xbox 360 but what, if anything, can we gleam from those polls? Could such an unrepresentative, self-selecting sample of respondents really have told us anything other than what the Xbox Live audience thought? We spoke with David Rothschild, an economist with Microsoft Research, about how the debate polling was planned and executed and what lessons were learned.
The poll questions were designed in collaboration with YouGov, an international market research firm that specializes in internet-based polling. “They’re an organization that has really looked forward to not [only] understanding how great polling has been successful over time but thinking [about] what are the next steps,” said Rothschild.
“Where is polling going to go, how do you use maybe non-representative samples, how do you maybe use self-selecting groups, how do you think of what the next steps are going to be? They’re a great group to work with and everything is done in collaboration with them as far as the questions go, in order to think of what do we want to test out so that we can use this handful of people, this Xbox group of people, and not just how to be fun which we wanted, to be engaging, but also meaningful.”
The first step in designing a poll or survey is deciding what information you want to gather from the respondents, and to then design questions that gather the desired information. In the case of the Xbox Live debate polling, Microsoft not only wanted to gather information about the political leanings of the Xbox audience but also to test how well the interactive polling engaged them, in order to prepare for future interactive events. The Presidential debate polling is probably best understood as an experiment.
“Some of these types of questions have been used on YouGov’s panels or in my research previously, but we didn’t go out and have a special test panel for this. Quite frankly the Xbox users in a way are that test panel, right?” said Rothschild. Some of the questions polled how users felt during different segments of each debate, such that Microsoft could present poll results when the debates were finished. Other questions were meant for long-term analysis.
“No one’s ever done this type of stuff before,” said Rothschild. “There are some questions which we haven’t really pulled aside to talk about, because we asked them so we could look at the data afterwards and say “Well, what does this really show us? What kind of information are people giving us?” so then maybe next time around we can actually really use this.” The poll questions weren’t just about measuring how the Xbox Live audience felt during this election cycle. They were also about planning for the next cycle, and the cycle after that.
It should also come as no surprise to anyone that the Xbox Live audience is skewed heavily towards young and male respondents. The audience was also heavily in favor of the Democratic slate throughout all four debate events. Therefore, it wasn’t the raw numbers which interested Rothschild the most, but rather watching the movement of respondents who indicated at the beginning of a debate that they were either undecided or only leaning towards one of the candidates versus watching the respondents who had already chosen a candidate.
“I’ll emphasize the fact that all of this polling apparatus is separate from any sort of other private information, so we’re not doing anything to track individuals beyond their polling experience, but we do track the same person during the course of the debate,” Rothschild told us.
At the beginning of each debate, baseline questions helped Microsoft break the respondents down in terms of who they supported. 85% of the Xbox Live users who participated were strong supporters of either Barack Obama or Mitt Romney. “We focused in on the ten to fifteen percent of people in this debate that we classified as either leaning Romney, leaning Obama, and undecided, and looked at how they did during the course of the debate, and that was really what gave us a good understanding about what was going on.”
Most pollsters essentially take a snapshot of their respondents and then analyze that data. Microsoft had the ability to track the movement of respondents’ answers throughout four 90-minute events, and that’s what made the opportunities so valuable. “We see huge amounts of people who switch their vote,” Rothschild told us. “We get to think about who these people are, what are their demographics, why are they moving, and even if it’s not fully representative it’s definitely quite informative and definitely something that people had previously not had the opportunity to see.”
This year the interactive polling helped to gather information on a young, mostly male audience which is notoriously difficult to poll. Rothschild hopes that, considering the direction in which Microsoft is trying to angle the Xbox 360 (and assumedly will try to angle the next generation console to follow), that this beginning step will open the door to improved data gathering.
“I’m absolutely positive if we do this in two years or four years we’re going to have a much more representative group, if we keep moving that way as the Xbox becomes this home entertainment system,” Rothschild said.
“We’re really excited to see, as we pursue this new interactive TV technology, and the new polling technology, new questions, new graphic interfaces, new ways to reach out to people, I really think we’re going to be at the forefront. One of the things that has been at the basis of all the work that I’ve done as a professional since grad school is thinking about this idea that it’s increasingly hard to reach people in the standard polling methodology, and I do think we’re at the forefront in what’s going to happen in the future.”