Live Blog: DICE 2009 - Power Panel! How Do Review Scores Effect Game Development?


Posted February 19, 2009 - By bleahy

It's time for a POWER PANEL. A collection of gaming giants, converging on a single stage to discuss how review scores impact game development. Participating in the panel are:

  • Mona Hamilton, Vice President of Marketing, Capcom
  • Rich Hilleman, Chief Creative Office, Electronic Arts
  • Chris Taylor, Creative Director, Gas Powered Games
  • Danny Bilson, Senior Vice President and Creative Director, THQ

The panel is being moderated by Julianne Greere, editor-in-chief of The Escapist, which doesn't do game reviews. Well, except for Yahtzee's Zero Punctuation. This will be the last live blog of the day, but be sure to follow TheFeed on Twitter for live coverage of tonight's AIAS Awards. They won't let me bring my laptop, but they can't make me give up my iPhone. You'll be getting updates under 140 characters during the whole show.

4:31 PM - And away we go. Are we designing games for the consumer or the reviewers? Do developers let review scores change the way they make games? Data shows that strong review scores in many genres lead to high sales.

Have you seen any games designed specifically for reviewers?

Chris: No. I aboslutely don't think about reviewers. Ever. There are hundreds of reviewers and potentially millions of customers. If you are part of your audience, you can thankfully design for yourself. You have to hope that the reviewers are part of your market.

Danny: I agree with Chris. The first person you have to design for is yourself. If you're not a gamer, you probably wouldn't be designing games in the first place. I've actually spent some time thinking about where they can improve the Metacritic score without touching the code.

Danny (cont'd): If developers pay a little more attention to things like story, there can be a 3 to 5 to 10 percent increase on Metacritic. I've convinced people to spend money on writers and actors by explicitely saying it will raise the Metacritic.

Rich: The hardest thing in the business is to get someone to say "I'll buy that." You're trying to get a customer to buy a game and hopefully reviewers are part of that audience. Both are focused on the first 30 minutes of the game as well as the clarity of what you're trying to say in the game. If you've done an effective job of creating something for the customer, it should hold over for the Metacritic.

Q: Are all game developers hardcore gamers?

Rich: No. We're getting older. We have other lives now, but you've got to be able to listen, like, and relate to your audience. I've never made a game without a specific person in mind.

Mona: You definitely have to design games for the consumer. Who is that consumer? If it's a core game, the reviewer or other developers might fall into the target audience anyway. As we spend more and more on game development, you have to have a game that can span past the hardcore market and reach the casual as well.

Q: Do marketing spends affect game reviews?

Mona: No, but I think they can affect game sales. We've had games with 60-70% ratings that can sell well. I think NIntendo has proved this by going after the mass-market consumer. Games like Iron Man, with a lot of spending around a license, can sell well.

Rich: I think marketing spends can affect how many reviews for a game exist. You can look at games with 140 reviews and then some with 3. Marketing affects this.

Danny: I think marketing affects everyone. Take a look at Assassin's Creed. It didn't top the review charts, but sold extremely well with a lot of marketing behind it.

Q: How many of you have seen an error in a review?

Mona: There are times where I've seen reviews that point to a reviewer stopping mid way through.

Q: There is already bias. Certain genres just rate higher on Metacritic. There is also platform bias. Many genres rate higher on the Xbox 360 when compared to the Wii. Casual genres score higher on the Wii.

Rich: I think the Wii wouldn't rate higher for those casual genres without Miyamoto.

Q: Can you openly talk about your opinions on the review system?

Rich: I don't get too upset about reviews. They are many people's opinions. I think it is important to understand a reviewer's perspective. For the Wii, the reviews are informative, but you have to pay attention where the review is coming from.

Danny: I don't have a problem with the system. It is how your company uses these review scores. A team of THQ's was presenting a game for the first time before getting the go ahead. The marketer wouldn't shoot for higher than an 80% Metacritic score. He wouldn't put 90% because it would affect the game's forecast, which would change the amount of money that would be spent on the game. The forecast is someone's assumption, but can be used as a weapon by the marketer. If they hold the Metacritic target lower, they get a better bonus if the game does well. I wish it was like how they review movies. Movie reviews don't really affect box office numbers. If you can put a number on art, you're better than me. I use the Metacritic number, but all I'm saying is that a game is "good."

Chris: I see it as a tool for consumers to judge an experience with very little effort. If you're shopping for a car, but don't know anything about the newest models. You can do research, but then you go see the car and test drive it. You should be downloading demos before you buy something. If you don't download the demo, you're just phoning it in. Who are reviewers writing them for? The publishing executives. I alread get the news when a game I make isn't selling, but I don't need to look at a 60% on Metacritic after that. The fact that there's a correlation between sales and Metacritic scores is natural. I got a contract recently that promised me a huge bonus if I hit a Metacritic score over 91%. It's ridiculous that they were going to pay me based on review scores and not sales.

Rich: For me, Metacritic scores can be a good indicator of the strengths of a new IP.

Chris: I threw that contract in the trash.

Danny: It's sad that we're not working together anymore.

Rich: The most statistically focused reviewer has been Consumer Reports and they don't review games, or movies, or music.

Danny: You can't put a score on art.

Mona: Reviews are subjective. It's that reviewers experience. With Metacritic, you're taking qualatative data and putting it into a single quantatitive number.

Danny: When you look at one guy bringing your review down with a 50% who just has some blog it hurts. Time Magazine gets the same weight as some random game dork. When you get a 3 out of 5 in a magazine it could translate into a 70% (should be 60%, but hey, whatever).

Rich: In our sports business, sequel scores keep going down. With shooters, the top three shooters and the top three on Metacritic.

Danny: Well, good is good. Common sense is a big thing. I like to think I know what a good game is and I think I know what a bad game is.

Chris: A couple of years ago, we put a game out and the first couple of scores set the bar. We tried to game the system. It didn't work. The idea was to have them put more weight to competant press. These are reviewers that play the game to the end. Metacritic scores are for people that don't want to do their homework. It means nothing. Let's move on.

Danny: Of course it depends on what kind of game. I agree with you on family and casual games. When I left EA and I wasn't getting free games. Now I'd go to Metacritic to see where I wanted to spend my money. Chris, I think you're right. If you play a demo you know if a game works for you, but Metacritic isn't going away.

Mona: High game scores don't translate to high sales. Especially on the Wii.

Danny: Da Blob hasn't cracked 400,000 units.

Mona: Okami on the Wii capped out at 160,000 units.

Rich: The mainstream press tends to get things correct. Metacritic is a consolidation of every report and there's no test for inclusion.

Chris: You can now hire someone to give you a review score in advance just to tell you that you need more time and money.

Danny: He's right. Most of the management in the game business don't play games. It's for the management that don't play games. We know what a good game is. I can call a Metacritic within 2 points.

Rich: EA had its best year last year defined by Metacritic, but we had two 49% titles this year. It brought down our average by two points. Those products deserved the 49% ratings, but it wasn't representative of the rest of EA's catalogue. Metacritic is a sample of a sample. If you heard what Gabe talked about last night, now we can look at data immediately. We don't have to look at samples any more. We can use open betas to make products better. How relavent will Metacritic be in a world where we have perfect data about our customers.

Chris: I don't think people really go to Metacritic. They go to their favorite site, talk to their friends, and play the demo.

Rich: It matters in the retail world. If one out of twenty-five retail clerks see the Metacritic score, they can impact countless customers.

Danny: I think we're selling to retailers now and not customers. Piracy is a really important issue. We had a game ship today (Dawn of War II), but it was cracked three days ago. Our forecasts on Dawn of War II were half the amount of pre-orders on Steam.

Rich: I think being afraid of data is wrong. You just need the whole picture and not just a sample.

Danny: Just so I'm really clear about data. Data is not inspiring. You need to find something that inspires you as a developer and not just look at what you think people want to play. Success is about innovation. Metacritic and other data can confuse and pollute the creative efforts in development.


This concludes the panel. Very great stuff.

Live Blog: DICE 2009 - Power Panel! How Do Review Scores Effect Game Development?


Blog Tags

  • International Sexy Ladies Show: Messy Cat Fight

    Posted: January 27, 2010

    644,696 Views | 00:49

  • Sara Underwood's Naked Bike Ride

    Posted: June 22, 2011

    1,316,284 Views | 05:20

  • Laser Snake Robots, More Mars Rovers and BigDog Is Back

    Posted: September 25, 2012

    1,290 Views | 03:00

  • NBA 2K13 Launch Trailer

    Posted: October 4, 2012

    5,601 Views | 01:53

  • Casual Vomiting - Web Soup Investigates

    Posted: March 30, 2011

    7,570 Views | 02:52