Is the Metascore Algorithm for Rating Video Games B.S.?

Guiding internet users to useful content is one of the most lucrative businesses in the world. This process is called aggregation. Google and other search engines form the top of the food chain, aggregating all of the content on the web in response to queries. There are all sorts of other important aggregators though, and you probably use at least one every day: Fark and Reddit for web content, Rotten Tomatoes for movie reviews, and Metacritic for a variety of media, but most importantly, video games.

Metacritic has gone much further than its brethren by attempting to codify the reviews they aggregate. They don't simply sort reviews into the positive and the negative, like Rotten Tomatoes or count up-and-down votes like Reddit. They have created an algorithm that attempts to assign each review, regardless of the scale it uses (if any), a 0-100 score. It then takes the 0-100 scores from many reviews of the product, weighs them based on the reputation of their respective publications, then averages those numbers into one final 0-100 score for the game. Finally, the converted score is color-coded according to the scale below:

This number-and-color summary has become the industry and consumer standard for evaluating the quality of video games and their creators. This has led game developers, publishers, and journalists to wonder: are these scores right? And why are they so important?

The answer to the first question is often no. Every game website, magazine, or TV show Metacritic aggregates the reviews of, uses it's own rubric for evaluating games and assigns scores on a different scale. Some use 1-5, some 0-10, some A-F, and some none at all. But Metacritic converts each of these reviews into their scale in particular way that does not always match what the review source intended.

Let's compare the 1-100 scale to A-F. For a 1-100 review-scale site like PC Gamer, a score of 49 means the game is ever-so-slightly below average. On the rubrics above, 49 is red and "Generally Unfavorable", which is very scary-looking when surrounded by yellow and green games on the site. A-F publications like 1UP present similar problems. Is an F equivalent to a 50 or a 0? It depends on the publication, reviewer, and tone of the individual review. Thus, the iOS game Super QuickHookhas a Metacritic 96 and is in the top-20 all-time while Starcraft, widely considered one of the greatest games ever, has an 88 and doesn't even make the cut for "Universal Acclaim". That might seem like a small difference, but it is hugely important for consumers inundated with choices, and publishers trying to steer them towards their products. The point is that there is a lot of room for human error in the grade translation.

THQ and the other big game publishers have been learning the answer to the second question the hard way. In March, THQ released their big-budget multiplatform shooter Homefront. Its Metacritic score is 70. That's not terrible. That's "Mixed or Average" according to the rubric above. And yet that score sent THQ stock tumbling 21% in one day after the score became public, provoking outrage from their staff and shareholders. They were slammed again a month later when Red Faction: Battlegrounds came out and received a 49, launching another burst of complaints. Ubisoft staff expressed discontent over the 81 score its Wii game Red Steel 2 received when it came out last year.

Metacritic scores are so important to these big-budget game companies that EA and Activision, the two biggest game publishers on Earth, tie salaries and bonuses to them for some of their employees. I have conversed with a former highly-placed EA employee who spoke rhapsodically of the high Metacritic scores his games had received and said openly that these scores had become EA's standard for evaluating employee performance. Recently, Metacritic caused a fresh bout of controversy by implementing and then removing a system to track reviews for individual people, making it even easier to use these questionable scores to pass judgment on the work of an individual working on a massive group project.

Metacritic scores do not deserve to be the measuring stick for consumers to base their game-buying decisions, much less for employers to base decisions about employee compensation. But that doesn't mean that Metacritic is bad. I use it almost every day. If you want to find out about a game, you can go on Metacritic and get a quick impression of how good or bad it might be. And then we get to the good part. Right there on one page you have links to all the major reviews of the game. Just go read them! Pick some good and bad ones. You'll have a much better idea of how good the game is actually going to be. If you do that enough, you may even find that you engage intellectually with certain reviews or publications and value their reviews more than the others. Then you will have become your own walking, talking Metacritic, able to base your game-buying decisions (or hiring and firing decisions, depending on your occupation) on the well-reasoned arguments of authorities you trust instead of a machine-generated number.

Photo by Joystiq