You’re Doing it Wrong Metacritic

Earlier this week Duke Nukem launched for the Xbox 360, PS3, and PC.  As you may or may not have heard, the game is bad, and critics have pulled no punches in their reviews of the game.  Popular game reviews aggregator, Metacritic, took it a step further, taking the liberty of giving the game a 0 out of 100 and then subsequently defending that stance.  Now I’m not one to tell people how to operate their businesses, but when you are as popular as the metacritic site is, and publishers use your scoring system to dictate payment to employees, as well as have the ability to swing the underlying stock price of a publically traded company, there definitely needs to be some level of accountability.

Metacritic uses a 100 point scale, which means that they have to convert the scores they harvest to their scale, whether that be converting ten point scales, five point scales, or other less traditional scoring mechanics. In these less traditional scoring systems is where things begin to get a bit cloudy since these sites use scores that are not numerical.  In these instances, Metacritic rewards a score to the game that is interpreted by them.  The problem is, their interpretations are wrong.  By their logic, which they vehemently defended recently, the company claims:

Metacritic converts scores from all publications to the full 0 – 100 scale.  So the lowest possibe score from a publication – i.e,  0/100 (from GameSpot or IGN, for example), 0/5stars (from GiantBomb or GamePro for example), and “F” (from 1up or GameShark) converts to Zero on Metacritic.  The middle score (i.e., 2.5/5, 50/100, or “C”) converts to 50 on Metacritic.  The highest score (5/5, 100/100, “A”) converts to 100 on our site. Sure, some schools in America convert an “F” to 58, but those schools rate on a 42 point scale from 58 – 100.

In this recent case, 1up, who grades video games with basic A-F scale, should be broken down with 100 points for an A, 80 points for a B, and so on.  That so on, stops at 2/5 though.  An F, which is the score that they recently issued to Duke Nukem Forever, is a zero on Metacritic.  It works for 1up in their scoring system, the game can be deemed anything between  2.5/10 – 0/10.  But being that Metacritic uses a blanket policy for their metascore, 5/5 (A/A) equals 100%, but 1/5 (F/A) equals 0%, so how does a scale which includes five values A,B,C,D,F turn into a four point scale?  The answer is, it doesn’t, it’s an arbitrary decision that further proves that the most resourced site for quantifying a game’s success is driven by a  fundamentally broken mechanic that is derived completely out of thin air.  Even in their rebuttal, Metacritic is wrong.  Giant Bomb, who uses a five star scale, can also opt to award zero stars, but if their lowest score was 1 star, would a one star rating be worth 0/100?  Of course it wouldn’t, it would be worth 20/100.  Simple fractions which they teach you in elementary school will tell you that.  This stance by metacritic places significant weight on the lower end of the scale.  By this method of thinking, not only is the F rating inaccurate but it would also throw off the rest of the scale, with a D needing to be worth then at least 25/100, which metacritic at least follows up on.

The biggest problem isn’t that Metacritic is wrong, but it’s that they aren’t the authority on games, it’s a non-expert site that uses a flawed scoring system to undermine the actual expert scoring system of the aggregated content.  When 1up places a D rating on a game, in their scoring system the lowest possible score in that letter category is a 3/10 and extends all the way to 4.5/10.  The most this score can be on Metacritic is a 2.5/10.  This fundamentally drags down the average at every point other than A.  Of course, you can always take into consideration that for every metascore that is dragged down by the way Metacritic scores games, there are also just as many instances where scores are artificially inflated by the use of 100/100 for their top marks, whether or not the original source of the review felt the game to be sheer perfection or not.  Either way, it’s pretty clear that the system is flawed in more ways than just one.

Metacritic has seen its popularity soar in recent years.  Developers and publishers are always looking to score highly with the ratings company, but this stance on mathematics isn’t doing them any favors. As it stands the scoring system gives those that use the website for purchasing decisions an inaccurate representation of the actual reviews in some cases. Worse yet,  how the bright people at these publishing houses, analysts, developers, and other industry decision makers can actually look at this ratings system and base multi-million dollar business plans off of what is basically a flawed system at the most basic level.