Why the Industry Shouldn't Rely on Metacritic

[GameBanshee News]

An interesting op-ed piece on Gamasutra discusses how the industry tries to use Metacritic as a benchmark of quality...and shouldn't. On the one hand because it doesn't represent sales, which is sometimes claimed, on the other because it doesn't actually represent critical reception.

If Metacritic worked as objectively as Sir Galton’s analysis of every piece of data, we would quite possibly have an indicator of quality, and therefore an accurate measure of our gaming cow. Sadly, it does not. Metacritic does not include the entire data set, only those selected by Doyle:

"This overall score, or METASCORE, is a weighted average of the individual critic scores. Why a weighted average? When selecting our source publications, we noticed that some critics consistently write better (more detailed, more insightful, more articulate) reviews than others. In addition, some critics and/or publications typically have more prestige and weight in the industry than others. To reflect these factors, we have assigned weights to each publication (and, in the case of film, to individual critics as well), thus making some publications count more in the METASCORE calculations than others."

I get why he does it, and he likely had the best intentions, but it doesn’t work. The critical view is subjective. Doyle's determination of the critic's value is also subjective. So we are really getting a third generation facsimile of a subjective view of the quality of a title. If you factor in the uncertainty of the gallant, but flawed effort to convert A to F scales to numerical equivalents, Sir Francis Galton would certainly call foul.

[scrumdiddly]

I generally just scroll straight to the most negative reviews for any game. They're usually the best, because you can usually tell if their points are valid. I think our judgement of games should be balanced between the ways it falls short of perfection and what it does right, but so many reviews aren't like that. They skew one way or the other and the rating usually doesn't reflect the review anyway!

My main gripe with Metacritic right now though is the fact that some stupid baseball game is #2 on the PC all-time high scores list, all because it only has 4 deliriously positive reviews. What was that about "weighting"?

[DesR85]

I rely on Game Rankings a lot for video/PC games. If a game have lots of hype surrounding it, I will be very suspicious of the reviews of that particular game. I prefer reading reviews of games that don't have lots of publicity surrounding it.

[Kipi]

Originally Posted by scrumdiddly

I generally just scroll straight to the most negative reviews for any game. They're usually the best, because you can usually tell if their points are valid. I think our judgement of games should be balanced between the ways it falls short of perfection and what it does right, but so many reviews aren't like that. They skew one way or the other and the rating usually doesn't reflect the review anyway!

My main gripe with Metacritic right now though is the fact that some stupid baseball game is #2 on the PC all-time high scores list, all because it only has 4 deliriously positive reviews. What was that about "weighting"?

Same here. lack of negative aspects is far better than lack of especially goody aspects. For example, if game A has couple of positive side that shines out and few very bad aspects and everything else is mediocre, it is usally worse than game that is missing those negative sides even though it doesn't have much shinies neither. So, critics gives better picture if the game is worth of testing or not.

Also, reviews that only praise the game tend to focus only couple of best aspects, and usually won't even mention others. On the other hand reviews than deal the negative and positive sides equally usually tend to concentrate to far more aspects, giving actually better picture of the game.