Metacritic Weighting

[schild]

http://www.gamasutra.com/view/news/189448/Metacritics_weighting_system_revealed.php

But since Gamasutra is hard to look at:

He also reminded attendees that often a publisher's Wall Street stock can change on the basis of a Metacritic score., and as such the site's workings are of practical interest.

The findings will also be of interest to consumers as, if accurate, they reveal that some official magazines and sites (which are sponsored by platform holders in some cases) are assigned a greater weighting than independent sites and critics.

Here is the full listing of score weightings used by Metacritic according to Greenwood-Ericksen's research:

Weighting -- Critic/ Publication

Highest (1.5) -- Dark Zero
Highest (1.5) -- Digital Chumps
Highest (1.5) -- Digital Entertainment News
Highest (1.5) -- Extreme Gamer
Highest (1.5) -- Firing Squad
Highest (1.5) -- Game Almighty
Highest (1.5) -- Game Informer
Highest (1.5) -- GamePro
Highest (1.5) -- Gamers Europe
Highest (1.5) -- GameTrailers
Highest (1.5) -- GotNext
Highest (1.5) -- IGN
Highest (1.5) -- IGN AU
Highest (1.5) -- IGN UK
Highest (1.5) -- Just Adventure
Highest (1.5) -- Machinima
Highest (1.5) -- Planet Xbox 360
Highest (1.5) -- PlayStation Official Magazine UK
Highest (1.5) -- PlayStation Official Magazine US
Highest (1.5) -- Telegraph
Highest (1.5) -- The New York Times
Highest (1.5) -- TheSixthAxis
Highest (1.5) -- TotalPlayStation
Highest (1.5) -- VGPub
Highest (1.5) -- Videogameszone.de
Highest (1.5) -- Wired
Highest (1.5) -- Xboxic
Highest (1.5) -- Yahoo Games
Highest (1.5) -- ZTGames Domain

High (1.25) -- Absolute Games
High (1.25) -- ActionTrip
High (1.25) -- Adventure Gamers
High (1.25) -- Computer & Video Games
High (1.25) -- Console Gameworld
High (1.25) -- Da GameBoyz
High (1.25) -- Darkstation
High (1.25) -- Edge Magazine
High (1.25) -- EGM
High (1.25) -- EuroGamer Italy
High (1.25) -- EuroGamer Spain
High (1.25) -- G4 TV
High (1.25) -- Game Chronicles
High (1.25) -- GameDaily
High (1.25) -- Gameplayer
High (1.25) -- Gamer 2.0
High (1.25) -- Gamervision
High (1.25) -- Games Master UK
High (1.25) -- Gamespot
High (1.25) -- GameSpy
High (1.25) -- Gaming Age
High (1.25) -- Gaming Nexus
High (1.25) -- Maxi Consoles (Portugal)
High (1.25) -- Pelit
High (1.25) -- Play.tm
High (1.25) -- PlayStation Universe
High (1.25) -- PlayStation Official AU
High (1.25) -- PSM3 Magazine UK
High (1.25) -- PS Extreme
High (1.25) -- RPG Fan
High (1.25) -- Strategy Informer
High (1.25) -- Team Xbox
High (1.25) -- The Onion (AV Club)
High (1.25) -- Totally 360
High (1.25) -- WonderwallWeb
High (1.25) -- XGN

Medium (1.0) -- 1Up
Medium (1.0) -- CPU Gamer
Medium (1.0) -- Cubed3
Medium (1.0) -- Cynamite
Medium (1.0) -- D+Pad Magazine
Medium (1.0) -- DailyGame
Medium (1.0) -- Destructoid
Medium (1.0) -- Eurogamer
Medium (1.0) -- Everyeye.it
Medium (1.0) -- Game Revolution
Medium (1.0) -- Game Shark
Medium (1.0) -- Gameblog.fr
Medium (1.0) -- GameKult
Medium (1.0) -- Gamereactor Denmark
Medium (1.0) -- Gamers' Temple
Medium (1.0) -- GameShark
Medium (1.0) -- Gameblog.fr
Medium (1.0) -- GamesNation
Medium (1.0) -- GameStar
Medium (1.0) -- GameTap
Medium (1.0) -- Gaming Target
Medium (1.0) -- Gamereactor Sweden
Medium (1.0) -- The Guardian
Medium (1.0) -- Hardcore Gamer Magazine
Medium (1.0) -- HellBored
Medium (1.0) -- NiceGamers
Medium (1.0) -- Joystiq
Medium (1.0) -- Just RPG
Medium (1.0) -- Level 7.nu
Medium (1.0) -- Modojo
Medium (1.0) -- MondoXbox
Medium (1.0) -- Multiplayer.it
Medium (1.0) -- N-Europe
Medium (1.0) -- Netjak
Medium (1.0) -- NGamer Magazine
Medium (1.0) -- Nintendo Life
Medium (1.0) -- Nintendo Power
Medium (1.0) -- Nintendojo
Medium (1.0) -- Nintendo World Report
Medium (1.0) -- NZGamer
Medium (1.0) -- Official Nintendo Magazine UK
Medium (1.0) -- Official Xbox 360 Magazine UK
Medium (1.0) -- Official Xbox Magazine
Medium (1.0) -- Official Xbox Magazine UK
Medium (1.0) -- PALGN
Medium (1.0) -- PC Format
Medium (1.0) -- PC Gamer (Germany)
Medium (1.0) -- PC Gamer UK
Medium (1.0) -- PC Gamer
Medium (1.0) -- PC Powerplay
Medium (1.0) -- PGNx Media
Medium (1.0) -- Play Magazine
Medium (1.0) -- PlayStation LifeStyle
Medium (1.0) -- Pocketgamer UK
Medium (1.0) -- PT Games
Medium (1.0) -- Real Gamer
Medium (1.0) -- SpazioGames
Medium (1.0) -- Talk Xbox
Medium (1.0) -- The Escapist
Medium (1.0) -- Thunderbolt
Medium (1.0) -- Total VideoGames
Medium (1.0) -- Worth Playing
Medium (1.0) -- X360 Magazine UK
Medium (1.0) -- Xbox World 360 Magazine UK
Medium (1.0) -- Xbox World Australia
Medium (1.0) -- Xbox360 Achievements
Medium (1.0) -- Xbox Addict

Low (0.75) -- 360 Gamer Magazine UK
Low (0.75) -- 3DJuegos
Low (0.75) -- Ace Gamez
Low (0.75) -- Atomic Gamer
Low (0.75) -- BigPond GameArena
Low (0.75) -- Console Monster
Low (0.75) -- Deeko
Low (0.75) -- Eurogamer Portugal
Low (0.75) -- Game Focus
Low (0.75) -- Gameplanet
Low (0.75) -- Gamer Limit
Low (0.75) -- Gamer.nl
Low (0.75) -- Games Radar (in-house)
Low (0.75) -- Games TM
Low (0.75) -- Gamestyle
Low (0.75) -- GameZone
Low (0.75) -- Gaming Excellence
Low (0.75) -- Gaming Trend
Low (0.75) -- Impulse gamer
Low (0.75) -- Kombo
Low (0.75) -- MEGamers
Low (0.75) -- Metro Game Central
Low (0.75) -- MS Xbox World
Low (0.75) -- NTSC-uk
Low (0.75) -- PS Focus
Low (0.75) -- PSW Magazine UK
Low (0.75) -- Video Game Talk
Low (0.75) -- VideoGamer

Lower (0.5) -- Armchair Empire
Lower (0.5) -- Cheat Code Central
Lower (0.5) -- Game Over Online
Lower (0.5) -- Game Positive
Lower (0.5) -- Gamer's Hell
Lower (0.5) -- Gamereactor Sweden
Lower (0.5) -- Gamers.at
Lower (0.5) -- Giant Bomb
Lower (0.5) -- PS3bloggen.se
Lower (0.5) -- RPGamer
Lower (0.5) -- Vandal Online

Lowest (0.25) -- 9Lives
Lowest (0.25) -- Boomtown
Lowest (0.25) -- Computer Games Online RO
Lowest (0.25) -- GamerNode
Lowest (0.25) -- GamingXP
Lowest (0.25) -- IC-Games
Lowest (0.25) -- Insidegamer.nl
Lowest (0.25) -- Jolt Online Gaming
Lowest (0.25) -- Kikizo
Lowest (0.25) -- LEVEL
Lowest (0.25) -- Meritstation
Lowest (0.25) -- My Gamer
Lowest (0.25) -- Official PlayStation 2 Magazine UK
Lowest (0.25) -- Play UK
Lowest (0.25) -- WHAM! Gaming

lol

[Abelian75]

This is just really dumb.  They literally say in the article that their findings were "almost entirely accurate".  That means their findings are wrong.  If your algorithm does not match theirs 100%, then it isn't their algorithm (in fact, even if it DOES, it isn't necessarily their algorithm, but it at least might be, and given enough data you can reasonably assume that it is).

I'd love to crucify metacritic over something stupid that they are doing, but this isn't it, sadly.

[Rendakor]

Even if his algorithm and thus that list isn't accurate, the fact that they weight scores at all is fucking stupid.

[schild]

I can not even begin to describe how hilariously bad this weighting is, whether or not its even related to them sponsoring the site. God. Just. So hilarious.

Even if his algorithm and thus that list isn't accurate, the fact that they weight scores at all is fucking stupid.

Counterpoint: I disagree.

[Abelian75]

Even if his algorithm and thus that list isn't accurate, the fact that they weight scores at all is fucking stupid.

Oh, I don't disagree.  But I mean... they're clearly trying to be like "LOOK AT THEIR DUMB RANKINGS LOL" while downplaying the fact that they failed to find the algorithm, and therefore those are not their rankings.

[schild]

While the actual weighting may be a little more granular, I definitely enjoy living in a world where IGN is important enough to warrant a top tier position.

Haha, just kidding. Metacritic is the worst thing in the media and I fucking hate them and want to see it dismantled.

[Venkman]

Edit: eh, fuck it. Metacritic has always been weighted, they've said so in the past, and companies still rely on it. Not much else to say.

[schild]

Wall Street's view of gaming (that they even have one, really, given how insanely volatile the industry has always been) is hilarious and misguided.

The word of the day is apparently hilarious.

[Phildo]

They lost me at Full Sail students.

[Margalis]

The whole idea of weighting runs completely counter to the idea of using an aggregate score and relying on the wisdom of the masses.

Once you weight one outlet above another you are basically stating that their reviews are more reliable - that they more closely match your own internal subjective ranking of the games in question. If you have your own subjective score in mind you might as well just print that and leave the psuedo-scientific rationale out entirely.

This is the problem with stuff like the BCS. They tweak the BCS algorithm to be better - where "better" means "ranks teams in the order we think they should be ranked." But if you know how teams should be ranked why do you need an algorithm?

[schild]

This is the sort of system that should honestly be somewhat transparent anyway.

[Abelian75]

The whole idea of weighting runs completely counter to the idea of using an aggregate score and relying on the wisdom of the masses.

Once you weight one outlet above another you are basically stating that their reviews are more reliable - that they more closely match your own internal subjective ranking of the games in question. If you have your own subjective score in mind you might as well just print that and leave the psuedo-scientific rationale out entirely.

This is the problem with stuff like the BCS. They tweak the BCS algorithm to be better - where "better" means "ranks teams in the order we think they should be ranked." But if you know how teams should be ranked why do you need an algorithm?

Yeah, and that's pretty much why it should be transparent, imho.  It's so easy to convince oneself that one knows for sure what the result should be, and therefore believe that your tweaking of the algorithm is the One True Thing to do.  But without transparency there's nobody to double-check your work.

I imagine they are worried about people gaming the system?  Or just stealing their precious algorithm (unsure why, given that anyone can steal the result of their algorithm anyway)?  In any case, maybe the system SHOULD be open to gaming, and it should be obvious that it is occurring, so that people trust it less (i.e. trust it more similarly to the degree it should actually be trusted)

[UnSub]

Metacritc have responded to say that they disagree with the findings.

I'd really have to see full paper to get a better view of what they are talking about when they say 'weighting', because what those figures suggest (to me) is that if IGN gave a title 100% as score, Metacritic would turn that into 150% for overall analysis purposes.

[Fabricated]

Destroy all gaming media forever.

[Venkman]

This is the sort of system that should honestly be somewhat transparent anyway.

I'm not sure such a system could be devised. This is a tail wagging the dog. Reviews are subjective by nature. Did movie reviews used to have a big impact on movies in, say, the 70s or 80s? I wasn't paying attention, so asking honestly.

I'm actually curious how after MC has been around for so long there's still so many places that do reviews and that people still subscribe to an individual site's review without just going to MC. Years ago I thought MC would go away because they made individual reviews irrelevant, which in turn would make MC impossible.

Or maybe that was just a dream.

[Paelos]

Destroy all gaming media forever.

I would be fine if regular business media covered it. The business journals aren't going to care about getting free copies of shit and swag.

[Abelian75]

Metacritc have responded to say that they disagree with the findings.

I'd really have to see full paper to get a better view of what they are talking about when they say 'weighting', because what those figures suggest (to me) is that if IGN gave a title 100% as score, Metacritic would turn that into 150% for overall analysis purposes.

It's more like they'd make it so that if IGN gave a title 100%, they'd treat it as one and a half review sites giving it a 100%.  (Though, again, it appears that the weights listed by the Fullsail guys are probably a bunch of bullshit)

[Margalis]

Metacritc have responded to say that they disagree with the findings.

Of course they didn't say what their actual scheme is, nor does it really matter.

They've said they weight sites, and no matter what the numbers are or how it works it would probably look pretty silly if exposed.

[cironian]

With or without weighting, the rule of "garbage in, garbage out" applies pretty well to MC game scores. I mostly use it to quickly find a sampling of differently scored reviews and see what those who liked a game and those who hated it have to say in contrast.

[Ingmar]

At least Tom Chick isn't on there anymore.

[lamaros]

Metacritc have responded to say that they disagree with the findings.

I'd really have to see full paper to get a better view of what they are talking about when they say 'weighting', because what those figures suggest (to me) is that if IGN gave a title 100% as score, Metacritic would turn that into 150% for overall analysis purposes.

No. I expect it is more like if it has two reviews and one is weighted 1.5 and one is weighted .25 then the score would be 1.5/1.75 x score A + .25/1.75 x score b.

Make sense? Late night mobile phone math...

[UnSub]

Metacritc have responded to say that they disagree with the findings.

I'd really have to see full paper to get a better view of what they are talking about when they say 'weighting', because what those figures suggest (to me) is that if IGN gave a title 100% as score, Metacritic would turn that into 150% for overall analysis purposes.

No. I expect it is more like if it has two reviews and one is weighted 1.5 and one is weighted .25 then the score would be 1.5/1.75 x score A + .25/1.75 x score b.

Make sense? Late night mobile phone math...

Yes - I really should have finished my thoughts properly on that point. Originally I'd written something longer about how if two 1.5 weighted reviews were used to calculate a Metacritic score (and yes, they need at least 4, but as an example) then the extra weighting on each would cancel each other out so their real weighting would end up being 1 in those cases, and that giving something like IGN a 50% bonus to its review scores through weighting makes achieving a good IGN score much more important than impressing lower tier sites, but then I cut it back into the not-quite-correctly expressed sentence above.