European Release: Chaos ensues

[Jeff Kelly]

Well it seems like Blizzard's success story continues. World of Warcraft was released in Europe today and sales are record-breaking.

Most online-retailers like Amazon are completely out of stock and have been for nearly two weeks now. Official sale in Germany started at around 8:30 today (two hours ago) with the occasional midnight sale to celebrate the occasion and most retailers have already sold through most if not all of their stock. Big retail outlets are swamped with people who want to get a copy of WoW and even fights over copies broke out on several occasions. To paraphrase a Media Markt (Big electronics chain in Germany) official "a bunch of orcs would have behaved better than those people".

I will post some links to stories as soon as I can find english sources. I am too damn lazy to translate german articles just to post them here.

Logged into WoW before I left for work (around eight o'clock this morning) and most servers were already at medium population with most french servers already at "high".

So I am positive that WoW will break the one million mark in boxes sold during the next few weeks. I think that WoW will sell between 100,000 and 200,000 Copies in germany alone. Amazon got 30.000 Copies and has already sold all of them so I think that 200,000 is a realistic estimate. This would put WoW in the same ballpark as GTA San Andreas or Half-Life 2.

Jeff

[Jeff Kelly]

Update: As of 13:00 nearly all european servers have hit the high population mark. Blizzard better get some more servers on or some people (including me) will really be pissed off when they are nota bel to log in tonight

Do they ever learn?

Jeff

[Signe]

I was reading a bit about it last night.  There are enough threads in English that I didn't have to run entire sites through Babelfish.  The servers were going up and down.  The first crash came withing 2 minutes of launch, no?  The Scandinavians threatened to invade, the Germans complained and then apologised for complaining, the French asked for their money back and the English whined like little girls.  Loads more little girls than anything else.  It was amusing. 

The servers have the same names as the US ones, don't they?  That should be confusing for common boards, like IGN. 

And no... no one ever learns anything.

[Toast]

Learning what you need to do is different than learning how to do it...

Interesting stuff from the Europe launch. It's a shame that Blizzard made its capacity issues worse before fixing some of the issues in North America first.

[Jayce]

Now it wouldn't be fair to make Americans go through days of waiting in queues just to have the server crash when you finally get in, and Europeans not have to go through that, now would it?

[Jeff Kelly]

Update 2: Now all european servers have server queues with waiting times in excess of one hour. Server population is high for all servers except the occasional PvE or RP Server. Account creation in germany is offline due to the onslaught. Blizzard is "working on it" (to paraphrase the german community manager) and they consider adding servers to alleviate the load. I am having serious deja vues at the moment.

The flaming on the official message boards was fun until the login server gave out and died.  :-D

So people had to move to different message boards to flame blizzard staff.

Same ol', same ol'.

I think it's rather funny.

Jeff

[Rodent]

Are we playing the same game? Apart from the signup server not going online until 1 am, I've had zero complaints. Good latency ( between 50-100 on a high population server ), no problems getting in on the server either.

[Signe]

Evidently, they've taken European account creation offline for everyone.  If you're lucky enough to have created an account early, you can play.  Well... if you are patient enough to queue up for a bit.

[Trippy]

I wonder how the Korean launch went, in comparison.

[ahoythematey]

Probably something like this, stuttering framerate included.

[Abel]

Picked my copy up in the shop somewhat past noon.

Wrestled my way through account registration (admittedly took like 45 minutes).

Picked a server and have been playing happely since then. No serious lag or any other issues.
However all those who tried to get in a couple of hours where out of luck ...

2 rules of thumb for MMORPG releases :
- register during off-time
- don't pick the already overpopulated servers

[Righ]

Probably something like this, stuttering framerate included.

That was hysterical. They LARP'd the looting lag.

[Signe]

My neighours poodle used to do that. 

[Stephen Zepp]

Something I just don't get, given the fully expected issues regarding release that have been demonstrated by no less than 5 previous MMOGs:

Why in the hell don't publishers send out their product at least a week early, let the users install, create their user accounts, and explore a trainer/demo mission offline prior to activating the server architecture?

You could even let the players finish off the "learner levels" of whatever version you have in the game, and port their characters (with of course highly accurate hack checks) to the main server structure. This would help to alleviate the "everyone is level 1" syndrome by using the expected behaviour that not everyone will max out the starter level limit in the "pre-live" week.

They use pre-ordering to handle the logistics issues (or at least try to) for getting the game into people's physical hands...it's just not that much of a leap to use a similar paradigm for server launch.

[ahoythematey]

Because a lot of these MMO's have stinky cheese-eating frogs as their publishing overlords who are only interested in lining their fucking pockets, not making an enjoyable experience for their customers.  That being said, I didn't experience anything problematic with the US launch, but it's current state is doing wonders to my normal rate of alcohol intake.

I hate the french, and I am also starting to hate the word paradigm, consider how often I am seeing it thrown around.

[jpark]

If SOE was wise, they would outsource some of their EQ2 servers to WoW :)

[Jayce]

(with of course highly accurate hack checks) .

No such beast.

Because a lot of these MMO's have stinky cheese-eating frogs as their publishing overlords who are only interested in lining their fucking pockets, not making an enjoyable experience for their customers

True, because so many companies in other countries REALLY CARE DEEPLY about their customers, and profit is secondary.

... ?

[Stephen Zepp]

(with of course highly accurate hack checks) .

No such beast.

Back-checking an imported character for what should be a very small subset of your total character development functionality is one of the more trivial aspects of online game security. I'm not talking about tracking 100 levels of character development, I'm talking about basic checks on stats, experience vs level, skills/abilities, etc.

I'd venture to opine that 75% of security/game hack issues reside within failure to check for "illegally" modified game packets, and/or implementing functionality client side that is more correctly (from a security standpoint) implemented on the server side of the equation. Another 15 percent come from poor database design, and the remainder is "unexpected" utilization or incorrect implementation of game functionality.

[Jayce]

I'd venture to opine that 75% of security/game hack issues reside within failure to check for "illegally" modified game packets, and/or implementing functionality client side that is more correctly (from a security standpoint) implemented on the server side of the equation. Another 15 percent come from poor database design, and the remainder is "unexpected" utilization or incorrect implementation of game functionality.

And I'd venture to opine that you're talking out of your ass, without any related professional security experience to back it up.

[HRose]

Update: As of 13:00 nearly all european servers have hit the high population mark. Blizzard better get some more servers on or some people (including me) will really be pissed off when they are nota bel to log in tonight

Just so you know:
The servers will never be all "high". This isn't possible because the flags show a proportion, not the direct numbers of players logged. They are set so that there will always be high, medium and low servers, no matter of the situation.

[Stephen Zepp]

I'd venture to opine that 75% of security/game hack issues reside within failure to check for "illegally" modified game packets, and/or implementing functionality client side that is more correctly (from a security standpoint) implemented on the server side of the equation. Another 15 percent come from poor database design, and the remainder is "unexpected" utilization or incorrect implementation of game functionality.

And I'd venture to opine that you're talking out of your ass, without any related professional security experience to back it up.

And in that opinion, you not only would be wrong, but completely wrong. Good try however.

You seem to be representing yourself as knowing something about MMOG security as well, so care to share what you think are the vulnerability risk areas?

[Righ]

The most significant security flaws in online games are introduced in the design phase. You already introduced them in your design by, as you say, "implementing functionality client side that is more correctly (from a security standpoint) implemented on the server side".

[Calantus]

I can't see how it would be difficult if you built the newbie area right. You would have a maximum amount of XP a char would be allowed to accumilate, so even if they hack their XP it doesn't matter. Then you limit the items that can be dropped, and the money and items that can be carried and there you have it... they can hack it all they want and it wont really matter. You just check every char to make sure they fall within the right categories, and ONLY check for certain things (IE. race, class, xp, money, and items... that way they can't cheat on stats) and make a character on the server by the information you gather.

[Stephen Zepp]

The most significant security flaws in online games are introduced in the design phase. You already introduced them in your design by, as you say, "implementing functionality client side that is more correctly (from a security standpoint) implemented on the server side".

Exactly. Just like the vast majority of MS products (and just plain computer products in general), security is "added on" as an afterthought, instead of being a fundamental design requirement. This is reflected in MMOG's as:
--speed hacks: (most of the time it's simply cranking up your packets sent to the server at a much higher rate than a "normal" player)
--ability hacks: many times, this is done by exploiting calculations that are performed on the client's exe, but should be done on the server.
--"super user" hacks: while also using social engineering and/or some form of password cracking that is external to the MMOG itself, the simple fact that many games (although not recently) allow for super user status on any client, instead of designing a non-shipped admin client, open up this vulnerability in the design phase.
--peer to peer denial of service: While this hack has mostly been designed out of more modern games, it was a huge flaw in many games that broadcast IP addresses of opponents to all players.
--trusted client: even when a game doesn't necessarily implement calculations client side, a lot of hacks fall through the cracks due to not properly validating client data sent to the server. This can allow for packet modification giving the hacker "out of bounds" skills, stats, abilities, or whatever applies for the game.
--server boundary hacks (also, trade hacks): a lot of these are a combination of design and database, but in general, objects are duplicated without the database being properly updated. For example, in Shadowbane, you could have someone kill you with millions of gold on your corpse, have them loot the corpse, but not accept the respawn. Since SB did daily server reboots, once you logged back in after the reboot, you had your gold back, since the last time the database was updated was before you died, not when your corpse was looted.

Jayce, if you can come up with an extended list of existing hacks or security flaws in various (commercial) MMOG's that demonstrate my percentage estimates were incorrect, I'd love to hear them.

[Jeff Kelly]

Situation is somewhat normal now. Account creation is online again and you can actually play on most servers, although server population on the pvp realms is still high.

Jeff

[Trippy]

Update: As of 13:00 nearly all european servers have hit the high population mark. Blizzard better get some more servers on or some people (including me) will really be pissed off when they are nota bel to log in tonight

Just so you know:
The servers will never be all "high". This isn't possible because the flags show a proportion, not the direct numbers of players logged. They are set so that there will always be high, medium and low servers, no matter of the situation.

Are you sure about that? That's not how the US servers have been working. I used to monitor the server populations during the day and then the evening and could watch as more and more servers hit "High" during prime time hours. According to your hypothesis that would never happen.

[HRose]

Just so you know:
The servers will never be all "high". This isn't possible because the flags show a proportion, not the direct numbers of players logged. They are set so that there will always be high, medium and low servers, no matter of the situation.

Are you sure about that? That's not how the US servers have been working. I used to monitor the server populations during the day and then the evening and could watch as more and more servers hit "High" during prime time hours. According to your hypothesis that would never happen.

Yes, I'm sure. I did many tests myself an the "flags" do not follow precises rules, they work on proportions.

I wrote more about this here.

As a simple test you can log in during the early morining and you'll see that many servers are still flagged "high", even if the population is nowhere high.

[Trippy]

Are you sure about that? That's not how the US servers have been working. I used to monitor the server populations during the day and then the evening and could watch as more and more servers hit "High" during prime time hours. According to your hypothesis that would never happen.

Yes, I'm sure. I did many tests myself an the "flags" do not follow precises rules, they work on proportions.

I wrote more about this here.

As a simple test you can log in during the early morining and you'll see that many servers are still flagged "high", even if the population is nowhere high.

Okay well right now (4:51pm Pacific) the breakdown is:

Low: 17
Medium: 49ish (they move around a lot)
High: 20

That's clearly not a 33/33/33 split. I'll check it again later to see how it changes.

[Jayce]

I'd venture to opine that 75% of security/game hack issues reside within failure to check for "illegally" modified game packets, and/or implementing functionality client side that is more correctly (from a security standpoint) implemented on the server side of the equation. Another 15 percent come from poor database design, and the remainder is "unexpected" utilization or incorrect implementation of game functionality.

And I'd venture to opine that you're talking out of your ass, without any related professional security experience to back it up.

And in that opinion, you not only would be wrong, but completely wrong. Good try however.

You seem to be representing yourself as knowing something about MMOG security as well, so care to share what you think are the vulnerability risk areas?

I suppose the thread is good and derailed, so I might as well.

Disclaimer: I'm no security expert - I'm a developer who (hopefully) knows what I need to to write code as securely as possible, and pays attention to the security world for that purpose.

- Illegally modified game packets:  If you are talking about Gear and its packet-shoving derivatives, that indeed was a problem for a lot of MMOGs at one point.  75% is pretty high though.  AC1 and UO are the only two that I know of that had major problems with them. Well, a bunch of FPSes too but I don't follow that world very closely.

-Implementing functionality client side : yeah , client in the hands of the enemy and all that.  I highly doubt any modern MMOG from UO onwards was/is dumb enough to fall into that one.  This is the canonical newb mistake, and none of the players in the mainstream MMOG space are exactly newbs to network gaming.

- Poor database design:  In a loose sense maybe.  I was about to say this is the most BS example you gave because database design has exactly zero to do with exploits, but I see in a later post you mean this to include dupe bugs.  That's not database design, that's the method and rules by which the info is persisted, which has nothing to do with the design of the data store itself.  The fact that you refer to it in such loose terms is what makes me think you don't actually have any real-world experience in this stuff.  If you do, fine, I could be wrong... tell us what these credentials are.

- unexpected utilization :  that's just a fancy way to say "exploit".  Tell us more, genius.

And no, I have no extended list of leet sploits for you.  I am not really sure why you ask.

I could go on, but this is really a topic for game design...  I now return you to your regularly scheduled thread hijacking....

[HRose]

Okay well right now (4:51pm Pacific) the breakdown is:

Low: 17
Medium: 49ish (they move around a lot)
High: 20

That's clearly not a 33/33/33 split. I'll check it again later to see how it changes.

I don't know the exact formula they use. But you can log in in the borderline server between low and medium during the peak and you'll have a number of player. Then you can log again in a server broderline between low and medium during off peak and you'll have a much lower number.

So I do not know how it works but there isn't a fixed number of players to make a server go from low to medium and then high.

The only direct fact is that above 3300 players you see the queues. That's the roof.

[Trippy]

Okay well right now (4:51pm Pacific) the breakdown is:

Low: 17
Medium: 49ish (they move around a lot)
High: 20

That's clearly not a 33/33/33 split. I'll check it again later to see how it changes.

I don't know the exact formula they use. But you can log in in the borderline server between low and medium during the peak and you'll have a number of player. Then you can log again in a server broderline between low and medium during off peak and you'll have a much lower number.

So I do not know how it works but there isn't a fixed number of players to make a server go from low to medium and then high.

The only direct fact is that above 3300 players you see the queues. That's the roof.

How are you checking server populations in game? Do you have some sort of add on? Remember that /who caps at 49 people per listing. So unless you are using something like the WoW Census tool, you aren't going to get very good results. It's possible that they are using something other than straight population numbers to determine the server population classification. E.g. they might be using some sort of server load measurement instead. Or they may be adjusting the scale as they upgrade the hardware. But in your original statement you were saying in effect that a server's population classification is relative to other servers rather than calculated independently and I don't see any proof of that.

As of 8:17pm Pacific the populations are at:
Low: 18
Medium: 40ish
High: 28

[HaemishM]

So not only did Blizzard not learn anything about releases, neither did Euro MMOG customers. BUY IN A FRENZY YAY YAY YAY! CAN'T LOGIN, WHAT A FUCKING SURPRISE! VIKLAS!

They have instancing. They could have designed newbie areas (or all zones for that matter) to instance if population exceeds some magical number. You know, like City of Heroes did. Or have beta customers or pre-orders given a 3-day "head start" period to play, so that they can login and advance beyond the crush of stupid.

Seriously, the lessons are out there, if anyone gave a fuck enough to listen.

[Rasix]

The instancing doesn't work that way in WoW and the newbie lands are situated in the middle of the (mostly) seamless world.   Now, this solution would work for EQ2, since technically, like CoH, the entire game is instanced.  I don't want this in WoW.  I don't want to run to defend Tauren Mills, only to find out that instance 1 is being attacked, not 2.  Once you open up this quick fix, instancing,  band-aid solution there's not an easy way to close it.

The glut and lag will be there until people separate through natural leveling.  This will be present in any game that does not have an instances main world and enjoys any sort of mediocre success. 

The lesson I think here would be to stagger the release.  But yah, I don't think Blizzard cares.  They'd proven without a doubt that people will put up with just about anything and still come back screaming for seconds. 

[WindupAtheist]

-Implementing functionality client side : yeah , client in the hands of the enemy and all that.  I highly doubt any modern MMOG from UO onwards was/is dumb enough to fall into that one.

Shadowbane.   :-D