Woah.

[Rasix]

My server just went to queues.  Eeep.  This is a small backwater server too.  Either something's really wrong or the warriors are mad again.

[Trippy]

Almost all servers are like that right now (High, Medium, Low pop, doesn't matter) -- well at least the ones that actually show up on the server list (a whole bunch of servers are still down right now). Blizzard has apparently run out of chewing gum and bailing wire -- things are seriously FUBAR'd over there.

Edit: Fixed typo

[Raging Turtle]

Saw a thread on the general boards saying that it was possible for ANY character to make a new toon with ANY name, even if it already existed - thereby locking out the original players with that name.  Many other players tried and confirmed this, or at least said they could in the thread.  There was a blue response buried deep somewhere, but the thread was deleted before I saw what it said.

If its true, would be a pain in the ass for Blizzard to delete all the same-name characters, and figure out which was the original.

I'm at work, so I can't tell... are any servers up at all right now?

[Venkman]

Yea, everything is queued. The WoW forums have been a special kind of screwed up for the better part of the evening too, probably from a zillion people in Queue wanting to vent while they wait.

I imagine the servers are Queued because they patched today and want to try and keep the load relatively light. They mentioned they were going to be replacing some hardware on what amounts to the Eastern/Central servers (they don't call them that anymore of course, but the hardware is still there even though the time zone tags were removed).

Who knows.

[Morfiend]

A whole bunch of servers are down, so the population of the downed servers are all going to other servers.

[Paelos]

If anything will slay this giant, it will be something like this happening on a repeated timeline that eventually grinds down the playerbase.

[XMackenzie]

This is really messed up.  My server problems seem to have been increasing recently - with nightly crashes / disconnects etc. Now this little 17+ hour downtime better fix those problems .... or Umm.  Darn, it's a crying shame the game is so much fun to play when it works.  Kinda hard to threaten quitting when ya know ya won't.

[Shockeye]

I found this funny.

1. Accounts/billing section down.. Very clever | 3/29/2005 10:31:41 PM PST           
 
Well atleast we can't call the folks at blizzard stupid. The accounts/billing section is down at a time when most people would cancel.

Very smart.

[schild]

I don't want to say I told you so. So I won't.

If you all had PSPs you could play Lumines while waiting in queue. :-D

[Jayce]

I logged on to random_server_01 last night since my server was down.  I had /leave 1 typed out and almost hit enter, but I decided to stick around and see how bad it was.

There was the expected drivel, but surprisingly, there were a few people who were actually half lucid.  All in all, it was an interesting experience.

Kinda hard to threaten quitting when ya know ya won't.

Finally, someone around here who puts aside hyperbole for the truth, no matter how sad it is.

[schild]

People who threaten to quit don't. You never hear from the people that actually quit unless they talk as loudly as I do. That's not often and never happens in official forums. Turnover is near impossible to actually calculate without working the actual server statistics. In other words, 400 people might say THEY'LL QUIT IF THEY DON'T GET COMPENSATION and 100 will, but they won't have posted about it.

[Sky]

ER was down late last night (this morning?), I just popped onto Scarlet Crusade and played my alliance alts. Gotta have your backup plans. There was a queue of around 200 people, but it was maybe a minute long, no big deal. I get a chuckle out of how worked up people get because of stuff like this, I guess some people don't follow the 'shit happens' philosophy of life (a sure way to go nuts imo).

[Xilren's Twin]

Saw the carnage and didn't even bother trying to play last night, but I was able to hop on ER this morning before work for about an hour and there were no queues or problems.  Course you wouldn't really expect that at 7am est. They still have the warning up about duplicate names; if you get stuck you have to log in as another character and page a GM.  Good luck with that...

The continuing issues with technical problems make me wonder if this is a built in flaw in the way their infrustructure is designed, or it is just too hard to simulate the traffic they have gotten by have way more players than expected?

Xilren

[WayAbvPar]

Good Christ- now I am almost glad I spent last night doing my taxes instead of playing WoW. BTW, I fucking hate the IRS.

[MrHat]

Good Christ- now I am almost glad I spent last night doing my taxes instead of playing WoW. BTW, I fucking hate the IRS.

I'm going to second that IRS thing.  Try owning a small business.  Jesus fucking christ.  I do love H&R though, I took my papers, they did my taxes and charged 300 bucks for an hours work.  Wait, what color was sarcasm suppose to be in?

If you all had PSPs you could play Lumines while waiting in queue. grin

Waiting in queue, waiting to get a raid, waiting to be rezed after a wipe, waiting on a bird, I love my psp.  Still using the charge from last week too.

[Rasix]

At least when I logged in later at night, the queue was a whopping 4 people.

And during the earlier queue time, which was 10 minutes, I talked with my wife about our vacation in September (bitching here at up at least 2 minutes too!) That worked rather well.   There's also these things called books.  I hear they work well and don't require charging. 

Keep bringing up PSP and Lumines in threads that have nothing to do with it please.  My wife just offered to buy me any hand held or other system for our first anniversary. Don't make me get something other than the PSP just to spite you, schild.

Edit: I don't know how those of you that were on overcrowded servers could put up with this though on a continued basis.  Never mind crap about disconnecting and then being unable to rejoin your friends or a raid.   I think I would have quit or moved servers.  Hell, I know I would have least moved servers (easy with a 3 person group though). 

[Phred]

The continuing issues with technical problems make me wonder if this is a built in flaw in the way their infrustructure is designed, or it is just too hard to simulate the traffic they have gotten by have way more players than expected?

Xilren

I think you may be right. They have a serious database queing problem and it overloads easily, and has since beta. They appear to be trying to fix it by throwing more hardware at it but it hasn't helped from what I can see. I wonder if they really did use Oracle for the database as someone speculated. Oracle just isn't good at fast connection handling and that's sort of what I see happening here. As I said before on the old boards, we ran into the same problem trying to use Oracle as the back end for an ISP login database and ended up having to build a intermediary demon that handled queuing up transactions to Oracle to get the performance we needed out of it. Oracle falls over way too fast when it gets a ton of quick connects, updates and disconnects. It's fast enough once the connection is established though.

[MrHat]

Due to the increase in downtime caused by today’s hardware maintenance, we will be giving all players with characters on the realms listed below a one-day time extension. In addition, players on the realms below will also receive a rested experience bonus, whether or not they had left their characters logged out at an Inn.

We realize that this extended downtime is very inconvenient, but it is a necessary step for us to take in our efforts to increase performance and stability on these realms. For the last few weeks, these realms have experienced poor stability during peak times, which resulted in NPCs not spawning, lag, and frequent disconnects. We have determined that a large percentage of realms on the same backend infrastructure were operating at maximum capacity. This overpopulation caused problems with the service that ultimately led to the issues mentioned above.

Our first step in trying to alleviate the situation was to offer character transfers to reduce the amount of players on these high-population realms. The character transfers did reduce the total amount of players on the shared infrastructure, and we noticed an immediate performance increase. However, this increase only lasted a few days, as populations continued to rise.

Once it was shown that the character transfers did not sufficiently disperse the population, we began to modify the hardware backend for the affected realms. The most recent and substantial step of that process involved transferring the realm data to a new hardware setup. The goal of this procedure is to decrease the overall load on the infrastructure that the affected realms share.

The new hardware infrastructure was completed yesterday, and we are currently copying the character data from eight of these realms onto the new hardware. This transfer will significantly decrease the load on the current infrastructure, which should result in improved performance on all of the realms listed below. The data transfer is taking us longer than expected, but it is vital for providing stability to these realms.

We are totally committed to providing our players with the best play experience possible, and we will continue to streamline our code and add new hardware to our infrastructure until we accomplish that goal. We once again thank you for your patience during this time, and wish you continued good fortunes on your adventures in Azeroth!

Please note that this time extension discussed above is in addition to the two-day time extension announced yesterday that is being provided to players on all realms.

Sincerely,
Mike Morhaime
President, Blizzard Entertainment

Full rest exp is nice for alts.

[bhodikhan]

Of course they are also giving everyone "full XP Rest bonuses". However, that will require them taking the servers back DOWN on Thursday for three (3) hours!

Anyone want to bet on the actual duration?

[Jayce]

Oracle falls over way too fast when it gets a ton of quick connects, updates and disconnects. It's fast enough once the connection is established though.

And for the prices they charge Oracle should queue up the dishes and wash them, and fast, to boot.

[jpark]

I don't want to say I told you so. So I won't.

If you all had PSPs you could play Lumines while waiting in queue. :-D

Shoot the Vulture!

[AOFanboi]

And for the prices they charge Oracle should queue up the dishes and wash them, and fast, to boot.

Maybe they should do a Mythic and use MySQL instead. Could be hard if they have coded much into stored PL/SQL procedures and use functions and triggers and whatnot.

Oracle is an expensive piece of poo - as long as you don't have an equally expensive trained Oracle DBA on hand. And buy all the "extras" needed for load balancing and so on.

[Trippy]

The continuing issues with technical problems make me wonder if this is a built in flaw in the way their infrustructure is designed, or it is just too hard to simulate the traffic they have gotten by have way more players than expected?

I complained about the design of their login infrastructure back during Open Beta. Nothing I've seen since that time has changed my opinion of their network/server architecture. Yes the success of WoW is unprecendented at least as far as North American MMO releases go. However many of the problems they are having now were identified back in Beta (login server issues, mail/AH lag, loot lag, and so on) and now there are some new ones thrown in for good measure (disconnects/crashes on looting, rezzing, etc., etc.). Throwing more hardware at the problem has helped a bit but clearly there are fundamental problems with the design of the system. For example they still haven't fixed their character state synchronization problems which I also complained about right after the game was released. Just recently one of my clients crashed while trying to loot an object. When I finally managed to log back in I was standing way back at the previous zone I was in, though I was still carrying the items I had looted along the way. Somehow the server had lost track of my whereabouts for the previous 10 minutes or so.

[Trippy]

The continuing issues with technical problems make me wonder if this is a built in flaw in the way their infrustructure is designed, or it is just too hard to simulate the traffic they have gotten by have way more players than expected?

I think you may be right. They have a serious database queing problem and it overloads easily, and has since beta. They appear to be trying to fix it by throwing more hardware at it but it hasn't helped from what I can see. I wonder if they really did use Oracle for the database as someone speculated. Oracle just isn't good at fast connection handling and that's sort of what I see happening here. As I said before on the old boards, we ran into the same problem trying to use Oracle as the back end for an ISP login database and ended up having to build a intermediary demon that handled queuing up transactions to Oracle to get the performance we needed out of it. Oracle falls over way too fast when it gets a ton of quick connects, updates and disconnects. It's fast enough once the connection is established though.

Setting up and tearing down connections on databases like Oracle and DB2 is a relatively expensive operation. Hence all the enterprise-class software I've worked with that use databases like Oracle and DB2 as their primary data store use some sort of connection pooling system where there's a set of connections that stay open to the DB at all times and a separate process manages the allocation of those connections to the processes that need to talk to the database. It's possible, given the poor design of other aspects of WoW's infrastructre that Blizzard isn't using some sort of connection pooling system, but I would doubt it. On the other hand their DBA job position is still open so who knows? (Though to be fair, writing a connection pooling system isn't something DBAs typically do).

[Murgos]

My guess is that with a million and a half concurrant users they are running into issues of information flow that probably aren't fully understood.  Even if each 'server' was fully independent (which they aren't) with it's own resources there would still be issues with resource utilization and the handling of that much data.

I'm sure they are trying to be both cost-efficient and responsive but the two ideas are mostly mutually exclusive.  They could probably put themselves in debt up to thier eyeballs building infrastructure but that wouldn't be very smart from a business or engineering sense.  They are currently surfing that bubble between usable with periods of unusability and unusable with periods of usability; thier engineers are probably killing themselves trying to push that bubble more toward usuable without just going hog-wide with the purchase orders.

[Dren]

From what I read we get both rest exp AND 2 days off our bills, right?  RIGHT?

[Trippy]

From what I read we get both rest exp AND 2 days off our bills, right?  RIGHT?

That's correct. The servers were just down to add the rest experience to everybody. The two-day time extension was supposed to have been added to everybody's account but I haven't checked any of mine to see if that's true.

[WayAbvPar]

If a character was already at full rest, will they tack on more? I am guessing no, which means I should have played all of my characters last night... oh well.

[bhodikhan]

There was a CS post on the WOW boards that they would "top-off" all the characters to 1.5 levels of "rest bonus". That's 30 bubbles of XP. So if you had a char at full bonus then you wouldn't see any more.

[Dren]

Ah, that works for me anyway.  My main is 57 and was almost out of rest exp.  I actually come ahead on this.  However, I can see this as a kick in the pants if your lvl 60 and/or your alts all had full rest exp to begin with. 

Although, getting 2 days free is really probably good enough.  Painful? Yes.  Really out anything other than being forced to do something else? No.

[Sky]

Why don't you whiny twits have alts on another server? As soon as I saw all the problems with the 'dreaded 20', even though ER had been fine up to that point, I started my alliance characters on another server (Scarlet Crusade), to have some redundancy. Or play some guitar.

Rest exp was cool++. Popped me up to lvl 55.8 over the weekend. Not just the rest exp alone, of course, I went through a ton of quests in all the zones that recently opened up, and quest exp doesn't count toward the rest bonus.

And I wasn't even the slightest bit upset about the server being down a night, heh.

[kaid]

Tried to play wow last night with a friend on eonar a nice little medium pop server. His character had camped out in ironforge so he logs it in. He gets into the zone but no npc ever loads he runs around for about 10 minutes and nothing ever loads. So he figured meh just camp and come back. He also found out he could not camp so to try to log out and relog he Ctrl Alt Deleted. This led to some true weirdness. His character he had been logged in was still listed as in the game for 10 HOURS after he did this. The really weird thing is during this time he could log on his other characters fine and those characters could even send tells to the one that was hanging and not get a character is offline message.

Needless to say this freaky bug issue pretty much scotched our WoW night pretty good.


kaid

[Jayce]

Tried to play wow last night with a friend on eonar a nice little medium pop server. His character had camped out in ironforge so he logs it in. He gets into the zone but no npc ever loads he runs around for about 10 minutes and nothing ever loads. So he figured meh just camp and come back. He also found out he could not camp so to try to log out and relog he Ctrl Alt Deleted. This led to some true weirdness. His character he had been logged in was still listed as in the game for 10 HOURS after he did this. The really weird thing is during this time he could log on his other characters fine and those characters could even send tells to the one that was hanging and not get a character is offline message.

I have had that exact bug happen, but my character times out within a few minutes.  I'd say at hour 2 you should have paged a GM :D

[kaid]

We paged a gm after 30 minutes. They kicked him out of limbo but only after about 10 hours.


kaid

[Paelos]

I fell through the Eastern Plaguelands last night for first time. Funny part is that I got a screenshot of my dead spirit that I needed to rez before I could rez my corpse. And I couldn't rez my corpse because i was too far above whatever abyss it fell into when the ground gave way. GMs didn't answer while I was on. I just spirit rezzed and dinged 60. I also hit honored with the AG. Cheap potions are luv.