Server Problems

[Paelos]

As some of you know, I'm on Proudmoore, which is one of the 20 release servers that seems to be in the crapper now. There may be more than that, I'm not sure. However, what I do know is that instead of getting progressively better, login issues are getting worse. Server lag and world latencies are getting bigger. Now, over the last week, the Eastern Kingdoms have been crashing with wild abandon after 5PM server time. The weekends are horrific. If you accidently DC out of a raid, good luck making it back in within the hour.

On Saturday, we tired to run MC, but 12 people sat at the instance loading screen for an hour. When we wiped in a bad place and tried to run back, we sat in the lava as ghosts waiting for the server to even recognize we wanted back in MC. It displayed it's affection for us by crashing the East and sending us back to the graveyard when we logged back in. Rinse, wash, repeat.

At this point there's nothing to be done except watch and wait I suppose, but I've made it a point to cancel my sub and note why on the forums. It will mean nothing in the long run, I suppose, but if enough people are willing to do the same and play on borrowed time to see if Blizzard actually wants to play ball, it might do more. So, if you're pissed off as well, I encourage you to do the same I guess. And if not, just vent here about how much the servers suck for kicks.

[Kail]

Ditto here, but on different servers.  This was my week off, and I could only get on for two days.  Tonight I couldn't even sign on to anything; it got stuck at "handshaking" every time I tried it for about three hours.

What drives me nuts is that at launch, I chose a server with a low population because I wanted to play on a low population server, but it got overcrowded, so I made some alts on a new server which ALSO got full and on and on until I rolled up some guys on the new servers that went up in January (like three, four weeks ago), sometime, I think, and I was getting a queue on THAT this week.  And yet I STILL have to wait a half hour (on a good day) to get into a battleground.  Grr.

[bhodi]

Same, on icecrown. I don't understand why they've been so bad and getting worse since character creation has been frozen for weeks now. It's the communication with the backend database and world-server communication causing major "loot lag" and extended delays logging into the game and zoning... it's even included eastern continent world server crashing multiple times during peak hours last weds, thurs, and friday. It's starting to get rediculous and frustrating. I mean, I love getting dumped into the ocean as much as the next guy, but can't they buy some hardware with those money hats of theirs?

[Calantus]

Same with Frostmourne (the first oceanic PVP server, Aussies ftw?), we had to move onyxia ontime DKP back 1/2 hour because people were falling off boats or floating above stonetalon in ghost boats. Weird shit.

[LadyGuardian]

It sounds really stupid, but when I find myself getting stuck at handshaking (Stormrage) I can actually get through by cancelling, putting in the wrong password intentionally, letting that come back as password incorrect, and then it flies through handshaking when I give the correct password right after. I read that random post on the official forums months ago and it hasn't failed me yet.

btw, hi. I've been lurking since this board was created and I always mean to actually contribute regularly. No time like the present.

[Paelos]

Hi! Contributing is good. Just stay out of politics or that awful "other" WoW thread.

[Fabricated]

I'm on Proudmoore as well, and the server is foobar'd something good. Well, depending on what you're doing.

Going into/out of instances and across continents is a complete crapshoot. Other than that it has been relatively playable for me, ignoring the occassional massive lag spike.

If there's anything good about our server being a piece of shit, people were frantically trying to get out of the EK at the same time me and a couple of my guildmates were trying to get "The Calling" done in Silithus. We basically offered to have our 'lock port people's friends and guildmates from the EK in exchange for helping us do the raid. It took us 40 or so minutes to get 10 or so people together, and after we started offering ports, we got to 40 people in less than 30 minutes.

Needless to say, my warrior got some new tanking gear.

[Merusk]

I haven't had any problems with crashing over on Alleria.  This is worrisome, though, since Alleria was one of the first 4 "Central" time-zone servers and had problems like what you're describing at launch.  We have had queues for a few months now, but no promises of a char-creation lock or anything else.. of course our queue is only 220-300ish so I imagine that's why.

 This weekends problems were 'just' a lot of loot lag and AH lag.  I did notice that bodies weren't popping-up in AV for about 10-20s after a 'kill' but I attributed that to my crappy ram, since my machine HATES that instance. (20s lockups ftl).  Perhaps it's the beginning of greater problems like you all are experiencing.

[Alkiera]

I tried to log in a couple times this weekend, and was unable to get thru.  Got disconnected or 'Unable to contact server' before it even got to realm selection.  I was planning on canceling this month anyway(queues, and too much time spent in MRLG)... but I'll be actually doing so shortly, assuming I can log into their website.

Alkiera

[Dren]

Our server has been stable, but still have queues each primetime.  The worst has been over 300.

We get loot lag and occasional lag spikes, but many times I think it is on my end since nobody around me at the time experiences themselves.

I do not raid nor do BG's, so I cannot comment on those.  As for constantly rotating alts and playing the AH market?  All is fine.

[Glazius]

I mean, I love getting dumped into the ocean as much as the next guy, but can't they buy some hardware with those money hats of theirs?

My guess is that Vivendi is bleeding them dry except for maintenance, to prop up other failed ventures elsewhere in the company.

Either that or CS soaks up way more money than anybody could imagine.

--GF

[cevik]

My guess is that Vivendi is bleeding them dry except for maintenance, to prop up other failed ventures elsewhere in the company.

Either that or CS soaks up way more money than anybody could imagine.

Or, more likely, the problem isn't something you can just throw extra hardware resources at and have it magically go away..

[ahoythematey]

It's not magic when money is spent to hire professionals.

[bhodi]

Or, more likely, the problem isn't something you can just throw extra hardware resources at and have it magically go away..

Normally, I would agree, however the fact that it isn't happening on all servers indicates the problem is due to that particular site's configuration or connection.

Furthermore, it's cllear to me that all of the servers with the issue are the initial servers running on older hardware. Since the code is the same across all realms, and the new realms don't have the issue when they're listed as 'full' with (presumably) generally the same amount of concurrent players, it has to be either the hardware or the colo connection. Since I don't know their specific hardware configuration, I can only make educated guesses based on the symptoms. It's clear most of the problems are caused by a database bottleneck, either on the database or world server side, from memory/cpu limits, disk i/o, or bandwidth bottlenecks. I'm fairly sure it's on the database becuase latency remains constant even with the loot lag. If the world servers were being overloaded, everyone's latency would increase as the world server queues up operation requests.

Blizzard blames it on the colos, of course, and I suppose it could be them. The problem to me seems more internal to the farms though and that's normally handled by blizzard's equipment. Thus, in this instance, I believe you CAN throw hardware at the problem. Either that or get a better colocation facility.

Note: this does NOT address the "authenticating" issue, which an entirely different server farm that ALL clients go through before they get transfered to their realm server. I am speaking of realm server issues such as loot lag, extended zone transfer time, frozen buttons, and client-server update issues.

[cevik]

Or, more likely, the problem isn't something you can just throw extra hardware resources at and have it magically go away..

Normally, I would agree, however the fact that it isn't happening on all servers indicates the problem is due to that particular site's configuration or connection.

Furthermore, it's cllear to me that all of the servers with the issue are the initial servers running on older hardware.

There are a myriad of differences between older established realms and newer realms that probably extend WAY beyond hardware (and in fact, probably have little or nothing at all to do with hardware because I bet they have standardized on a hardware platform and are likely not deviating from that standard).

Just one example (and the most likely culprit, as I doubt that the hardware differs and it clearly is an issue related to this system):  Database Size.

I bet that the original realms have several orders of magnitude more database entries than the more recent realms.

[Morfiend]

I bet that the original realms have several orders of magnitude more database entries than the more recent realms.

That is my guess as well. People in general like to horde stuff. Hell, I have a bank FULL of junk I will probably never use again, but I like to keep it "just in case".

[Yegolev]

DB issues were around from day one, so it's no surprise to see them now.  Let a DB churn for a while and it needs a reorg, which is time consuming and affects performance.  Also, shitty SQL can cause all sorts of fuckups.  I personally think that they must not be hiring more admins and operators while adding infrastructure, and at some point you are just going to be trying to keep the lights on instead of fixing shit.  I'd guess that they are spending their resources in adding new infrastructure rather than updating or replacing existing stuff, and I'd be surprised if they didn't have heterogeneous hardware, if only because any vendor is going to be pushing the next chunk of inventory he gets in.

This sort of thing is one reason I'm not playing WoW anymore (again!), besides EVE and other issues.  If you smell sulfur, it's because I have been tempted to try EQ2.

[bhodi]

Eh, maybe they have the same hardware. MAYBE. That's not a sure thing, since general lifecycle on hardware is 3 years, who knows how many development realms they had and how much hardware before they even went beta. It's possible so for this, I'll assume it's same hardware, since anyone with half a brain wants hardware standardization if at all possible. That makes it a colo or network issue.

Census says Icecrown's the 29th most populated server in the realm by 30 day activity, running at 22,000 players, and blizzard's already said they're implementing quotas on servers, and the quotas are rougly 5000 concurrent players at any given time. The difference between the top of the middle 1/3 and the bottom of the middle 2/3 is about 30%.

This is all beside the point, since it's clearly not population or server age. If it were, the top 30 realms on that list would be having these issues. They are the oldest, with the highest active population, the highest concurrent connections at peak time, the highest alt numbers and (by logic) the highest canceled accounts.

I don't see how anything but site-specific issues could be causing this problem since it's not affecting all older servers equally. They have the same database (by your standardization argument) and roughly the same user population and database activity. If it were the database or software in any way, all the top servers would be at their knees from 7pm-9pm.

[Trippy]

I don't know if this is still the case but WoW's servers used to be "clustered" in the sense that groups of servers would all share the same DB infrastructure. That would mean that even if your server didn't have a lot of players on at the time you could still experience DB lag issues (AH, mail, looting slowdowns, etc.) if there were very high population servers in your same cluster. So if WoW still has the same shared-DB intrastructure setup, looking at your server's population to figure out why you are getting DB lag is meaningless unless you know the cluster setups.

[bhodi]

I don't know if this is still the case but WoW's servers used to be "clustered" in the sense that groups of servers would all share the same DB infrastructure. That would mean that even if your server didn't have a lot of players on at the time you could still experience DB lag issues (AH, mail, looting slowdowns, etc.) if there were very high population servers in your same cluster. So if WoW still has the same shared-DB intrastructure setup, looking at your server's population to figure out why you are getting DB lag is meaningless unless you know the cluster setups.

True, but unless they've moved the servers around, it stands to reason that the server clusters that have been there the longest are going to have the highest population, on all the servers. They were the 'launch servers'. Though you can't tell which servers are together, and from what I remember the servers are in quads, it shouldn't matter.

[Xanthippe]

I play on Shattered Hand, same lag problems.  And yes, it's gotten way worse lately.

[Phred]

I play on Cenarius, one of the terrible 20, and it's been completely unplayable the last 3 days. We've had to cancel raids because we couldn't zone.

[heck]

It sounds really stupid, but when I find myself getting stuck at handshaking (Stormrage) I can actually get through by cancelling, putting in the wrong password intentionally, letting that come back as password incorrect, and then it flies through handshaking when I give the correct password right after. I read that random post on the official forums months ago and it hasn't failed me yet.

omg you hax

Kargath and Frostwolf have the same problems.  Loot lag, getting pushed off the zeps/boats, npc conversation lag, turning in quest lag, and the famous handshake lag.  People in the party going LD is getting more and more frequent.  The depressing thing is that the most common response in guild chat, game channels and voice chat is that Blizzard has no practical reason to fix any of this.  The consensus is that they aren't losing money over it, so nobody sees it as worthwile to complain. 

[Kail]

People in the party going LD is getting more and more frequent.

Yeah, I tried to do Wailing Caverns today.  Ugh.  Went in with a party of five, killed the first boss, two of our party went link dead, killed the second boss, and a third goes link dead, leaving just me (a priest) and a mage to do the remaining like, three-quarters of the dungeon.

So, I log off, do other stuff for a while, log back on, and get stuck at the "retrieving character information..." screen for a while.  I hit cancel, it boots me back out to the auto-suggest realm thingy, so I figure, fine, whatever.  It suggests some wierd realm I've never heard of, I hit "okay" to prepare to roll up my nine millionth alt, and it puts me into the server queue... even though the server is listed as low population.  Huh?

[bhodi]

la la la, it was hardware, probably network.

We wanted to let you all know the steps we are taking to fix the problems that have recently affected the realms listed below. These performance issues include: corpse retrieval, loot lag, and combat lag.

We have been working with our hardware provider to locate the source of these issues for nearly a week. Yesterday we were able to isolate the problem to a specific part of our system. We immediately took action and are currently in the process of obtaining and setting up the hardware necessary to replace the faulty system. Please note that this is a significant hardware replacement -- it will take some time to prepare the new hardware even with our team working around the clock.

We sincerely apologize for these performance issues and will provide you with a more detailed timeline as soon as more information becomes available.

Blackrock
Cenarius
Deathwing
Doomhammer
Frostwolf
Hyjal
Icecrown
Kel'Thuzad
Kil'Jaeden
Kilrogg
Lightbringer
Ner'zhul
Proudmoore
Silver Hand
Tichondrius
Uther

Over the past 72 hours the following realms have been impacted by several infrastructure failures. The result of these failures has caused many of the performance issues experienced by these realms during this time period. Technical staff have been working non stop to correctly identify and resolve the problems. Many of the required tasks are complete; however, work continues at this time which may cause some additional interruptions. We will do everything in our power to provide as smooth as game play as possible.

We thank those affected for their patience and understanding in this matter.

(snip - same list as above)

[Azazel]

[There are a myriad of differences between older established realms and newer realms that probably extend WAY beyond hardware (and in fact, probably have little or nothing at all to do with hardware because I bet they have standardized on a hardware platform and are likely not deviating from that standard).

A year is a long long time in terms of computer hardware (or any technology). I can't see any way that they'd be using the exact same models of everything, 14-15 months on.

[Tale]

I'm on Proudmoore too. However we've cleared MC and killed Onyxia this week with no trouble. The lag hasn't really seemed a problem inside instances, only outside. But we are an Aussie guild, so not raiding in US peak hours, though we've still had severe lag in Ironforge and outdoor zones. I did play during daytime last weekend in Azshara and there was atrocious lag, but again it seemed to disappear when I went into an instance.

[Mesozoic]

All of Kalimdor crashed the other night on Thorium Brotherhood (new RP server), so its not just the original servers.

[bhodi]

A year is a long long time in terms of computer hardware (or any technology). I can't see any way that they'd be using the exact same models of everything, 14-15 months on.

At my job, I'd fight to the death for a 3-5 year tech refresh cycle. 15-15 month cycle is retarded to try and keep up with... you end up having to administrate half a dozen different models of servers, keep spare parts for each on hand, tune and test every operating system and patch on every model you have... and if you run something like windows, you get to create ghost images for every platform... no thanks.

At my previous job, we actually had an agreement with sun to continue to produce and support our Netra T-1125s WAY past official EOL, becuase we bought so many of them.

There's a lot to be said for standardization above all else; you may waste some CPU cycles on a machine that's too powerful for what is running on it or have to split slower applications out on more than one box, but you save it in the long run on maintenence, reliablility, performance tuning, purchasing, and projections.

[Azazel]

The thing to bear in mind is the unexpected popularity of WoW that took Blizz by surprise, followed by the half-assed method they've added servers since then.

When they added the recent "Oceana-preferred" servers about 2-3 months ago, they instantly had server load worse than Proudmoore (a fubared-since-launch, launch server). The hardware they bought for those O-P servers recently would have to be different to the stuff they had set up in place last, say, September ready for launch, simply because you just can't buy the old/identical stuff anymore.

So my thoughts at least (worth fuck-all, I know) is that they need to pull their fists out of their arseholes and buy a shitton of new hardware, then roll the servers offline for a day or two, or however long is needed to upgrade the hardware, add a day or however long to players' subs (like they used to do) and get the shit fixed and done and fucking over finally.

But yeah, they're making enough money to not really need to care, and it seems, that's exactly what they're doing.

[Trippy]

The thing to bear in mind is the unexpected popularity of WoW that took Blizz by surprise, followed by the half-assed method they've added servers since then.

The problems they are having now with their servers is the exact same set of problems they were having in *Beta*. The server architecture design of WoW has been fucked up from the very beginning and throwing hardware at the problem is the only way they know how to temporarily fix it.

[Paelos]

We ran MC tonight, and things were much improved. I'm hoping that carries into the weekend.

[Azazel]

The weekends on PM are nearly always unplayable. I hope this weekend is different, but I wouldn't count on it...