It's official, Blizzard is retarded.

[Lakov_Sanite]

Thorim is deceptive. They make you think the top is an aoe fest but you actually have to single target down most of those mobs and have people focusing on certain types. When we do it my spriest is actually the only one up there aoe'ing.

[Phred]

If SAP can do it, a game server can do it.

I don't believe SAP updates every client connected to it every few seconds.

Even Oracle falls over when the transactions per second go massive.

I totally agree with your points.  SAP is a badly-written application by the estimation of Many, but it does not update a client like a MMO.  I am interested in this topic because I'm now tasked with some performance analysis on SAP/Oracle animals and the WACKY things some people are doing to alleviate bottleneck... well it's just fascinating.  I'm of the opinion now that there might be (probably are) things that could be done to improve performance but maybe too unpalatable to implement.  Too bad I won't ever find out for real from a Blizz tech.

Apologies but I'm just somehow strangely drawn to this thread.

Having worked in the technical end of a Local ISP who was forced to update our login database to Oracle (buy out blues) I know one common solution is to put an intermediary demon between the database and the query. That way the demon can serialize the queries through an already up connection rather than reconnecting constantly. Setting up and tearing down connections to Oracle (and probably most databases)  for every seperate query, beats the crap out of them.

[Fabricated]

Pay some smart devs in castles made of gold to rewrite the backend to run on big iron, make a server farm that makes Google jealous. Print money.

Well, okay, that's impossible. It'd be cool though.

[dusematic]

No, it smells like some asshole just made a bad joke. 

Are you seriously trying to start some shit?  On a site that discusses MMOGs?  On a forum that discusses MMOGs?  In a thread that's discussing an MMOG?  By whining about how much you suddenly hate MMOGs, because you've seen the light, and now you must pull back the curtains for the rest of us?

Because yeah.  That guy?  That's the real asshole.  Now shoo.  Adults are talking.

There's plenty of people that hate MMO's on this site.  It's called being jaded.  Fuck off. 

[Paelos]

I had fun in Ulduar tonight wiping on the Council. Good fight, hard hitting boss, high dps needed, and some movement adjustments. It's something we have to learn and think about, as well as the 3 trash pulls on the way up being meaningful rather than 20 stupid ones. The design is solid here, and I'm in favor.

[apocrypha]

Quick point about the EVE comparisons - remember that large fleet battles in EVE have *always* been a major part of the game. Even if most players weren't engaged in them they've been used as a selling point forever and most of the omgdrama in EVEs politics comes from 0.0 alliance warfare, for whom large fleet fights are bread'n'butter.

Thus CCP have been trying to crack the 200+ person in one place issue for several years, and it's only relatively recently that they've managed to do it. For the majority of EVEs life the large fleet fights have been ultra-lagfests that people worked round anyway.

PvP in WoW has always been an afterthought and badly implemented, especially world PvP. Their focus has always been PvE and instances which don't have nearly the same technical problems as 200 people, all in different gear, having to be rendered at the same time in the same place in a non-instanced way. So Blizzard are relatively fresh to this problem. Yes, I'm sure they could fix it and as CCP show it can be done in some situations if you bash your head and your coders heads and a fuckton of hardware on the wall for long enough, but it ain't gonna happen overnight.

[Phred]

Quick point about the EVE comparisons - remember that large fleet battles in EVE have *always* been a major part of the game. Even if most players weren't engaged in them they've been used as a selling point forever and most of the omgdrama in EVEs politics comes from 0.0 alliance warfare, for whom large fleet fights are bread'n'butter.

Thus CCP have been trying to crack the 200+ person in one place issue for several years, and it's only relatively recently that they've managed to do it. For the majority of EVEs life the large fleet fights have been ultra-lagfests that people worked round anyway.

Does anyone know if CCP patented anything in their new stackless i/o design?

[Trippy]

EVE uses Stackless Python. Given the license, it's possible for CCP to have it's own private fork without need to release the source code (i.e. Python and Stackless Python do not use a GPL-style license) but looking at the Stackless Python page it looks like CCP is supporting the efforts of the Stackless Python group rather than doing their own thing.

[Murgos]

Ship activity in EVE is much more mathematically predictable than character movement in Warcraft. The ships have speed and momentum that determines their turning radii. Guns fire at fixed rates and have enormous ranges. On the macro level, it's all just plugging assumptions into a bunch of formulae. WoW lets character turn 360 degrees instantaneously without losing any speed, has a number of split-second attack rotations and some very tight range requirements.

EVE can fudge the numbers a bit and nobody notices. The more players there are, and the further their ships are from one another in a chaotic battle, the less anybody notices if something is slightly off. None of that works at melee range.

Not to derail this thread any further but you are completely wrong.

Acceleration, speed, range and angular velocity combined with the rotational speed (tracking) of the guns used all have an affect in Eve, and are obvious in small scale combat.  At a guess I would be pretty confident that the only factors WoW needs for combat are distance to target and facing at the time the attack is pressed.  A 'split second' turn is important to a human, to a computer?  Not so much.

It's all beside the point of whether or not large scale combat is *possible* though.

[Modern Angel]

Also limitless Z axis? Hi.

[Vash]

Do you know how long it would take to implement the hardware changes that would be necessary across the entire WoW server farm? This isn't just a 'hay guys lets put more memory in' sort of thing, this would be a massive amount of planning, a massive amount of hardware, and almost certainly would require redesigning the way they build the Northrend part of the server clusters, and thus rebuilding them. They can't just go wave a wand and fix it, I'd guess it would take 6 months to implement something like that, minimum. Maybe if everything is running on VMs they could do it faster than that, but I kind of doubt that's their setup.

I never said it would be cheap, easy, and happen in a day.  They've made major server hardware upgrades in the past so it's not exactly unprecedented, nor is it out of their budget.  It's not just the world servers getting slammed by WG either, instance servers are getting overloaded as well with such a large portion of the playerbase raiding on a regular basis.  

WoTLK has been out for 6 months now and these problems have been present/apparent since week 1.  If they had committed to making some upgrades right after it went live and they saw how big a problem it was on live servers, the upgrades would likely be in place by now or going in as we speak.  Heck, if they had a large enough beta population (didn't pay enough attention to it know how extensive the Wrath Beta was) they should have seen how big a problem WG was going to be for their servers before the expansion even launched.

[Soulflame]

Thorim is deceptive. They make you think the top is an aoe fest but you actually have to single target down most of those mobs and have people focusing on certain types. When we do it my spriest is actually the only one up there aoe'ing.

Yeah, we switched it up last night.  Ranged in tunnel, melee in arena, focus fire champions as soon as they spawn.  The arena was solid, but the tunnel then had healing issues (yeah, it was me  )  Once I got my shit together, it only took 3 actual tries on Thorim to down him.

Then we wiped a bunch on Freya because of DPS not understanding two things.
1.  When the tank says stop dpsing the lasher, he means right now.
2.  When the raid leader says stop dpsing the little adds, he means... oh crap, half the raid died.  Whoops.

[Merusk]

Thorim is probably my most favorite fight right now.  My guild's 7/14 but I haven't seen Freya or Iron Council myself since they did those on days I wasn't online.  Hodir is a headache and a half, if only because our healers are whiny bitches about having to move and our current MT doesn't quite get "move the guy so melee can stand in the lights."

[Lakov_Sanite]

I loved our freya attempt where the dps lead says "Ok snap lasher is a little high" and then every single dps turns and blows it up in two seconds leaving the other two completely untouched.

[Soulflame]

Hodir?  That fight isn't too bad.  Make people bring two pieces of FR gear, and it's a lot simpler.

The only thing I could wish for on that fight is to have the happy fires stand out a little more.  I rarely bother looking for them after the first freezing breath or whatever it is, because it's difficult (for me) to pick them out in the sheer chaos of 25 people running around, dodging icicles, and when I get a chance, healing the MT.

[Fabricated]

The problem we're having with hodir is that their computers aren't geared enough for the fight and they can't see the circles.

Also breaking the NPCs out of their ice when they get flash frozen again.

[Merusk]

Yeah, the mechanics aren't too tough but it's a combination of things that are making it such a headache.  I'm sure we've got guys with the same problem as Fabricated mentions, they just don't say anything and run with the group.  Doing it in 10 vs 25 probably helps us there.   We're just not pushing enough damage out because our two highest DPSers are me and a rogue.. and we never get to stand in the light at all during the fight. He's enraged on us twice when we were close to winning the two times I've gotten in with him.

[Fordel]

If WoW was as 'responsive' as EVE is in regards to combat, we would all have stopped playing WoW 10 mins after we tried it.



Press a button, one, two THEN it goes. Double click in space, one, two THEN you start to move. 'Good' fleet lag in EVE is a 5-10 second delay. Even in a system with only 4 other people and nothing happening, you'll have a 1-2 second delay in everything you do.

[Yegolev]

I know one common solution is to put an intermediary demon between the database and the query. That way the demon can serialize the queries through an already up connection rather than reconnecting constantly. Setting up and tearing down connections to Oracle (and probably most databases)  for every seperate query, beats the crap out of them.

That is also interesting.  I had assumed the issue with most MMOs was DB IO in the first place and this could be one of the problems.  Maybe it's a network bottleneck but my assumption is that there is a disk IO bottleneck, which can be tuned to varying degrees based on the OS and RDBMS.  Of course, could be CPU or RAM.  Can't see the servers or disk subsystems so... eh, I'm left to pick one of the items that cannot be easily fixed by overallocating hardware.

I'll also say that EVE is not terribly responsive but at the same time I don't believe the lag was ever caused by client updates.

[Phred]

That is also interesting.  I had assumed the issue with most MMOs was DB IO in the first place and this could be one of the problems.  Maybe it's a network bottleneck but my assumption is that there is a disk IO bottleneck, which can be tuned to varying degrees based on the OS and RDBMS.  Of course, could be CPU or RAM.  Can't see the servers or disk subsystems so... eh, I'm left to pick one of the items that cannot be easily fixed by overallocating hardware.

I think with the current market of fast disk solutions, any MMO with disk i/o problems must have been designed on the back of an envelope or something equally silly. It should be relatively easy to design in a disk i/o subsystem that won't bog down and is still quite cost effective.

[Hutch]

Hodir?  That fight isn't too bad.  Make people bring two pieces of FR gear, and it's a lot simpler.

The only thing I could wish for on that fight is to have the happy fires stand out a little more.  I rarely bother looking for them after the first freezing breath or whatever it is, because it's difficult (for me) to pick them out in the sheer chaos of 25 people running around, dodging icicles, and when I get a chance, healing the MT.

We put raid icons on the NPCs' heads. Want to find the happy fire? Look for the skull.

[Tarami]

I doubt much is done against the DB in real-time, especially writes as you have to perform exclusive locks in an environment as active as a game. Auction house and other changing, highly tabular data might be done real-time, but most data structures in an MMO aren't dynamic. It's simply wasteful to do DB lookups for things that don't change more often than with every patch.

My guess would be that nearly every data projection on the server-side is done through fairly specialized, cached memory structures that have no more look-up latency than simple memory addressing. These are populated on server boot-up and are never really changed, just referenced from a bazillion places that do change (like an instanced mob has a pointer to its mob class, rather than to an ID that it looks up in the DB.) The only significant drawback with this approach is memory comsumption and memory is relatively speaking very cheap and it scales very well.

Just to continue my delusional technical wankery, I would say that 200+ PvP sitations become unmanageable because they're exponential in nature. For every player you add to a small area, each of those players will consume more CPU time per update. Adding one player to a group of 150 might be like adding five players to a group of ten. It all depends on how much overhead there is on having players in eachothers' proximity, but there would be some overhead to it, regardless. Almost no other client-server applications are designed to allow all clients can communicate freely with all other connected clients via the server, but games are, for obvious reasons.

This is greatly simplified and it wouldn't scale this badly in practice, but just to illustrate - something like distance-checking AoE effects, with six people á two teams you have a potential 18 distance checks to do (each player AoEing and having to distance check three opponents), with 75 people per team you get a potential ~11,000 checks, just in case everyone decided to fire off an AoE at the same time.

[bhodi]

You are essentially correct in that you don't generally hit the database itself for local real time data. The slowness issues partly stem from the fact that updates must be handled sequentially and you run into serious causality issues if you try and sidestep this fact.

I believe one of the developers here, possibly mahrinskel or stephen zepp, enumerated it a year or two ago in a long post, about planetside if I recall correctly, that some creative searching might dig up. The example was about a grenade/rocket and destructible terrain and how two clients getting out of sync are a bad thing.

[pxib]

Acceleration, speed, range and angular velocity combined with the rotational speed (tracking) of the guns used all have an affect in Eve, and are obvious in small scale combat.  At a guess I would be pretty confident that the only factors WoW needs for combat are distance to target and facing at the time the attack is pressed.  A 'split second' turn is important to a human, to a computer?  Not so much.

It's all beside the point of whether or not large scale combat is *possible* though.

It's not about possibility, and I didn't mean to imply that the fixed mathematics of EVE's simulation are hand-waving. Quite the opposite. They help make largescale online battle possible. Battles in EVE take place not only at enormous range, but each gun has an enormous range of ranges. There are no substantial LOS issues. If you start shooting at the limits of a particular gun's specs you don't expect to hit very often because the math takes such things into account.

What gets fudged is your client's understanding of the exact locations of enemy ships. All that math happens server side and feeds you failures and successes. The difference between your understanding and the server's is minimally important because ship motion is extremely predictable and the statistical error bars are large. There's only very rarely a baffling player-not-in-range experience because distribution of hits and misses will be wrong in less obvious ways than...

...in the tighter ranged, walls and pillars world of Warcraft, where very little of this applies. Somebody steps around a corner and your spell fails, somebody moves in an unpredictable way and you get all sorts of "out of range" or "facing wrong" messages. Blizzard fudges ranges a little to hide latency, and that fudging is immediately obvious when some rogue is backstabbing you while you run.

EVE avoids much of the combinatorial explosion Tarami and bhodi (yeah, I remember that post was enlightening) are talking about because individual clients don't need to be particularly promptly informed of it.

[Fordel]

Just to continue my delusional technical wankery, I would say that 200+ PvP sitations become unmanageable because they're exponential in nature. For every player you add to a small area, each of those players will consume more CPU time per update. Adding one player to a group of 150 might be like adding five players to a group of ten. It all depends on how much overhead there is on having players in eachothers' proximity, but there would be some overhead to it, regardless. Almost no other client-server applications are designed to allow all clients can communicate freely with all other connected clients via the server, but games are, for obvious reasons.

There is evidence that this may be the case. Most of the WG changes before these current ones revolved around trying to spread everyone out across the zone, instead of just clustering at the keep.

[Lantyssa]

I believe one of the developers here, possibly mahrinskel or stephen zepp, enumerated it a year or two ago in a long post, about planetside if I recall correctly, that some creative searching might dig up. The example was about a grenade/rocket and destructible terrain and how two clients getting out of sync are a bad thing.

Try this thread..  Zepp and I (and maybe others, just skimmed enough to find it) are talking about why it isn't going to work.

[SurfD]

The problem we're having with hodir is that their computers aren't geared enough for the fight and they can't see the circles.

Also breaking the NPCs out of their ice when they get flash frozen again.

I believe there is a setting in the graphics options called "projected textures" or something similar.  Have the people who cant see circles turn this on.  Even when running with every other setting turned down to low, this should still allow you to see a big blue circle on the ground, without all the fancy particle / lighting effects from spells killing your framerate.   My videocard suck (i get about 5 fps in a 25 man raid on a good day) and projected textures has saved my ass a number of times.

[Merusk]

What surf said.  I learned about this on Friday and turned it on.. all of the sudden it didn't matter that I couldn't see the stupid blue light that indicates the snowfall.  Instead there's a horking big blue rune on the ground.   Why such a critical piece of visual information is defaulted to OFF I have no idea.

[kildorn]

The option was added last patch. We had FUN with that on heroics and the like. Randomly WoW seemed to default this setting off regardless of graphics settings. So one patch I can see void zones, the next patch I can't see my own team's freaking Consecrate or hostile flamestrikes. <3

[Nevermore]

Holy shit, is that why I suddenly couldn't see the fucking fire patches anymore?

[kildorn]

Holy shit, is that why I suddenly couldn't see the fucking fire patches anymore?

Yup. Worst Default Setting Ever.

[Koyasha]

Even worse than defaulting autoattack to A?   

[kildorn]

So much worse.

Though my current lark is needing to rebind Ctrl-S from enable/disable sound. Since I hit it a lot when using ctrl-click binds to heal and trying to move out of fires.

[Paelos]

I enable sound in raids at least once a week because it's my push to talk button.

[Ingmar]

Just to continue my delusional technical wankery, I would say that 200+ PvP sitations become unmanageable because they're exponential in nature. For every player you add to a small area, each of those players will consume more CPU time per update. Adding one player to a group of 150 might be like adding five players to a group of ten. It all depends on how much overhead there is on having players in eachothers' proximity, but there would be some overhead to it, regardless. Almost no other client-server applications are designed to allow all clients can communicate freely with all other connected clients via the server, but games are, for obvious reasons.

There are actually 2 lag issues with Wintergrasp; you're only describing one of them, the lag that happens just as a normal consequence of a bunch of people fighting all together. I expect you have nailed why that one happens pretty well. This doesn't explain the massive Northrend-wide lag that hits briefly when Wintergrasp ends, though. That doesn't seem to care much about proximity. That's the one that I assume is due to some kind of I/O problem as it updates everyone.