The Long and Morbid Tale of Sigil Games Online: Interview Edition

[Morat20]

Trippy:

No, they are not.

I would say they are -- they have 10 times the concurrent users WoW does, they're not planning on ever sharding it so concurrent demands are simply going to grow, they collect a LARGE amount of detailed data, and the market is one of the primary DB drivers and it's fully integrated into the game. At any given moment, more people are utilizing EVE's market alone than are on the average WoW shard. Now, I'm aware Blizzard shares DB servers across multiple shards, but I can't see any way in which Blizzard's design calls for even a tenth as much DB usage as EVE's -- it's just an entirely different design.

If you agree that multiple WoW servers share the same DB then it's the same problem.

They do -- and it's obvious when WoW's DB is under stress (I played on one of the older realms, so it was common for most of the servers to be sharing a DB server to be close to full). Mail lag, auction lag..ugh.

I don't recall the specifics on how many servers share a DB on WoW, but it'd have to be 10 or so to hit the same concurrency as EVE (fewer now -- they did up the players per server before they start a queue). Concurrency isn't the only issue -- it's DB accesses and whether it can be cached, and WoW pretty much only hits the DB when you're using the AH, your mailbox, the bank, or loading. I don't know if they sever the AH from the mail and bank systems (it would be pretty easy, but I'm not sure it's necessary) but the AH doesn't have nearly the same sort of load requirements as EVE's market. (Seriously, that thing has to be a cast-iron bitch. EVERYTHING is done through that. You can do a lot of caching for an AH-style thing, but for an EVE-style market that's going to be really limited).

That and their asset tracking system (I can find -- and often manipulate -- stuff all over the place, not just local to me) really adds a lot that WoW lacks.

WoW's got the better end of the stick by far, and it shows in gameplay. It took a lot to stress their DB systems to begin with, although it still took a year or two for them to finally catch up to the game becomming far more popular than they anticipated. (Which is a problem everyone wishes they had!).

kfsone: Sorry, I have good reason to hate it. I find it really boring, and I'm the only one qualified in my department to do it. Luckily it doesn't come up much, but some of our stuff is considered man-rated or is under FOIA requirements, so when I am called in to do something more complex than "add a field, please" it often requires a LOT of effort to prove to people up and down the chain that it won't break things, won't lose information, is in fact an improvement and will do what was required, etc. I've been trying to con them into hiring a full-time DBA, but since they'd just sit on their asses 9 months of the year -- not really viable.

As for the article -- I think you're reading too technically from what is more of a laymen's article. Latch/wait time improved 6-fold (According to the article), but I don't think that's where they were getting the 40-fold improvement numbers from. I think that was just them tossing in an example of something that improved -- it makes sense that not all performance measurements would change equally, and I suspect the 40x one game from the most improved performance indicator -- undoubtably an outlier. From a gameplay perspective, it looks to be a 10 to 15-fold improvement (judging by wait times for things like the market in Jita) -- which came despite an increase 33% increase in concurrent users. That performance was massively improved was pretty obvious. :)

They're still gonna run into the dual overheads of MS SQL and running it under Windows. And from conversations I overhead with CCP staff at a couple of conferences, it still doesn't sound like they've taken the approach that banking, military and gaming industries use of putting proprietary/custom authoritative proxies infront of the database.

Yeah, I'd like to corner their staff and ask questions -- not that I think they'd answer all the ones I had in mind. :) I'm not sure how much front-end work they plan to add later, but I suspect it's on their list. DB access is going to remain a bottleneck, and they've never shown any signs of being shy about hiring outside experts. (They grabbed one of the stackless python gurus to come verify their micro-threading approaches, for instance, back when they were doing the initial rounds of performance upgrades). Since performance upgrades are a near-constant fact of life for them (single-sharded design forces that), I'd suspect that their DBAs are not just sitting on their asses. :)

I said a long time ago that if I was going to design an MMORPG, I'd try to hire at least one DBA guy from the banking industry. Some of the requirements they work under are amazing -- so are some of the solutions.

[Morat20]

25 years ago I used to work in the data center of a mainframe for a medical insurance company. This consisted of users sending down punch cards requesting a given data tape full of records, at which point we would fetch the data tape off of a library shelf and load it into one of the tape drives.

That's right -- 25 years ago, I was part of a hard drive.

Data access techniques have improved since.

It seems "I was around for punch cards" is a bragging right in every variant of the IT field.

[Nevermore]

About 2500 years ago I used to maintain a database using stone tablets and a hammer and chisel.

[Morat20]

About 2500 years ago I used to maintain a database using stone tablets and a hammer and chisel.

*sneer*. It was probably in 1st NF. Lazy-ass Babylonian DBA's, nothing more than spreadsheet jockeys.

[Miasma]

My old boss used to talk about his first job with the bank in the IT department, he had to pour palettes full of water jugs into the mainframe's cooling system.  Sometimes the raised flooring got badly gummed up and he would have to clean it out, six inches of cigarette ashes and all.  I can't imagine someone smoking in an office now, people would freak out and start fainting at the sight of it, the police would have to be called to make sense of the situation.

[kfsone]

kfsone: Sorry, I have good reason to hate it. I find it really boring, and I'm the only one qualified in my department to do it. Luckily it doesn't come up much, but some of our stuff is considered man-rated or is under FOIA requirements, so when I am called in to do something more complex than "add a field, please" it often requires a LOT of effort to prove to people up and down the chain that it won't break things, won't lose information, is in fact an improvement and will do what was required, etc. I've been trying to con them into hiring a full-time DBA, but since they'd just sit on their asses 9 months of the year -- not really viable.

I've tried to learn what I know about databases from people who eat that crap up - aware that as a programmer I am automatically disqualified from being a good DBA. Old Bald Angus' writing style always reminds me of a scottish guy I used to work with at Granada Media, who was probably the best DBA I've ever come across; his eyes lit up at stuff that was stale enough to freeze the nuts off satan in the middle of a good blowjob.

But what made Angus different from a lot of DBAs was that he cared enough about his data to know when to tell the developers when something wasn't the databases job. Angus built scheduling and indexing maps, so that when you went to him to grovel for a new query, you had to present him with documentation of how it would be invoked, and he'd benchmark it during testing. But he could tell you what the effects were going to be - with a good degree of accuracy - before you started using it. And based on your PID, he could probably suggest an alternate approach of aquiring the data.

"Mah daytey'baase isn'y gonna do that"

As for the article -- I think you're reading too technically from what is more of a laymen's article. Latch/wait time improved 6-fold (According to the article), but I don't think that's where they were getting the 40-fold improvement numbers from.

That was just a link I had to hand - my point was that I've never seen any numbers, in any articles or blogs, that affirm a 40x improvement. And besides, take a moment to think about this. Those figures should be a little alarming... That solid state disk ought to be providing 100x and above performance increases. To be fair: maybe they have a lot of retuning to do - it could be that their disk-oriented tuning is more of a hinderance.

I'm also a little jaundiced about the bits and pieces I read about external consultants; I've always found that good consultants solve the problem you have rather than the problem that causes it unless the amounts of money are vast. The more of your problem a consultant tackles, the more she/he is lining themselves up for a fall. As well as having worked for several companies that threw millions at consultancy, I also spent several years as a consultant.

{O} If you aren't going to put a head infront of the database to "run" the economy, you are asking for commit pains. {R} If you're insistent on running the economy direct from database ... {A} case you can go for spatial tables (but not with MS SQL until 2005 and I gather they're running 2000) and partition them heavily. {C} Or you use the spatial attributes to partition which table/database/disk/dbserver items will be in. {L} Course you need a database that can do that kind of clustering for you transparently. {E}

I said a long time ago that if I was going to design an MMORPG, I'd try to hire at least one DBA guy from the banking industry. Some of the requirements they work under are amazing -- so are some of the solutions.

Aye, Angus was a banker. Well, a former banker. At his age, you tend to tone it down. I also did a 6 month stint with Barclays Bank in Warrington during my consultancy phase, and their DBAs were pretty good. Shame the hiring of coders wasn't as good as the hiring of DBAs ;)

There are also some pretty good DBAs in the "adult" sector. Their business seems to be capable of generating incredible access rates per site.

Not that I'd know anything about that.

- Fred "notkfsone" Jones

[Morat20]

{O} If you aren't going to put a head infront of the database to "run" the economy, you are asking for commit pains. {R} If you're insistent on running the economy direct from database ... {A} case you can go for spatial tables (but not with MS SQL until 2005 and I gather they're running 2000) and partition them heavily. {C} Or you use the spatial attributes to partition which table/database/disk/dbserver items will be in. {L} Course you need a database that can do that kind of clustering for you transparently. {E}

I don't disagree. I don't think CCP ever forsaw how big the game would get. They were probably hoping -- in their wildest dreams -- for a 12k concurrency game and a 60k user base. It would have kept them in vodka and hardware and spending money and let them just tinker with the game for as long as they kept their users. Once they hit 20k concurrents, a lot of early assumptions had to be ditched.

We are currently in the process of upgrading our DB servers, and it's been a PITA. (Part of that was it was part of a cluster of changes, including reliability and backup stuff that got folded in at -- as usual -- the last minute). Given none of the stuff we were moving is really that intensive on the DB end, it was still a surprising amount of work.

Trying to transition a mature game with a schema as complex as CCP's has to be, with that large a DB and a low tolerance for errors -- just upgrading your software is a lot of work. Trying to transition to an entirely different server as well (from MS to whatever) is something that's a major deliverable, on par with one of EVE's expansion-sized patches. And it's fairly transparent to users, which means that they're not going to want to feel they've been neglected with "real" changes while you spend the time and effort.

If nothing else, unless I was hiring outside talent either experienced with the new system or with transitions, I'd need to have a dedicated team and give them some serious time to familiarize themselves with it. Then there's designing the new systems, handling the transfer, all the inevietable errors, trying to fix those....Major effort. I suspect it's on their "to-do" list -- but faced with a critical and game-breaking problem (huge lag) that popped up over less than six months (they had a serious period of growth), if your current setup is even semi-competently done, hardware is all you can do -- especially if your main bottleneck is disk I/O. Everything else has a time-frame of six to 18 months (depending on what you're doing) and what that buys you is iffy. Probably much better scaleability, in the long-run.

I don't think CCP runs the economy directly from the DB, but it is by nature a transaction-oriented thing. No matter how clever you are, there's just going to be lots of micro-requests off the DB -- and because of the breadth of the market, it's not inherently cacheable. There's just no real predictability and little repeats of queries (at least in the short-term timeframe).

I don't doubt CCP's got long-term plans for updating and changing their DB solutions -- it's just not as sexy to talk about (at least as far as players are concerned) as enchanced graphics, avatars, new ships, etc.

[Margalis]

Rerail!

http://www.1up.com/do/newsStory?cId=3159832

F13 praise from 1up.

[schild]

Yea, Zonk. Saw that the other day. Was nice of him.

1up.com needs new, faster, more robust servers.

[Zonk]

/hattip

And tell me about it.

[CharlieMopps]

That was a good Overview of what happened. I have to agree with everything he said there. I kind of wish someone would have mentioned just how much this sort of thing was brought up in Beta. There were literally dozens of us SCREAMING in the Vanguard Beta forums for them to put on the breaks, only to be repremanded by the Devs. I was told that "Maybe you aren't cut out for a Beta if you can't handle a few bugs." pffft

[Cyndre]

That was a good Overview of what happened. I have to agree with everything he said there. I kind of wish someone would have mentioned just how much this sort of thing was brought up in Beta. There were literally dozens of us SCREAMING in the Vanguard Beta forums for them to put on the breaks, only to be repremanded by the Devs. I was told that "Maybe you aren't cut out for a Beta if you can't handle a few bugs." pffft

Thats funny, I never had a Dev tell me that, but the Beta population would jump on anyone that suggested anything wasn't working properly.   There was a forum mafia that canabalized anyone who wasn't a rampant fanboi of every failed design choice they failed to impliment.

[Cyndre] {Bug Report} I fell through the world while learning skills in...

[ForumMaggot213] Well maybe you aren't hardcore enough to understand Vanguard.  Go back to WoW if you aren't up to the challange this game provides true gamers..

[Cyndre] umm...  ok, I'm tired of falling through the world anyway, and we have a MC raid tongiht. 

[sam, an eggplant]

There were literally dozens of us SCREAMING in the Vanguard Beta forums for them to put on the breaks, only to be repremanded by the Devs. I was told that "Maybe you aren't cut out for a Beta if you can't handle a few bugs." pffft

That happens at the end of every MMO beta. Vanguard was hardly unique.

Don't get me wrong, most of the time the users are right.

[Mandrel]

There were literally dozens of us SCREAMING in the Vanguard Beta forums for them to put on the breaks, only to be repremanded by the Devs. I was told that "Maybe you aren't cut out for a Beta if you can't handle a few bugs." pffft

That happens at the end of every MMO beta. Vanguard was hardly unique.

Don't get me wrong, most of the time the users are right.

While it does happen in most Betas, it happened early and often.  When your Beta community is so overtly hostile to any comparisons to other games, it doesn't bode well.  Heaven forbid if someone talked about how one of WoW's game mechanics was superior to one of Vanguard's, or suggested to implement something successful from another game.  I've never seen a more venomous, superior, holier than thou fanboi community than was exhibited in Vanguard.  That's probably part of the reason the beta populations were so incredibly low as compared to the number of invites sent out.

[Nebu]

To be fair, I took a lot of heat in the WoW beta for pointing out what I considered glaring problems... many of which were still present after release.  Fortunately, Blizzard is a lot more open-minded about the process and many of the holes were fixed (albeit slowly).  The Vanguard beta was similar in strength of fanboism, but had the blinders on a lot more tightly than I've seen in a beta.  The Vanguard tester base was bent of a penal system the likes that would never sell to a mass market.  It's a wonder noone figured this out earlier. 

[HaemishM]

The Vanguard beta was similar in strength of fanboism, but had the blinders on a lot more tightly than I've seen in a beta.  The Vanguard tester base was bent of a penal system the likes that would never sell to a mass market.  It's a wonder noone figured this out earlier. 

Lots of people figured it out, as evidenced by all the people who didn't buy the game. The only ones who seemed immune to this infection were the devs and vanbois.

[Nebu]

I guess my amazement is that none of the money people had enough of a clue to force the hand.  I guess the whole thing was such an admitted disaster at this point that the best way to slow the bleeding was to release anything and see how much it would recoup.  If we see another "shove it out the door in whatever state it is and we'll fix it later" projects, we'll know that noone learned a damn things from this debacle.

[sam, an eggplant]

That's probably part of the reason the beta populations were so incredibly low as compared to the number of invites sent out.

Not necessarily. I've been in tons of MMO betas, and it's very common for people to stop playing once the initial flush of "I'm in a beta, this is so cool!" dies out. Hell, in LOTRO they resorted to bribes to get people to play, giving out free ipods and best buy gift certificates and such. At the time I took it as a sure sign that the game was doomed to fail miserably. Shows what I know!

[Trippy]

That's probably part of the reason the beta populations were so incredibly low as compared to the number of invites sent out.

Not necessarily. I've been in tons of MMO betas, and it's very common for people to stop playing once the initial flush of "I'm in a beta, this is so cool!" dies out. Hell, in LOTRO they resorted to bribes to get people to play, giving out free ipods and best buy gift certificates and such. At the time I took it as a sure sign that the game was doomed to fail miserably. Shows what I know!

They did that on the closed Beta stress-test days to encourage maximum participation (and it worked too). It wasn't a regular thing.

[sam, an eggplant]

Well, I remember about one offer per week for two or three months, mostly $200 bestbuy gift certs but also ipods and wiis. At the time I thought it was a spectacularally bad sign, particularly since it immediately followed the DDO release.

Anyway, every MMO beta I ever played (and I tried a lot) had trouble getting people to login except for WoW. I wasn't in the UO or EQ betas.

[Azazel]

LOTRO's beta also coincided with WoW's opening-up of the PVP rewards to non-catasses and to the release of BC.

I can imagine either of those causing a drop-off, let alone both of them.

[Hoax]

To be fair, I took a lot of heat in the WoW beta for pointing out what I considered glaring problems... many of which were still present after release.  Fortunately, Blizzard is a lot more open-minded about the process and many of the holes were fixed (albeit slowly).  The Vanguard beta was similar in strength of fanboism, but had the blinders on a lot more tightly than I've seen in a beta.  The Vanguard tester base was bent of a penal system the likes that would never sell to a mass market.  It's a wonder noone figured this out earlier. 

I agree even in the OB the vocal minority (that I was a part of) asking just where the fuck this "pvp system" was were shouted down by fanbois to the point where I just gave up talking about it.

Also hunter bugs were reported for months and didn't get ironed out completely until well after launch.  Hunter being the last class (if I remember right) to receive talents.

*removed extra )*

[Ironwood]

Hunters were a shambles at release.

[Cyndre]

Hunters were a shambles at release.

To be fair, every class was pretty borked.   Warlocks were all but useless, Hunters were downright broken, Shaman could kill you from 12 zones away without targeting you...   the list goes on.

In all honesty, there arent major design issues, imo.  I feel like Blizzard launched a good product and then set about making it better.  Comparing the errors to Vanguards is like two people complaining about the water at a resturant.  One got lemon in his without asking for it, the other had fecal matter floating around.

Also, the Hunter community in Beta was almost unanimously pointing to the same flaws, as were the Warlocks etc..   The PvP system was acknoledged to be flawed by almost everyone, and most of the design staff.   In Vanguard, pointing out a class or design flaw made you the target of vehement protest and ridicule, rather than part of the wow rant machine.

[Ironwood]

You project a LOT.

[Hoax]

Yeah dont get me wrong, WoW had the best pvp class balance at launch I've seen from a title worth mentioning.

But it still had some wacky bugs and pvp was obviously a last minute bolted on job.  Its a testament to how much better at their jobs Blizzard's people are when compared to say Mythic.  When WoW's balance (even w/ Shaman and Shadow Priests at launch) in pvp was better then DAOC's at launch.

Also Blizzard may be magical but their forum groupies are just as lame and fanboi as anyone else's also there are a shitton more of them.  If I had a dollar for every post that was "its just the beta" when it was obvious we were playing the current version of a game that was due to launch in less then a month...

[Cyndre]

Also Blizzard may be magical but their forum groupies are just as lame and fanboi as anyone else's also there are a shitton more of them.  If I had a dollar for every post that was "its just the beta" when it was obvious we were playing the current version of a game that was due to launch in less then a month...

I guess I had a different impression but it was a while ago.  As a warlock, I felt pretty well unified with every other tester who was ranting about how stupidly broken the class was.  With regards to a lot of the other bugs the only place I encountered a lot of the 'its only beta' resistance was in reporting quest bugs.  I didn't spend much time in the PvP discussions, so I can't really comment there.  As a matter of fact I think I could count on one hand the number of times I've read any PvP forum for WoW.

[Nebu]

Yeah dont get me wrong, WoW had the best pvp class balance at launch I've seen from a title worth mentioning.

Balance how?  1v1?  5v5?  50v50? 

I think it's difficult to make such a statement as you aren't comparing apples to apples.  DAoC has 3 realms, each with significantly more classes.  WoW balance should have been near perfect given that there were fewer classes, two realms, and fewer abilities.  With such a significant decrease in the variables, I'd argue that they actually did a poorer job than they should have at release.  WoW also suffers from the fact that gear = toon.  This is much less the case in DAoC. 

[Hoax]

WoW = Shadow Priest was god at 1v1, also Shaman's were pwning people pretty hard.  But everyone even bugged out hunters could have fun.  This is the key so let me repeat it for you.  EVERYONE COULD HAVE FUN.

This is 1v1 pvp (least fun cuz some classes would just pwn others) to GvG to large sized crazy battles.  This is pre-BG's back when we actually had world pvp mind you.


DAOC = So many classes were completely unplayable.  Clothies that weren't horribly overpowered (thurgs at launch) were just insta-gibbed by archers from stealth.  Everyone else was food for whichever group had the minstrel or the equivalent who hit aoe mez faster.  Then we all got to stand around and watch the group die one by one.  Fucking awesome.

Dont even get me started on light tanks at launch for Hib/Alb though the Thane was pretty pimp.

DAOC's balance sucked from pve, to 1v1 pvp, to full scale pvp.  I dont care if it was le awesome 6 months later or whatever you are going to tell me.

At launch IT WAS NOT FUN.  This is called bad balance.  When your choice of class can eliminate the fun from your game.  You have done a bad job.  Mythic did a bad job.  Blizzard did not.

[Ironwood]

Balance = Fun ?

Dime Bar ?

[Hoax]

That went right over my head.  Some kind of euro joke?

[schild]

God I love that bar.

[Nebu]

DAOC's balance sucked from pve, to 1v1 pvp, to full scale pvp.  I dont care if it was le awesome 6 months later or whatever you are going to tell me.

At launch IT WAS NOT FUN.  This is called bad balance.  When your choice of class can eliminate the fun from your game.  You have done a bad job.  Mythic did a bad job.  Blizzard did not.

So, you're comparing PvP of a game made 4+ years later to DAoC at release?  That's silly.  Blizzard had DAoC, Shadowbane, Planetside, Darktide, and many other games to learn from in their development and they still released a pvp experience that was a mediocre tack-on.  I'm not even going to mention the differences in development resources. 

Compare PvP in DAoC now to PvP in WoW as both games are today.  I think you'd discover that it's a matter of taste.  I find DAoC to be a much richer and more complex pvp system that takes a long time to really appreciate.  There are many layers to it and the "sandbox" feel is definately more to my liking.  YMMV.

[Hoax]

So your counter point is four years ago class balance wasn't seen as important?  Or it just wasn't invented yet?

Did I not call WoW's pvp a tack on?  Yes I did, more then once in this thread I believe.  Did I not also point out how that makes it especially telling that they still did such a good job of balancing their classes for a tacked on, total afterthought type of gameplay?  Because they are good at what they do.  Unlike, well fucking damn near everyone apparently who makes MMO's.

The difference in resources may have been huge.  But you know what we always say around here.  Stop designing shit you can't produce.  Dont plan to patch in the fun later.  Fucking polish your core gameplay and make sure it works.  Dont just throw a bunch of pie in the sky halfassed innovations at the players and hope some stick.  Etc etc etc.

I dont need to compare shit today.  DAOC at launch is the only fucking thing I've ever been talking about.  If the fun had to be patched in, well l2dev.

Yeah dont get me wrong, WoW had the best pvp class balance at launch I've seen from a title worth mentioning.

But it still had some wacky bugs and pvp was obviously a last minute bolted on job.  Its a testament to how much better at their jobs Blizzard's people are when compared to say Mythic.  When WoW's balance (even w/ Shaman and Shadow Priests at launch) in pvp was better then DAOC's at launch.

Emphasis added to the statement I made that started this little aside.

[Nebu]

So your counter point is four years ago class balance wasn't seen as important?  Or it just wasn't invented yet?

My counter point is that it's a silly comparison.  The class balance in DAoC at release wasn't as good because they had a) less precident to draw from, b) fewer resources to draw from, and c) had fewer and less experienced beta testers to query.  It's not an apples to apples comparison.  Given the same 4 years of history to work with, DAoC created a superior pvp experience with fewer resources.  That's why I suggested that a better apples to apples comparison would be DAoC now to WoW now.   If you disagree, that's fine.  History is a very important resource to draw from.  It allows you to learn from the mistakes of others.  Blizzard had much more mistakes to learn from than EQ or DAoC.  They also had the resources to do something about those mistakes. 

If you want to make a better comparison, I'd argue that WoW vs Vanguard makes your point in a more compelling fashion.