The Long and Morbid Tale of Sigil Games Online: Interview Edition

[Lum]

Does this all mean you love us again Lum and you've forgiven us for thinking games that dont work aren't cool?

No.

[Ironwood]

Be fair - He's busy getting his own game that won't work out the door.

 

[WayAbvPar]

Be fair - He's busy getting his own game that won't work out the door.

 

Well played, sir.


To kfsone's credit, I was a subscriber when he came on board at WWIIOL. Things definitely turned around and started heading in the right direction soon after, IIRC. 

[kfsone]

So why does the "garage mentality" still abound in MMOs?  Is it because the product is such an undefineable beast? With profits so high that control over them never really solidified into a disciplined method?

Roots and balls. Those are the guys who broke the ground, and therefore have the standing experience in actually applying any kind of development to MMO gaming. I don't consider them incompetent, because they've done that. On the other hand, the paedophile clergyman doesn't prove all religion is excrement.

As Lum says, there is a time and a place for scripting, and the early attempts were attempts to reinvent the wheel and so reaffirmed existing ideas of it being unsuited to MMO development.

But if you build your game with scripting in mind - at least for the purpose of scaffolding your "dog and pony" during early development, and long term for unit, automation and QA testing, you'll build a better application.

There's no direct or immediate guarantee you will build a better game, but I think we're beyond the point where the quality of product counts now - cf WWII Online - where quality of product was everything; there must be something to the game itself, or it wouldn't still be around with enough customers to register on SirBruce's charts. But the product they shipped was ... abysmal. And that window of acceptance was gone.

E&B was the flipside of the coin; superbly developed but lacking the "spirit" of an MMO. I think a lot of people dropped out because the game shared more in common with Monkey Island than UO.

[kfsone]

Be fair - He's busy getting his own game that won't work out the door.

 

Ouch :) I haven't shipped a game; in that sense I am as much a spectator as others here, that's perhaps why Lum made the point for me. Rather I've had the chance to confirm my speculative theories by working the other side of the fence; and in keeping WWII from the grave that was looming over its head, prove one or two of my others.

- Oliver

[HaemishM]

I'm starting to like Oliver.  Oliver can come round my house and fuck my sister.

I might even have a rational reply once I've fully digested that wall of text.  Can it be put in a flat file ?

I dunno, but if we put it in an ipod and wrap a condom around it, maybe your sister won't need me? :)

Ok, you've won me over with that comment. I still think WWIIO sucks it, but at least you recognize reality when it cockslaps you in the face. That means you have it all over some of the devs I've seen in the game industry, particularly the MMO Medium.

The rockstar mentality needs to DIE FUCKING DIE DIE DIE. There are few folks whose abilities I would trust to allow their ego to run rampant like that. Actually, I can't think of any. Warren Spector was responsible for Deux Ex: Invisible War. Garriot allowed Ultima 9 and lives in a fucking castle. Both Carmack and Romero have been responsible for shit like Doom 3 and Daikatana.

Your ego is always just one clusterfuck away from making mobile phone games.

[CharlieMopps]

So why does the "garage mentality" still abound in MMOs?  Is it because the product is such an undefineable beast? With profits so high that control over them never really solidified into a disciplined method?

Roots and balls. Those are the guys who broke the ground, and therefore have the standing experience in actually applying any kind of development to MMO gaming. I don't consider them incompetent, because they've done that. On the other hand, the paedophile clergyman doesn't prove all religion is excrement.

As Lum says, there is a time and a place for scripting, and the early attempts were attempts to reinvent the wheel and so reaffirmed existing ideas of it being unsuited to MMO development.

But if you build your game with scripting in mind - at least for the purpose of scaffolding your "dog and pony" during early development, and long term for unit, automation and QA testing, you'll build a better application.

There's no direct or immediate guarantee you will build a better game, but I think we're beyond the point where the quality of product counts now - cf WWII Online - where quality of product was everything; there must be something to the game itself, or it wouldn't still be around with enough customers to register on SirBruce's charts. But the product they shipped was ... abysmal. And that window of acceptance was gone.

E&B was the flipside of the coin; superbly developed but lacking the "spirit" of an MMO. I think a lot of people dropped out because the game shared more in common with Monkey Island than UO.

I have to say that the more I read about this, the more beta makes sense. I've been in other betas... and when there was a problem, there would be subtle tweaks and changes to make it work. In Vanguards beta, they would introduce a quest or whatever... and if there was a problem they would just scrap it. BAM! whole thing gone... and they would come up with something entirely different. It just seemed horribly out of wack to me...

[kfsone]

Your ego is always just one clusterfuck away from making mobile phone games.

But thanks to the WWII Onlines and AC2s, and the many I never heard of except in someone later lamenting, Rockstars are neccessary to keep interest (particularly $$$ interest) in the market so we don't skip right ahead to the EA phase ;)

But then there are the SWGs and the Sagases of heroses. I dunno if they teach us to shun the rockstar or ... the spinal tap ;)

[sam, an eggplant]

It's not a binary choice between

a) Like SWG, every single action is a database transaction
b) Like EQ, everything is logged and saved to flat files

And not because databases are too slow to do it, you can always throw hardware and tuning at that, but because it simply isn't appropriate use of technology. MMOs should be built upon a distributed tiered environment built on inexpensive horizontally scalable linux boxes or blades. The thick client (yourgame.exe) connects to auth servers, which hand off to game servers, which then commit changes to the database in batches every so often. The DB doesn't need to be beefy, because it isn't asked to do very much. So you can use a cheap solution like oracle standard with a HA cluster on the backend.

[Sairon]

It's not a binary choice between

a) Like SWG, every single action is a database transaction
b) Like EQ, everything is logged and saved to flat files

And not because databases are too slow to do it, you can always throw hardware and tuning at that, but because it simply isn't appropriate use of technology. MMOs should be built upon a distributed tiered environment built on inexpensive horizontally scalable linux boxes or blades. The thick client (yourgame.exe) connects to auth servers, which hand off to game servers, which then commit changes to the database in batches every so often. The DB doesn't need to be beefy, because it isn't asked to do very much. So you can use a cheap solution like oracle standard with a HA cluster on the backend.

Wow, you really managed to get a lot of buzz words in there, lets see if anyone can top it 

[sam, an eggplant]

Not everything you fail to understand is bullshit. Words to live by.

[CharlieMopps]

Database vs flatfiles is actually an interesting microcosm of the whole argument about scripting vs hard coding, since SQL is basically a scripting language that allows you to do things with data.

Scripting means that your designers can offlload more of the game's "work" onto code that they write themselves. It makes it far more likely that you will see cool and interesting combat features, for example. However, a script will never run as fast as well-written C code. So, it's a tradeoff. How much do you let your designers script vs how much do you hardcode and rely on programmers to write everything. It's an argument I'm well enmeshed in right now (and ironically, I'm coming down more on the scripting side of the equation!)

Games (like any other large ungainly software engineering project) are the art of the possible. It's always a juggling act between features vs. speed vs. stability vs. efficient budget vs. time to market. You can't have all of the above - it's unpossible.

I would, as an end user, come down entirely on the side of scripting. I mean, yea, the initial release of the game could possibly be cooler because of the lack of scripting... but you and I both know that no MMO is going to pay that kind of cash for expansions and whatnot. Wouldn't it make it a lot cheaper for the company to introduce new content? I think the main problem with MMOs now is they are too small... If someone just made a game in which creating lots of new content was easy and cheap, it would keep people interested a lot longer.

[Lum]

It's not a binary choice between

a) Like SWG, every single action is a database transaction
b) Like EQ, everything is logged and saved to flat files

Yeah, generally my rule was, anything where the user is trained to expect a brief wait (opening a search window for a auction house, say) should be database-driven, while anything where the user does NOT expect a brief wait (say, examining a monster) should be the quickest access possible. Which may or may not be a SQL database, but 4 years ago (when I gave my DB talk) it absolutely was NOT -- even Oracle DB clusters had multi-second waits between queries for most MMO-scale applications. Now, not so much.

[Morat20]

then commit changes to the database in batches every so often.

At this point, basic transaction theory rears it's ugly head.

Are you holding the uncommitted transactions in server memory, bypassing the DB entirely? If not, how are multiple clients accessing the same data assured that they're getting the right data, that their transactions are parsed in the correct order?

You do what you just described, and either you'll have a fuck-ton of overheard in server memory (the exact sort of shit DB's exist to handle swiftly), or you end up with inconsistent states among clients and some sort of luck-based write to the DB based on first access, or last access, or whatnot.

You don't think EVE moved to solid-state RAM for funsies, do you? Their DB access was a real issue -- lots of micro reads and writes -- and their hard drives couldn't keep up with it. The DB handled it just fine -- the bottleneck was disk I/O.

[DarkSign]

The really funny thing is, the most expensive part of making an MMO should be the art assets! Especially if you're doing something modern or futuristic with all the details and props.

BigWorld, Gamebryo, and the like arent the hugest dent in the budget. Even if you used Unreal3 (not a great engine for an MMO) you'd still only be paying somewhere between half and 3/4s of a million for an engine.

I really hope that McQuaid is never given the chance to make another MMO. Yeah one's accomplishments are worthy of praise, but if even half the stuff coming out of his own mouth are true, he should be done.  Not listening...willful ignorance...crying? Are you friggin kidding me with the CRYING??

[Sairon]

Not everything you fail to understand is bullshit. Words to live by.

I didn't call bullshit, in fact I don't even question your expertise on the area. But you DID in fact manage to squish a whole lot of buzz words into those few lines. 

[sam, an eggplant]

Searching for auctions is a perfect example of the kind of thing that really needs to hit the database. Basically you run everything you can at the app tiers. The database doesn't know about transactions until it needs to; changes are presented and processed at the app tier and committed when necessary on demand.

I haven't had the chance to examine CCP's environment, so I can't make an educated statement about their management. Maybe they did need solid state storage, but I really seriously doubt it. I know they're running SQL server on windows with support from IBM, so their IT staff probably isn't very savvy.

[kfsone]

Worth adding that Lum did eventually point out that he wasn't just using flat text files; the text files were managed by a suite of code Mythic had developed on previous projects which more-or-less could be construed as a database ;)

I mean, yea, the initial release of the game could possibly be cooler because of the lack of scripting...

Not sure if you're refering to client-side macros. I'm talking about building the game from the scratch with the notion of something that can dynamically run bits of the code. Building in the ability for the code to run headless.

That leads to an ability to automate the process of testing - either code or data.

[Margalis]

I don't understand the scripting convo going on here - even red names have been arguing that scripting is not good for a MMORPG. To me that seems incredibly silly. Sure, data-driven stuff is great but it isn't one or the other, and you can convert scripts to data stuff later. If you have a designer trying to make a quest what do you do:

1. Give him a scripting language and let him make the entire quest.
2. Have him request new product features, each of which need a full build before testing.

Hmm...tough choice. Easy instant turn-around that can be done by designers vs. slow turnaround that only coders can perform. And yes there is the testing aspect as well.

An MMO is a lot closer to system programming than app programming. It's just too much stuff to throw together without testing frameworks, unit testing, etc.

[Phred]

I have to say that the more I read about this, the more beta makes sense. I've been in other betas... and when there was a problem, there would be subtle tweaks and changes to make it work. In Vanguards beta, they would introduce a quest or whatever... and if there was a problem they would just scrap it. BAM! whole thing gone... and they would come up with something entirely different. It just seemed horribly out of wack to me...

I saw the same thing in WoW-TBC beta. There were a bunch of Hellfire peninsula quests that disappeared with no comment from the devs, and the quest rewards moved to other unrelated quests. Of course that's about the only similarity the two games have.

[robusticus]

Scripts (client-side macros) get a bad rep because they are generally lower quality, and harder to maintain.  That argument is moot against code generators, however.  Excel, Test Automation, etc.  Flash is, like, a whole dev environment, it looks like, even though it is called ActionScript.  I wonder if Maya generates code?  It comes with a script engine too, I understand.

XML files in a DB gets my vote.  Ca-chunk-a-chunk.

[Morat20]

Searching for auctions is a perfect example of the kind of thing that really needs to hit the database. Basically you run everything you can at the app tiers. The database doesn't know about transactions until it needs to; changes are presented and processed at the app tier and committed when necessary on demand.

I haven't had the chance to examine CCP's environment, so I can't make an educated statement about their management. Maybe they did need solid state storage, but I really seriously doubt it. I know they're running SQL server on windows with support from IBM, so their IT staff probably isn't very savvy.

You might -- and this is just a tiny suggestion -- look it up before you talk about it. Their enviroment calls for 30k or so concurrent users working with -- among other things -- a highly detailed and realistic market. Imagine the mother of all fucking auction houses, running with 30k concurrent players who hit the DB every time they transition star systems, every time they move through the market, virtually everything they do. CCP's database needs are light-years ahead of what games like WoW requires -- their single-shard design ensures that, and their highly detailed market (and associated market tools) are just the icing on the cake.

As for transactions -- once again, I think you underestimate an MMORPG's needs and the savvy of a lot of people working in the field. Vanguard was a clusterfuck of monsterous proportions, but most MMORPG's require highly experienced and proficient folks for both the network and the DB layers -- MMORPGS without experienced DBAs (from design through Live) don't live long. Among other things, duping gets out of hand.

Oh -- if I played an MMORPG that did batched writes to the DB at intervals, I could exploit sixteen kinds of shit out of it. I imagine it'd be very fun at the zone lines.

[Trippy]

CCP's database needs are light-years ahead of what games like WoW requires

No, they are not.

[kfsone]

Oh -- if I played an MMORPG that did batched writes to the DB at intervals, I could exploit sixteen kinds of shit out of it. I imagine it'd be very fun at the zone lines.

Seems you would have benefited from the aforementioned Databases seminar.

[sam, an eggplant]

You're just guessing. You can't eyeball the gameplay and intuitively infer how many and what types of transactions it would require to create that experience without any knowledge of their architecture. I see no particular reason why EVE's DB load would necessarily exceed other games in the genre. If you work for CCP please feel free to shut me down cold.

[Mandrel]

As for transactions -- once again, I think you underestimate an MMORPG's needs and the savvy of a lot of people working in the field. Vanguard was a clusterfuck of monsterous proportions, but most MMORPG's require highly experienced and proficient folks for both the network and the DB layers -- MMORPGS without experienced DBAs (from design through Live) don't live long. Among other things, duping gets out of hand.

Oh -- if I played an MMORPG that did batched writes to the DB at intervals, I could exploit sixteen kinds of shit out of it. I imagine it'd be very fun at the zone lines.

Well, duping in Vanguard has been so bad, people are calling for a barter system of trade on some servers, ignoring cash transactions:

http://www.silkyvenom.com/forums/showthread.php?t=20248&highlight=gold+dupe

[Jeff Kelly]

Prior to my arrival there (I worked my way up), the coders were the 'bottom' end of the business, everything flowed downwards to them, they had no say in timetables, design, or anything else.

This has been the case in any business I have ever worked in. Every time management dreams up a new feature or sales has promised the moon and the stars to another client gullible enough to buy something the Coders just have to "Make It Work"(TM) somehow. We were never asked beforehand if the things they tried to sell were actually feasible to do or at least could be done in the time they promised our clients.

If the schedule couldn't be upheld because of management's or sales's limited grasp on reality, lack of basic understanding about our products and technologies and the absolute inability to actually listen to things engineers and coders were telling them it was usually development's fault.

Maybe that just shows my lack of skills to chose the right employer but I have heard something like that or similar from too many people already to believe that.

I had a strategic meeting once at one of my last jobs where we were discussing a new product that was still in the most basic development stages. We are talking Ideas jotted down in a word document here. Our manager asked us how long it would take us approximately to reach the first prototype that could be paraded around on trade shows and exhibitions. We were pretty tight on manpower so we thought up a basic plan, explained the options and the risks involved and basically came up with about twelve months until a first prototype system. (Mind you to even begin development we first had to build the custom hardware we needed for this and we hadn't even begun planning that)

The answer to that was basically "Well you need to get it done in 6 months because we already have a customer for it and we told him that he could have the prototype by that time". I learned a lot about corporate business that day. To cut a long story short development still took nearly a year to complete as we had projected.

[CharlieMopps]

Yea, I write a lot of scripts and mini-apps arround my workplace to help my co-workers out. My Boss comes walking by one day and sees an app I had litterally just wrote... ask me what it does... he says: "Wow, that would save the team a lot of time!" 

I sware to God, 10min later he walks up to me and says "Hey, I'm in a meeting with 2nd shift right now... can you come in and train them how to use that app?"

[Engels]

- Mount masters have found a home near all rift way points. These masters will allow you to rent one of three flying mounts based on continent for a fee of 20 silver. You can use the flying mount as many times as you want within the 5 minute timer given. When the timer wears off you will be given a freefall effect for a short duration and the mount will be removed from your inventory.

Have fun and see you all in the skies of Telon!

A quick and easy way to ferret out the dupers? 20 silver for 5 min is a luxury only the very wealthy in game can afford.

I'm wondering if they've also fixed the Z axis issue, where flying over mobs, no matter how high up, would cause aggro down below. In any case, this is the first 'Sigil-free' patch for Vanguard.

[Morat20]

You're just guessing. You can't eyeball the gameplay and intuitively infer how many and what types of transactions it would require to create that experience without any knowledge of their architecture. I see no particular reason why EVE's DB load would necessarily exceed other games in the genre. If you work for CCP please feel free to shut me down cold.

I don't have to. CCP's been quite open with their DB design -- in fact, they had a number of really detailed Dev blogs last year as they were upgrading their systems to handle the lag.

They started having serious lag problems (multiple sources) once they hit about 20k concurrent. They had a number of consultants out to eyeball their DB server, their game server, their network setup, their threading  -- pretty much everything you can think of -- to look for problem areas and optimizaton possibilities. They ended up making a number of gameplay changes (most slashing down on the number of drones and finally getting around to handling bookmarks), I have no idea what sort of game server changes might have been necessary, but I recall their discussion of their DB woes quite clearly, as I was in the middle of a bunch of DB shit myself.

In short -- with a 20k+ concurrent user base, and the nature of their market and bookmark systems (other sources weren't as heavy a load), the DB was constantly being hammered for small requests that weren't inherently cacheable -- the requests rarely repeated over a short timeframe. The DB handled it well enough -- it couldn't effectively cache the requests, however, so it ended up reading and writing to disk a lot -- the bottleneck turned out to be disk I/O. IIRC, they stated write-queues were something like 10 to 13 deep.

Trippy:

No, they are not.

I would say they are -- they have 10 times the concurrent users WoW does, they're not planning on ever sharding it so concurrent demands are simply going to grow, they collect a LARGE amount of detailed data, and the market is one of the primary DB drivers and it's fully integrated into the game. At any given moment, more people are utilizing EVE's market alone than are on the average WoW shard. Now, I'm aware Blizzard shares DB servers across multiple shards, but I can't see any way in which Blizzard's design calls for even a tenth as much DB usage as EVE's -- it's just an entirely different design.

kfsone:

Seems you would have benefited from the aforementioned Databases seminar.

No thank you. I find DB design to be highly boring, and will only grudgingly have anything to do with it. We had to redesign one of our smaller databases, and I got tasked to do it -- I almost quit when my boss's boss wanted proof the new design (and the changeover) wouldn't lose data. Proving it was lossless wasn't that bad, but knowing the guy demanding it couldn't understand my proof or even really check if I did it right -- irks me. DBA type jobs seem pretty thankless, where you only get attention when it goes all fucked up.  I have a huge appreciation for the sort of serious database stuff that sits behind banking systems and the like, though -- and I know enough to appreciate the sorts of effort behind most MMORPG DB designs.

One of my gripes with the pre-Blizzard generation of MMORPGs is that it seems a lot of early DB design was done by programmers -- not qualified database folks. Database design is one of those areas where I'd want serious talent pretty early. It's not glamorous, but a poor design will hamper you for the lifespan of the game -- and changing it after the fact is a cast-iron bitch.

[sam, an eggplant]

I tried to read the dev blogs, but I guess you need to be a game subscriber. I'm incredibly skeptical that solid state storage was required. There are all kinds of things you can do to ameliorate physical I/O, but without the ability to make my own diagnosis I can't really make a judgment one way or the other. My off the cuff guess is that their consultants suck.

[Roac]

I tried to read the dev blogs, but I guess you need to be a game subscriber. I'm incredibly skeptical that solid state storage was required. There are all kinds of things you can do to ameliorate physical I/O, but without the ability to make my own diagnosis I can't really make a judgment one way or the other. My off the cuff guess is that their consultants suck.

Feel free to figure out why the design hosting a working solution is wrong after reading this.  Or google for the piles of articles on the subject.

[bhodi]

The hardware involved in that article makes me drool. I may be a geek, fine, but holy crap. I would love to manage a system like that. I've dealt with blade severs and multi-terabyte SAN storage, getting my hands dirty trying to tweak it for maximum performance -- but I've never touched anything like solid state storage. The performance on that must be unreal. Sometimes, you've got to just love it -- that would probably be my dream job.

Maybe even unreal enough to make Microsoft SQL Server perform reasonably fast...

I tried to read the dev blogs, but I guess you need to be a game subscriber. I'm incredibly skeptical that solid state storage was required. There are all kinds of things you can do to ameliorate physical I/O, but without the ability to make my own diagnosis I can't really make a judgment one way or the other. My off the cuff guess is that their consultants suck.

Keep in mind Sam that they ultimately went for the "easiest" and most definitely fastest solution to the problem - a metaphorical "bigger box". There's really nothing wrong with that. When you've got time constraints that don't allow for a year's (optimistic) migration path from design to functional testing to transition, sometimes you can just throw a band-aid at the problem. A really, really BIG EXPENSIVE band-aid. Often, it will delay the issue past the life expectancy of the problem. It may not be the most elegant, true, but in the end it's the results that matter. I'm surprised that you can't see that.

That said, I personally won't play Eve because of the lag involved in large scale battles. I understand something of the issues involved in hosting such an event, but as the game is designed around it, and it's what I would like to be seriously involved in if I were to play (beyond my 2 week trial a few months back) half-minute delays on actions and screen updates every 15 seconds is not my idea of a good time.

[Morat20]

That said, I personally won't play Eve because of the lag involved in large scale battles. I understand something of the issues involved in hosting such an event, but as the game is designed around it, and it's what I would like to be seriously involved in if I were to play (beyond my 2 week trial a few months back) half-minute delays on actions and screen updates every 15 seconds is not my idea of a good time.

CCP is learning what every MMORPG game ever has learned about PvP. If you design to handle 20v20, they'll show up with 40v40. Work like dogs to make 40v40 smooth as glass, they'll show up with 80v80. CCP's performance for large scale battles is really great -- except then people made the battles even larger.

Right now, IIRC, they're trying to move in game mechanisms that discourage large-scale battles -- trying to keep fleet actions down to the 100v100 range where the servers don't choke.

[Roac]

MMOGs are the idiot's guide to non-linear performance issues.