The Long and Morbid Tale of Sigil Games Online: Interview Edition

[Nebu]

CCP is learning what every MMORPG game ever has learned about PvP. If you design to handle 20v20, they'll show up with 40v40. Work like dogs to make 40v40 smooth as glass, they'll show up with 80v80. CCP's performance for large scale battles is really great -- except then people made the battles even larger.

Right now, IIRC, they're trying to move in game mechanisms that discourage large-scale battles -- trying to keep fleet actions down to the 100v100 range where the servers don't choke.

They should chat with Mythic.  They've done an excellent job of keeping large scale pvp pretty manageable.  I've seen fights that were 120 vs 120 run pretty smoothly on what I'd consider a pretty modest machine. 

[sam, an eggplant]

Keep in mind Sam that they ultimately went for the "easiest" and most definitely fastest solution to the problem - a metaphorical "bigger box".

I think you hit on the real answer. They threw money at the problem to fix the symptoms. It's not elegant or satisfying, but it does work.

[Morat20]

Keep in mind Sam that they ultimately went for the "easiest" and most definitely fastest solution to the problem - a metaphorical "bigger box".

I think you hit on the real answer. They threw money at the problem to fix the symptoms. It's not elegant or satisfying, but it does work.

I don't see another solution -- from the specs listed -- that doesn't require rewriting the whole back end, and cutting out a large swath of available functionality. Scaleability isn't infinite, and since the bottleneck was disk reads, that's the place to start.

I should note that they also made gameplay and design changes as well -- CCP has a three-pronged approach to performance issues. Upgrade hardware, alter gameplay, optimize code -- they do all three for any significant issue. (Currently they're working on gameplay an optimization for the fleet blob issues).

Given the expense of solid-state disks, they wouldn't have switched if merely hiring a good consultant or two would have fixed it. And after reading the article, I suspect the first thing a consulant would say would be "Have you thought about solid state disks?"
Nebu:

They should chat with Mythic.  They've done an excellent job of keeping large scale pvp pretty manageable.  I've seen fights that were 120 vs 120 run pretty smoothly on what I'd consider a pretty modest machine

That's where you get into the issues inherent in the specifics of how an engine was written, and the whole architecture of the system (and the problem isn't client-side -- it's server side). Generally a 100v100 battle is 100v100 ships, plus another 1000 or more drones and fighters. They also processor-limit themselves -- one server per star-system (multiple star-systems can run on a single server, but they won't break up a system onto multiple servers) -- and have huge overhead with the gang bonuses, AI for semi-independent pets (drones and such), massive AoE blobs, situational buffs and debuffs (EW ships swapping targets), turret tracking and traversal eats up a ton I'm told -- lots of independent and math-intensive stuf that changes constantly.

They just simplified the gang bonus system, which supposedly removed some of the server load for large battles. They cut down drones awhile back, and they constantly upgrade their hardware. I'm under the impression that code optimization is an ongoing process.

[sam, an eggplant]

I don't see another solution

There are TONS of possible ways to address physical I/O constraints; upgrading hardware is simply the most obvious and easiest answer. You don't see a solution because you haven't looked at their architecture; neither have I. You're falling into the usual internet armchair architect fallacy. You think you know stuff when all you know is what they told you. Which makes this an incredibly stupid argument. Lets stop wasting electrons and end it here, OK?

[Roac]

There are TONS of possible ways to address physical I/O constraints
...
You're falling into the usual internet armchair architect fallacy.

Too funny.

[Morat20]

I don't see another solution

There are TONS of possible ways to address physical I/O constraints; upgrading hardware is simply the most obvious and easiest answer. You don't see a solution because you haven't looked at their architecture; neither have I. You're falling into the usual internet armchair architect fallacy. You think you know stuff when all you know is what they told you. Which makes this an incredibly stupid argument. Lets stop wasting electrons and end it here, OK?

Wait, let me get this straight.

There are tons of solutions. But you don't know them, because you haven't seen their architecture. So how can you claim there are tons of solutions?

I've at least played the game, followed it's development, read a hell of a lot more on it than a single article, and have slightly more to go on than you. All the solutions I can think of are non-starters because they interfere with primary gameplay, or require multi-year rework of an established system, or require CCP jettisoning certain requirements that they have indicated are 'core' to the game.

I'm sure there are dozens of theoretical ways to fix it without hardware upgrades. For instance, we could eliminate the market entirely. That would fix the problem. I could shard the system -- break EVE up into a dozen or more systems with a peak concurrency of 3 to 4k. That would fix it. I could turn the market into a simple auction house instead of a functioning market -- that'd probably do it too.

However, given the existant game play -- the data and DB requirements necessary for a market, a single-sharded 30k concurrent game, you're still running into the same basic, common, simple problem -- the DB requests are frequent, small, and do not lend themselves to caching. There are only three ways out of that -- slow down the DB requests, make them larger (more data per request = fewer requests), or find a way to cache them.

You can't slow down the DB requests without sharding the game or removing the market -- that's the two primary drivers. You can't make them larger -- it won't slow down requests, which are driven by user browsing through market data. I'm sure requests are already cached, but given the nature of user browsing caching doesn't buy you much except not hitting the DB when you flip "back" to a previous item. User browsing is non-predictable, non-repetitive (over the short-term).

[Ironwood]

Guys, Guys, Guys; We Can't Fight Amongst Ourselves.

DON'T YOU KNOW THAT'S WHAT BRAD WANTS YOU TO DO ?

[Morat20]

Guys, Guys, Guys; We Can't Fight Amongst Ourselves.

DON'T YOU KNOW THAT'S WHAT BRAD WANTS YOU TO DO ?

I'm bored with Vanguard already. It doesn't have the staying power of a good UO/Trammel thread or your average SWG thread.

[cmlancas]

Guys, Guys, Guys; We Can't Fight Amongst Ourselves.

DON'T YOU KNOW THAT'S WHAT BRAD WANTS YOU TO DO ?

I'm bored with Vanguard already. It doesn't have the staying power of a good UO/Trammel thread or your average SWG thread.

Even though I am still currently subbed to VG, a person has to not be shit on continuously by McQuaid to care.

I know I'm a newbie here, but I took a good chunk of my day reading the hilarity of that which is the f13 UO/Trammel. I just may have peed a little.

Edit: Typo.

[bhodi]

LOL, Internet.

[sam, an eggplant]

You simply don't understand DB performance tuning. Which is fine since you're not a DBA. Take my word on it; getting faster hardware should always be the last solution. Or not, I don't give a shit. Wasting electrons, remember?

[Engels]

This is all very fascinating, but don't the performace issues with Vanguard have more to do with client side code than server side database access speed? Or have I just wandered into an entirely tangential conversation?

[bhodi]

I wonder if they had any consultants or contractors come in to consult on aspects of vanguard's architecture?

[HaemishM]

I wonder if they had any consultants or contractors come in to consult on aspects of vanguard's architecture?

Just Brad's ovaries.

[Morat20]

You simply don't understand DB performance tuning. Which is fine since you're not a DBA. Take my word on it; getting faster hardware should always be the last solution. Or not, I don't give a shit. Wasting electrons, remember?

Listen, numbnuts. I fucking understand DB tuning just fucking fine. I've had to do it on a number of occasions -- mostly because the DB in question had been designed by someone who fucking knew C, but his idea of a database schema was a spreadsheet. I can do it in practice, I can fucking do it in theory (fuck, I can turn it into fucking BCNF if you really want to, although I'd probably fucking quit first), in practice, and the only reason I don't make more money doing that instead of what I do is because I fucking hate it. Or did you mean just playing with Oracle settings? Been there too, although I finally convinced my boss Oracle was a waste of fucking money for what we needed.

And I'm also smart enough to know this: Upgrading to fucking solid-state disks is goddamn expensive. Buying them, maintaining them, replacing them -- Far, far, FAR more fucking expensive than a handful of experts and a few hundred hours of their time -- even at the godawful rates they're likely to charge.

And one last thing, moron -- if EVE's DB had enough wiggle room in performance to make a goddamn difference, it already would have shit itself and died by the time they were desperate enough to move to solid-state disks.

Engels: Judging by the proliferation and type of duping methods, it looks like their server-code wasn't all that great either. :)

[sam, an eggplant]

So who pissed in his cheerios this morning? What a spaz.

[Rasix]

So who pissed in his cheerios this morning? What a spaz.

Not enough fucks! More fucks now.  More fucks. More fucks. Common more fucks.  Ok, stop fucks.

[Margalis]

Are you two really arguing about how to optimize games in specific technical ways when you know nothing about the games at all?

[Megrim]

LOL, Internet.

Am i helping yet?  :-D

[sam, an eggplant]

He was. I said the same thing, that we don't know enough to do that. Then he got some sort of irritant in his vagina, and that's where we are right now.

[Trippy]

Trippy:

No, they are not.

I would say they are -- they have 10 times the concurrent users WoW does, they're not planning on ever sharding it so concurrent demands are simply going to grow, they collect a LARGE amount of detailed data, and the market is one of the primary DB drivers and it's fully integrated into the game. At any given moment, more people are utilizing EVE's market alone than are on the average WoW shard. Now, I'm aware Blizzard shares DB servers across multiple shards, but I can't see any way in which Blizzard's design calls for even a tenth as much DB usage as EVE's -- it's just an entirely different design.

If you agree that multiple WoW servers share the same DB then it's the same problem.

[Trippy]

I'm wondering if they've also fixed the Z axis issue, where flying over mobs, no matter how high up, would cause aggro down below. In any case, this is the first 'Sigil-free' patch for Vanguard.

It wouldn't be a McQuaid game if it didn't have Z-axis aggro issues.

[Raph]

This thread provides evidence that we don't need MMOs to make infinite threads. We should start a perfectly good one about normalizing DB tables and probably start massive flamewars.

[schild]

MY DATABASE IS FASTER AND MORE STREAMLINED THAN YOURS, RAPH.

That's right.

GAUNTLET THROWN.

Let's go.

Playground.

3:30PM.

[slog]

This thread provides evidence that we don't need MMOs to make infinite threads. We should start a perfectly good one about normalizing DB tables and probably start massive flamewars.

My current job is to gather requirements for reports that come from a  75 table relational SQL database thingy.  I tihnk the code monkeys hate me...

[kfsone]

Listen, numbnuts. I fucking understand DB tuning just fucking fine. I've had to do it on a number of occasions

Well pardon our confusion, you were the one that said

I find DB design to be highly boring, and will only grudgingly have anything to do with it.

Incidentally, EVE has often cited an improvement of 4000% / 40x, for which they give examples like

Another database performance measurement, "latches/total wait time" went down from 25,000 to 4,000

Hmm. I'm gonna have to break out calc for this. Nope. That's just an increase of 6.25x. I've never seen them give figures that confirmed anything above 8x improvement.

They're still gonna run into the dual overheads of MS SQL and running it under Windows. And from conversations I overhead with CCP staff at a couple of conferences, it still doesn't sound like they've taken the approach that banking, military and gaming industries use of putting proprietary/custom authoritative proxies infront of the database.

[CharlieMopps]

So wait... Eve has like the fastest, most expensive Database thingy on the planet... and yet it's still the most boring video game I've played since "Yu-Gi-Oh! Falsebound Kingdom"?

[Murgos]

Well, see, if you think Databases are exciting then you probably think Eve is exciting.

[bhodi]

And if you like excel, it's positively orgasmic.

[Endie]

And if you like excel, it's positively orgasmic.

Actually, if you like writing import routines to get csv stuff into Excel in a usable form, it's a peach.

[cmlancas]

And if you like excel, it's positively orgasmic.

I'm pretty sure they bundled EVE with MSOffice2007. I could be wrong though.

[Bunk]

I used to do maintenance on a DOS based Accpac db for a heavy equipment company, as part of my IT role. It consisted of running a maintenance routine that took about an hour, and then running a batch file that zipped the entire db up and copied it to another drive. That also took about an hour. I did this two hour procedure five days a week. I was doing this in 2001 - a DOS Accpac db.

It was awesome.

[Lum]

25 years ago I used to work in the data center of a mainframe for a medical insurance company. This consisted of users sending down punch cards requesting a given data tape full of records, at which point we would fetch the data tape off of a library shelf and load it into one of the tape drives.

That's right -- 25 years ago, I was part of a hard drive.

Data access techniques have improved since.

[Glazius]

And if you like excel, it's positively orgasmic.

Actually, if you like writing import routines to get csv stuff into Excel in a usable form, it's a peach.

Showing my terrible newbosity, what kind of CSVs need more than just opening the file with Excel (and thus implicitly accepting the import defaults)? Meaningful dates and times stored in strings rather than numbers?

--GF

[DraconianOne]

Data access techniques have improved since.

Yah.  These days, users just ring you up to find out what they want to know.  Phone > Punch Cards.