Designing Security

[Stephen Zepp]

This thread originated from a semi-hijack of this thread (Note: This url looks suspect, since it has "new" as a parameter--your milage may vary), where I posted a possible scenario to alleviate server congestion during initial launch of a MMOG in a particular geographical area, and Jayce attempted to flame me for my briefly stated suggestions. The following quote is the "current state" of a discussion about common security flaws in existing MMOG's, and where the common flaws reside based on a percentage ratio claimed by myself:

75%--either failure to validate client data sent to the server (packet overloading, packet modification, etc.) or poor client/server functionality placement.
15%--poor database design
10%--"unexpected" utilization or incorrect implementation of game functionality.

I'd venture to opine that 75% of security/game hack issues reside within failure to check for "illegally" modified game packets, and/or implementing functionality client side that is more correctly (from a security standpoint) implemented on the server side of the equation. Another 15 percent come from poor database design, and the remainder is "unexpected" utilization or incorrect implementation of game functionality.

And I'd venture to opine that you're talking out of your ass, without any related professional security experience to back it up.

And in that opinion, you not only would be wrong, but completely wrong. Good try however.

You seem to be representing yourself as knowing something about MMOG security as well, so care to share what you think are the vulnerability risk areas?

I suppose the thread is good and derailed, so I might as well.

Disclaimer: I'm no security expert - I'm a developer who (hopefully) knows what I need to to write code as securely as possible, and pays attention to the security world for that purpose.

- Illegally modified game packets:  If you are talking about Gear and its packet-shoving derivatives, that indeed was a problem for a lot of MMOGs at one point.  75% is pretty high though.  AC1 and UO are the only two that I know of that had major problems with them. Well, a bunch of FPSes too but I don't follow that world very closely.

-Implementing functionality client side : yeah , client in the hands of the enemy and all that.  I highly doubt any modern MMOG from UO onwards was/is dumb enough to fall into that one.  This is the canonical newb mistake, and none of the players in the mainstream MMOG space are exactly newbs to network gaming.

- Poor database design:  In a loose sense maybe.  I was about to say this is the most BS example you gave because database design has exactly zero to do with exploits, but I see in a later post you mean this to include dupe bugs.  That's not database design, that's the method and rules by which the info is persisted, which has nothing to do with the design of the data store itself.  The fact that you refer to it in such loose terms is what makes me think you don't actually have any real-world experience in this stuff.  If you do, fine, I could be wrong... tell us what these credentials are.

- unexpected utilization :  that's just a fancy way to say "exploit".  Tell us more, genius.

And no, I have no extended list of leet sploits for you.  I am not really sure why you ask.

I could go on, but this is really a topic for game design...  I now return you to your regularly scheduled thread hijacking....

Disclaimer: I am an experienced security assessment agent, with experience in both military and civilian environments, from physical/facility security through policies and procedures to penetration (physical and electronic) assessments, finally including process engineering for secure applications. I have a BS in Software Engineering with more than 15 years of applications development including classified systems , as well as 4 years of independent game development in a variety of dedicated client/server based cross platform environments. Currently, I am an Executive Producer for a MMOG in the prototype delivery phase.

- Illegally modified game packets:  If you are talking about Gear and its packet-shoving derivatives, that indeed was a problem for a lot of MMOGs at one point.  75% is pretty high though.  AC1 and UO are the only two that I know of that had major problems with them. Well, a bunch of FPSes too but I don't follow that world very closely.

75% is of course a personal opinion, based on individual research with an admittedly "statistically insignifigant" sample size. The problem is, there aren't enough MMOG's to provide a statstically signifigant population.

Speed hacks and other packet overload based hacks have been available throughout even recent MMOG's, including EverQuest, Shadowbane and SWG.

-Implementing functionality client side : yeah , client in the hands of the enemy and all that.  I highly doubt any modern MMOG from UO onwards was/is dumb enough to fall into that one.  This is the canonical newb mistake, and none of the players in the mainstream MMOG space are exactly newbs to network gaming.

You can doubt all you wish, Shadowbane was rife with client side calculations, mostly centered around ability/stat calculations. For example, the "Commander Rune" benefits, from movement speed buff to attack speed buff were calculated client side, and accepted by the server as trusted.

Based on "black box" observations (if you are a game developer of any type, you'll know what that means--for those that don't, it means observations of application functionality without any direct knowledge of implementation or design), Worlds of Warcraft utilizes client side processing for at least a small selection of processing tasks. Specifically, the camera controls (again, this is an opinion, but an educated one) are fully processed client side, which implies that all potentially observable game objects are transmitted to the client, and therefore subject to packet sniffing and decoding.

- Poor database design:  In a loose sense maybe.  I was about to say this is the most BS example you gave because database design has exactly zero to do with exploits, but I see in a later post you mean this to include dupe bugs.  That's not database design, that's the method and rules by which the info is persisted, which has nothing to do with the design of the data store itself.  The fact that you refer to it in such loose terms is what makes me think you don't actually have any real-world experience in this stuff.  If you do, fine, I could be wrong... tell us what these credentials are.

Fair enough--this was a generalization for the purposes of discussion on my part--although you can intuitively reverse-engineer what at least appears to be some pretty piss poor database design from observed results in not a few MMOG's, from trading exploits and dupes to issues like the one I mentioned. I'm willing to concede that this isn't exactly on point regarding database design, but it is absolutely on point regarding total object persistence. It's also why I limited it to 15% of the total vulnerability range.

- unexpected utilization :  that's just a fancy way to say "exploit".  Tell us more, genius.

Of course it is...there has to be a catch-all category whenever you are describing a range of probabilities. The intent here was to indicate that some "exploits" are really just unintended consequences of properly implemented (but most probably incompletely designed) game mechanics.

I have told you more, and since I do actually meet the definition of a "genius" as defined by Mensa, I'll take that last sentence as a compliment, instead of the insult that you appear to have intended. Thanks!

I could go on, but this is really a topic for game design...  I now return you to your regularly scheduled thread hijacking....

I would very much like for you to "go on", especially since this is now in the Game Design Forum. I do however apologize for the semi-hijack, but had you responded in a civilized manner instead of making a vain attempt at being "cool" by flaming, it wouldn't have happened in the first place...

Now let's see if this can become a content based discussion instead of a meager attempt to wave e-peens via flaming ;)

[Jayce]

I plan to respond to this, but I haven't gotten the time.  So I'll edit this post with some actual content soon!

[Evangolis]

I'm more interested in why the security issues existed.  The common belief is that it was poor upbringing in childhood, and the resultant moral deficiencies, but I wonder if there might not have been other contributing factors.  And how do you respond to the common agreement I heard among developers at the 2003 AGC that ultimately the data stream will always be comprimised at the client end, which they felt implied that a truely secure game was impossible, particularly outside the secure server MMO space?

[Roac]

That more or less meets my experience (dev, 7y). 

And how do you respond to the common agreement I heard among developers at the 2003 AGC that ultimately the data stream will always be comprimised at the client end, which they felt implied that a truely secure game was impossible, particularly outside the secure server MMO space?

Any dev who stated that needs a beating.  At least, "truly secure" isn't possible because it's really hard to find and kill every bug, but it is theoretically/technically possible.  That is, it isn't the technology that's limiting us, it's our ability to find and catch all programmer mistakes that does. 

It's equivalent to saying that a "truly bugfree" app is impossible.  No it's not - just that it's not cost-effective to produce a bug-free application of this size, since it's prohibitive to attempt to map every possible state.

[Stephen Zepp]

That more or less meets my experience (dev, 7y). 

And how do you respond to the common agreement I heard among developers at the 2003 AGC that ultimately the data stream will always be comprimised at the client end, which they felt implied that a truely secure game was impossible, particularly outside the secure server MMO space?

Any dev who stated that needs a beating.  At least, "truly secure" isn't possible because it's really hard to find and kill every bug, but it is theoretically/technically possible.  That is, it isn't the technology that's limiting us, it's our ability to find and catch all programmer mistakes that does. 

It's equivalent to saying that a "truly bugfree" app is impossible.  No it's not - just that it's not cost-effective to produce a bug-free application of this size, since it's prohibitive to attempt to map every possible state.

I think they may have been referring to the fact that if the client can decode the data stream being sent to it, a hacker will be able to do the same. You simply can't be agile enough in modifying your encryption methods/keys/whatever you are using to try to "protect" your data to stay more than several hours ahead of anyone hacking your client/decoding your data stream, so instead of trying (and therefore killing your customer satisfaction by endless patches, or performance loss due to decryption, etc.), I'm guessing that the comment was meant to suggest that you should design your system with the understanding from the beginning that your data is already compromised by the time the client receives it--and therefore, you need to protection your systems in different ways.

It's the design principle we are using: the client is totally untrusted, and the data stream is always suspect. It's kind of like the benefit of being a pessimist--you can never be dissapointed, only pleasantly surprised.

[Roac]

You're right, I was reading that in light of the first post.

Which brings up another point, one that's irksome.  Why is it people in MMOG industry (I hope not many devs) point out the fact that the client is "in the hands of the enemy"?  I know Raph made that line famous, but the issue at hand isn't even a technical one.  Franklin had it figured out years ago - "three people can keep a secret if two of them are dead".  The server can't tell the client any sort of secret and still call it such. 

It comes back to an issue of trust.  The 75% figure pointed out above is because the serve rmistakingly trusts the client to send good data in.  The issue here is that there is a potential security risk of the server trusts the client to keep a secret.  The client is not under your (the server) control, therefore not entirely trustworthy.  Security is handled not by trusting people, but by NOT trusting people and being highly cynical. 

[SirBruce]

It's important because it was one of the first "lessons learned" in the industry (refer to Farmer & Morningstar's Habitat article), and one which nevertheless needs to constantly be re-emphasized to new designers and programmers.  Because while the client is in the hands of the enemy, it's pretty much impossible for most MMOGs to operate without giving some trust to the client.  So it's important to identify in advance what things you have to let the client do, what things you can safely eliminate, what you will need to santiy check against on the back end, etc.

Bruce

[Jayce]

Disclaimer: I am an experienced security assessment agent, with experience in both military and civilian environments, from physical/facility security through policies and procedures to penetration (physical and electronic) assessments, finally including process engineering for secure applications. I have a BS in Software Engineering with more than 15 years of applications development including classified systems , as well as 4 years of independent game development in a variety of dedicated client/server based cross platform environments. Currently, I am an Executive Producer for a MMOG in the prototype delivery phase.

Cool, though you will probably get farther in this community if you get past the prototype delivery phase.  Especially if your company name starts with a "Red" and ends with a "Dragon Software".  But really, if we had a dime for every MMOG design/prototype, we could run all the Diaspora sites at a profit.

- Illegally modified game packets:  If you are talking about Gear and its packet-shoving derivatives, that indeed was a problem for a lot of MMOGs at one point.  75% is pretty high though.  AC1 and UO are the only two that I know of that had major problems with them. Well, a bunch of FPSes too but I don't follow that world very closely.

75% is of course a personal opinion, based on individual research with an admittedly "statistically insignifigant" sample size. The problem is, there aren't enough MMOG's to provide a statstically signifigant population.

Speed hacks and other packet overload based hacks have been available throughout even recent MMOG's, including EverQuest, Shadowbane and SWG.

The only hard part to this AFAIK is the overhead of testing each packet that comes in to see if it's within tolerances (both in terms of content and rate of arrival.  I do question why these even work if the server is (as it should be) the ultimate arbiter of location in the game.  Speed of packet arrival should not impact what the server thinks your current speed is.  But apparently a few MMOGs decided to base it off speed of packet arrival.  AC1 because it was an early attempt before a lot of these issues came to light, and Shadowbane because it was... Shadowbane.  No poor design decisions made by SB surprise me really.

I'm surprised at SWG though.  A lot of know-how was brought to bear on that development.

Worlds of Warcraft utilizes client side processing for at least a small selection of processing tasks. Specifically, the camera controls (again, this is an opinion, but an educated one) are fully processed client side, which implies that all potentially observable game objects are transmitted to the client, and therefore subject to packet sniffing and decoding.

First, It's World of Warcraft.  No "S".  Do you also say Wal-Mart's?

Second, of course the camera controls are client side.  It would be stupid to do that server side... can you imagine the network traffic every time you wanted to look around?

The real issue here is what the client knows -- the "secrets" that Roac referred to.  The fact of the matter is, anything you send to the client is potentially viewable, even if it isn't supported in your client.  If a stealthed rogue comes up to you and your client knows it, you can assume that there are players that know about it.  Even if your client is unhackable, someone could build their own client that "speaks" your server's language and display it all as stick figures.  And (as was mentioned in this thread) it's not feasible, even if it's possible, to build an unhackable client.

 you can intuitively reverse-engineer what at least appears to be some pretty piss poor database design from observed results in not a few MMOG's, from trading exploits and dupes to issues like the one I mentioned.

Somehow banks manage to avoid dupe exploits using transactional processing, but then again their demand for speed is much less.  And I would imagine concurrency also.  As I understand it, many MMOGs use some temporary store (a disk file or queue of some kind) rather than a relational database for on-the-fly operations because the overhead of that many simultaneous users and that many transactions would be too much to use a relational scheme directly.

Of course it is...there has to be a catch-all category whenever you are describing a range of probabilities. The intent here was to indicate that some "exploits" are really just unintended consequences of properly implemented (but most probably incompletely designed) game mechanics.

This is the main reason why I called bullshit on your original post.  It takes the tone that "this stuff is easy, developers are just stupid" which not only sells short the issues, but also fails Occam's Razor.  Poor design is a reality, but these problems are hard, and the people involved are not morons.  Well, most of them aren't.

I have told you more, and since I do actually meet the definition of a "genius" as defined by Mensa, I'll take that last sentence as a compliment, instead of the insult that you appear to have intended. Thanks!

Bragging about some nebulous genius classification isn't going to get you very far either.  In my experience, people who really are geniuses don't try to drop it in the conversation at any opportunity.  So you might or might not be one, but I have no way of knowing, and I don't care.

had you responded in a civilized manner instead of making a vain attempt at being "cool" by flaming, it wouldn't have happened in the first place...

I was pretty civilized for this board.  At least I didnt call you an assgoblin.  Turning "venture to opine" back on you was pretty smooth, if I do say so myself 

Now let's see if this can become a content based discussion instead of a meager attempt to wave e-peens via flaming ;)

I do like the idea that we can get some actual content going in this thread.  I will attempt to restrain myself when one of you assgoblins makes retarded comments from here on out 

[Stephen Zepp]

You are correct, in many ways I -am- frustrated with the fact that MMOG's continue to ignore security until the latter stages of implementation, instead of bringing it up front in the design phases. I do apologize that frustration came out as arrogance, but damn, it actually doesn't take a genius to think about the implications of various design practices.

The client side camera concept is one for example--you are absolutely correct, client side camera processing is both done for performance and latency minimization, but it also causes some extensive security issues. You mentioned the "stealthed player" issue, but depending on your game mechanics it can be even worse. If your combat mechanics are twitchskill based, the ability to move the camera without server control can give several informational advantages, and in some cases (if your muzzle point can be modified client side for example, and accepted server side), huge game winning advantages as well. This isn't the best example of what I'm getting at, but it's also difficult to present a hypothetical game mechanic to really illustrate the point totally "black box".

Actually, while it does seem counter-intuitive, most "game engine" implementations do actually base your movement on number of moves sent by the client, without real regard to the number of move events over a period of time. It's probably one of the implementation mechanics that has been around for so long in "standard implementation" of a simulation that it simply hasn't been refactored yet--from Everquest through Shadowbane and most MMOG's in between (hell, even the game engine we use is move event based, and susceptible to this hack), your move rate is given as a "move distance per move event", and not much regard is given to the moves per second received. Part of the main issue brought up in the thread originally--dev's tend to not worry about security until after the fact, and this type of design is by it's nature easily hackable.

If you are assessing incoming packets "inline" with your game/simulation processing, I would agree, that performance loss due to various checks could be a big problem. I wonder if it would be "latency transparent" (not noticable to your players) if you were to separate packet validation in a separate thread, or even a separate process...something to explore.

I liked your example about bank transactions, but two things to keep in mind:

--banks have been doing this a LOT longer than MMOG's have, and security was one of the highest functional design requirements from the beginning.
--ironically, bank transactions of various forms are much more vulnerable to some of the very same tactics game hackers use than you would think. One of the current fraud methods with credit cards currently is to set up a merchant account with a credit card company, and then flood the merchant account card authorization system with algorithmically generated credit card numbers (while a relatively complex algorithm, cc fraud operators figured out the major card number generation algorithms years ago), and it works.

Security vs functionality is an endless war that cannot be won by either side--the most secure computer is the one you never turn on, and the most functional application of any type assumes that no one will use the application in a method the designer did not intend. I do stand by the opinion however that even now, functionality gets hundreds of times the priority that security does, even with the complaints players continue to shout about regarding hackers and their effects on MMOG worlds.

[Evangolis]

(hell, even the game engine we use is move event based, and susceptible to this hack)

Do you plan to change this if you go to production?  If not, why not?  If so, why not in your prototype?

I do quite agree that issues like security need to be confronted upfront.  I would also place CS tools in the same category.  Users should be able to consult data about what they have done lately, such as items sold or otherwise gained or lost, logins/logoffs, failed logins, etc.  I should not need to contact a CSR to figure out that my little brother logged on my account and sold my shiny.  Security, customer service and administrative functions need to be in upfront or they will get cut in production, and added after launch, piecemeal and buggy.

[Jayce]

You are correct, in many ways I -am- frustrated with the fact that MMOG's continue to ignore security until the latter stages of implementation,

<snip>

(hell, even the game engine we use is move event based, and susceptible to this hack),

I'm confused here.  You are frustrated with the MMOG community's ignoring of security, but in your own prototype you are ignoring security?

Users should be able to consult data about what they have done lately, such as items sold or otherwise gained or lost, logins/logoffs, failed logins, etc.

These are nice-to-haves, but also nice to have is a ship date.

[Stephen Zepp]

You are correct, in many ways I -am- frustrated with the fact that MMOG's continue to ignore security until the latter stages of implementation,

<snip>

(hell, even the game engine we use is move event based, and susceptible to this hack),

I'm confused here.  You are frustrated with the MMOG community's ignoring of security, but in your own prototype you are ignoring security?

The point of my comment is that that is how game simulations are designed. For our prototype, we are using a game engine that is built like a large majority of the rest--and uses move event based input. One of the first things we have done is to change the input model to use completely different techniques, but the stock game engine uses move events itself.

For what it's worth, a prototype is a LOT different than production cycle implementation as well. And you may have hit upon a good point--it may be that some (many?) game devs turn their prototyping work into production cycle effort in an attempt to "save time", instead of throwing it away and starting fresh, once they've proven their gameplay.

[Jayce]

For what it's worth, a prototype is a LOT different than production cycle implementation as well. And you may have hit upon a good point--it may be that some (many?) game devs turn their prototyping work into production cycle effort in an attempt to "save time", instead of throwing it away and starting fresh, once they've proven their gameplay.

I see.  You are using a prebuilt engine that allows the vulnerability, but plan to refactor the parts that could use better security.That seems like a good approach.

"Make one to throw away" IIRC is Fred Brooks, The Mythical Man-Month.  In a later edition he says that this approach was wrong, but a quick google turns up mixed opinions on this.  It can be good when it's good, but the case can be made that refactoring the prototype as opposed to trashing it entirely is better.

The generally recognized worst case scenario, though, is when the decision is made to take the prototype to production with few or no changes because, hey, it works right?  Usually it works but is inflexible to changes, doesn't follow best practices, might be hard to read and maintain, etc.  But I suspect that this may unfortunately be the norm in some MMOG shops.

[Evangolis]

Users should be able to consult data about what they have done lately, such as items sold or otherwise gained or lost, logins/logoffs, failed logins, etc.

These are nice-to-haves, but also nice to have is a ship date.

Actually, logging data like this is essential to resolving CS issues, so you have to have it.  The additional step of making it available to players without CSR intervention is a cost savings and customer retention step.

CS support in an MMO is not nice to have, it is essential.  As Voltaire said of God, if it does not exist, it will have to be invented.

[Stephen Zepp]

Users should be able to consult data about what they have done lately, such as items sold or otherwise gained or lost, logins/logoffs, failed logins, etc.

These are nice-to-haves, but also nice to have is a ship date.

Actually, logging data like this is essential to resolving CS issues, so you have to have it.  The additional step of making it available to players without CSR intervention is a cost savings and customer retention step.

CS support in an MMO is not nice to have, it is essential.  As Voltaire said of God, if it does not exist, it will have to be invented.

I agree with this completely--it's not a "nice to have", it's one of the most critical things required to continue a profit margin. To pull a number completely out of my ass via research done a long time ago, I seem to remember a CS metric of "for every /report incident you have, you destroy the profit margin from another 2 player's subscription fees".

-Anything- you can do to keep your profit margins stable is a must have, especially for smaller subscription bases--you just can't afford the cost of the CSR staff, and their support if you want to stay profitable.

[SirBruce]

Back when I was teaching folks about computer security issues, I always emphasized first and foremost extensive logging.  All the security in the world won't do you any good if you don't know if it's working.  Moreover, your security will always be broken, eventually, one way or another.  Logging is essential for informing you of when it occurs, warning you quickly so you can compartmentalize the damage, identifying the attacker, and allowing you to reconstruct the exact nature of the breach after the fact.

Bruce

[Jayce]

Just for the record, I never advocated eliminating CS.  I know quite well that an MMOG is a service not a product and that all that logging is quite necessary.

However, the additional overhead of exposing it to the players, adding the UI, answering support calls related to the information (OMG IM BEING HACKED, no wait that was my little bro trying to guess my password) might not be worth it.

I probably don't have as much faith in Joe Player's ability to read the information and make the right calls as to its meaning as you, I suppose.  You can train a CSR to read the raw information -- the players are another matter.

[Stephen Zepp]

Just for the record, I never advocated eliminating CS.  I know quite well that an MMOG is a service not a product and that all that logging is quite necessary.

However, the additional overhead of exposing it to the players, adding the UI, answering support calls related to the information (OMG IM BEING HACKED, no wait that was my little bro trying to guess my password) might not be worth it.

I probably don't have as much faith in Joe Player's ability to read the information and make the right calls as to its meaning as you, I suppose.  You can train a CSR to read the raw information -- the players are another matter.

I agree that they wouldn't be able to interpret raw logs, and need to be spoon fed any data in an extremely intuitive (or at least well described) GUI environment. I admit I don't have any particular designs for the presentation of the information right now, but anything you can do to stop a CSR petition of any sort before it happens would be a good thing.

Now of course, if your information generates MORE CSR petitions, you are defeating the entire purpose!

[Evangolis]

Just for the record, I never advocated eliminating CS.  I know quite well that an MMOG is a service not a product and that all that logging is quite necessary.

However, the additional overhead of exposing it to the players, adding the UI, answering support calls related to the information (OMG IM BEING HACKED, no wait that was my little bro trying to guess my password) might not be worth it.

I probably don't have as much faith in Joe Player's ability to read the information and make the right calls as to its meaning as you, I suppose.  You can train a CSR to read the raw information -- the players are another matter.

I agree that they wouldn't be able to interpret raw logs, and need to be spoon fed any data in an extremely intuitive (or at least well described) GUI environment. I admit I don't have any particular designs for the presentation of the information right now, but anything you can do to stop a CSR petition of any sort before it happens would be a good thing.

Now of course, if your information generates MORE CSR petitions, you are defeating the entire purpose!

I'd suggest modeling your interface on an Excel spreadsheet.  Make it as dirt common and clear as possible.  Certainly you don't just spit back raw logs at them, but since you control the input data completely (or you should), creating clear output should be a pretty straightforward task.   As a business programmer, I've done this for any number of user systems, and it really isn't too tough, as long as the data parsing is minimized and the layouts are adhered to scrupulosly.

The advantage to exposing this info to players is that when the CSR call happens, the CSR is explaining what the player sees, rather than dictating to the player what happened.  The interation is supportive rather than commanding.  Having done years of technical support for a Consumer Call Center, I assure you it makes a huge difference in customer satisfaction.

[Lum]

Apologies for arriving late to this thread, and honestly the nested quotes are so tangled I no longer have any idea who said what. So on to the cutting of knots Gordian style!

75%--either failure to validate client data sent to the server (packet overloading, packet modification, etc.) or poor client/server functionality placement.
15%--poor database design
10%--"unexpected" utilization or incorrect implementation of game functionality.

The first two are related, vaguely, in so much as MMOs are client/server applications at their core and a poor design of either component will create problems.
The last will never be solved, unless you somehow manage to gain access to thousands of simultaneous full time QA testers motivated to break the game in obscure ways.

- Illegally modified game packets:  If you are talking about Gear and its packet-shoving derivatives, that indeed was a problem for a lot of MMOGs at one point.  75% is pretty high though.  AC1 and UO are the only two that I know of that had major problems with them. Well, a bunch of FPSes too but I don't follow that world very closely.

Gear is actually specifically altering the system's internal clock cycles to run faster, which means any client-side enforcement of game rules that are time-dependent will break. The greater the degree of sanity checking you do, the less responsive your client will be, since "lag prediction" is basically, well, moving that sort of thing to the client. So the question becomes where do you draw the line between what the server enforces and what the CSRs enforce.

-Implementing functionality client side : yeah , client in the hands of the enemy and all that.  I highly doubt any modern MMOG from UO onwards was/is dumb enough to fall into that one.  This is the canonical newb mistake, and none of the players in the mainstream MMOG space are exactly newbs to network gaming.

You'd be wrong, if the bug reports from the newest MMOs are correct. There is always the temptation to move logic client side, because server cycles are very, very, very expensive. Pretty much every MMO developer needs to learn this lesson personally, it seems.

- Poor database design:  In a loose sense maybe.  I was about to say this is the most BS example you gave because database design has exactly zero to do with exploits, but I see in a later post you mean this to include dupe bugs.  That's not database design, that's the method and rules by which the info is persisted, which has nothing to do with the design of the data store itself.  The fact that you refer to it in such loose terms is what makes me think you don't actually have any real-world experience in this stuff.  If you do, fine, I could be wrong... tell us what these credentials are.

I do database work on an MMO for a living. Credentials enough for ya?

A lot of the "database" problems with MMOs are less ones of systems design and more ones of architecture. There has been a distressing reliance in modern MMOs on relational databases as a cure-all panacea. The problem is that the more you pour into a relational database, the more you are making the classic tradeoff of power/performance for convienence. Flat files are old, boring, unsexy, hard to code for, and faster than anything in existence. My previous job, with a national consumer database firm, consisted in large part of a team that was porting data from an Oracle database to a flatfile system because Oracle was just too damnably slow.

- unexpected utilization :  that's just a fancy way to say "exploit".  Tell us more, genius.

They're called "bugs" actually, and every large scale project has them. Fixing them is why you have a live team.

[Margalis]

Just let me echo one thing here. I have a friend who worked on FFXI, according to him all the data was stored in flat files on some beastly NTFS machines. (And AFAIK there have never been major DB problems in FFXI) It seems to me that everyone that uses Oracle runs into problems, and spends a lot of money doing it. It got quite humorous with SWG and the "OMG we don't know what's wrong, but we have a call in to the Oracle support people!" that seemed to happen every half hour in the weeks after launch.

A big problem with something like Oracle is lack of transparency. It grinds to a halt and nobody knows why or how to fix it.

[Kageru]

I'd say there's an extremely high probability Blizzard are discovering this the hard way.

[Jayce]

-Implementing functionality client side : yeah , client in the hands of the enemy and all that.  I highly doubt any modern MMOG from UO onwards was/is dumb enough to fall into that one.  This is the canonical newb mistake, and none of the players in the mainstream MMOG space are exactly newbs to network gaming.

You'd be wrong, if the bug reports from the newest MMOs are correct. There is always the temptation to move logic client side, because server cycles are very, very, very expensive. Pretty much every MMO developer needs to learn this lesson personally, it seems.

Interesting.  I wouldn't have thought it, but I guess it goes to show there's nothing new under the sun.

I do database work on an MMO for a living. Credentials enough for ya?

A lot of the "database" problems with MMOs are less ones of systems design and more ones of architecture. There has been a distressing reliance in modern MMOs on relational databases as a cure-all panacea. The problem is that the more you pour into a relational database, the more you are making the classic tradeoff of power/performance for convienence. Flat files are old, boring, unsexy, hard to code for, and faster than anything in existence. My previous job, with a national consumer database firm, consisted in large part of a team that was porting data from an Oracle database to a flatfile system because Oracle was just too damnably slow.

Oh, your credentials are well-established here.  I was questioning Stephen Zepp's.

Regarding flat files:  what about XML?  It's new, non-boring, sexier, there are many well-established APIs for coding to it, and it's much faster than RDBMS.  The only problem I would see (and this applies to all flat files) is the one of write concurrency.  But I suppose you would have mentioned it if it were being used widely.  Why do you think it isn't?

[Lum]

XML is being used quite a bit on the client side for things like extensible interfaces.

On the server side, I haven't really heard of anything yet, although there's no real reason why not save the slight filesize overhead from the XML markup. I'm sure it'll get used eventually as XML knowledge spreads. Bear in mind the number of MMOs that have reached a production state are still really, really small.

[Stephen Zepp]

Getting slightly off topic here, but their has been a big trend to push for XML in a lot of message based systems in Healthcare IT, to the point where it's becoming a new data formatting standard for transmission.

Now, don't get me wrong, XML is great for the ability to dynamically define tags, but in general, when it takes 9+ characters (start tag, then end tag) to identify a single character data element, you are defeating the purpose of highly performance optimized data transmission--across the 'net, across a pipe on a platform, or inside an internal network, it just doesn't make sense to "pack" data for transmission by requiring a 9:1 (hypothetical obviously) ratio of tag to data.

I think that trying to use XML as an internal data representation format is a bad idea, simply from a performance perspective.

[Yak]

And, string manipulations are always expensive, so XML is indeed a costly transport mechanism.  Like you mention, the benefits are for working cross platform in an environment when the data dictionary is in flux.

Internally, where client/server versions and updates are controled and speed is a big factor, straight binary objects are preferable.. or using massive flatfiles for rapid data.  The best financial combo depends on speed needs/# of build in DBAs/# updates needed.

I think Stephen hit the nail on the head here.

External interfacing to the client using XML seems to work very well - if you want to encourage massive and quick front-end enhancements like Cosmos or information consolidation services like Thottbot for WoW that collects every location, quest, item, drop and MOB stats.  This also enables other easy front-end hacks that are more hacks of the business process (like the auto-fishing addon for WoW... just sit by a river and push a button, come back in 5 hours with 20 gold worth of fish).. and the next version is supposed to run to the nearest vendor, sell the fish for you and return to the fishing spot.  Instant economic implosion.

[Margalis]

Regarding flat files:  what about XML?  It's new, non-boring, sexier, there are many well-established APIs for coding to it, and it's much faster than RDBMS.  The only problem I would see (and this applies to all flat files) is the one of write concurrency.  But I suppose you would have mentioned it if it were being used widely.  Why do you think it isn't?

I would never store server-side character or item data as XML. I would make my own file format that is highly tweaked for the expected usage. Probably something like an intitial block that is a table of contents to the rest of the file data.

[Jayce]

Regarding flat files:  what about XML?  It's new, non-boring, sexier, there are many well-established APIs for coding to it, and it's much faster than RDBMS.  The only problem I would see (and this applies to all flat files) is the one of write concurrency.  But I suppose you would have mentioned it if it were being used widely.  Why do you think it isn't?

I would never store server-side character or item data as XML. I would make my own file format that is highly tweaked for the expected usage. Probably something like an intitial block that is a table of contents to the rest of the file data.

Sounds like you're talking about a hashtable-like format. Blizzard uses that client-side (their MPQ files). Only problem with that is that you have to rewrite the TOC everytime something changes.  For server-side dynamic world-information data, thats like, a lot.

I disagree with you guys on XML.  There is overhead from the tags, especially if they are long and human-readable.  However, I'm not talking about transferring this over the wire.  This is all in memory and on disk, and don't forget, you can serialize the XML into a binary representation in memory.  The advantages include: well-defined and pretty bulletproof APIs to work with the data (create, read, update, delete), the fact that you can create a relational scheme without the overhead of a RDBMS, and the ease of parsing it, facilitating logging tools and the like for CSRs.

If there's some obscure bug in your special file format, and your server crashes once a day because of it, that's a problem.  If you spend a month of dev time writing your special file format, when you could buy or find a free stable XML parser, that's another problem.

I have used XML for something similar in a project involving live field data, so I've seen and worked with the performance implications myself.  Just to show that I'm not ENTIRELY talking out of my ass.   :-D

[Typhon]

The lure of distributed computing, especially by shops with lower budgets, has got to be huge.  I don't think shops that do this are (necessarily) dumb, I think they are desperate.  Why not trust the client (for some data) for the moment, and have another process review data later?

Is it acceptable, from a business perspective, to catch the hack later?  example - create processes which check client communicated data against invariants (distance moved, cash made per transaction, etc) in a lazy fashion, and and gerenate reports to the CSR team any violations?  It seems to me that companies are doing this sort of thing already, but are struggling more with the business reality of cutting off clients.

Maybe the HackCatcher2000 first reports the infraction to the cheater directly next time he logs in, leaving the CSR team out of the loop until after the initial warning.  The fact that it's a lazy check creates uncertainty on the cheater side (are they not catching the hack? or have they just not busted me yet?).

[sinij]

Is it acceptable, from a business perspective, to catch the hack later?  example - create processes which check client communicated data against invariants (distance moved, cash made per transaction, etc) in a lazy fashion, and and gerenate reports to the CSR team any violations?  It seems to me that companies are doing this sort of thing already, but are struggling more with the business reality of cutting off clients

This just doesn't work well. If you can cheat someone will cheat, if you get busted for it people will just use throwaway accounts and laundering techniques to avoid it. Example: When big server-border ‘blackhole’ dupe hit UO in 99 developers started tracking it and banning people for using it without fixing problem. It wasn’t easy to fix this problem due to problem being in a data transfer mechanism between sub servers. Banning didn’t discourage duping at all – player just got really proficient in using fake accounts and ‘laundering’ resources by repeatedly re-stacking to renew object ID and overflow any kind of history that was tracked. When tracking was improved to compensate for this one group of dupers just started randomly spreading portions of incriminated gold to general population to increase ‘civilian casualties’ in carpet bans and another group came up with elaborate method to launder it.

[Jayce]

Is it acceptable, from a business perspective, to catch the hack later?  example - create processes which check client communicated data against invariants (distance moved, cash made per transaction, etc) in a lazy fashion, and and gerenate reports to the CSR team any violations?  It seems to me that companies are doing this sort of thing already, but are struggling more with the business reality of cutting off clients

This just doesn't work well. If you can cheat someone will cheat, if you get busted for it people will just use throwaway accounts and laundering techniques to avoid it. Example: When big server-border ‘blackhole’ dupe hit UO in 99 developers started tracking it and banning people for using it without fixing problem. It wasn’t easy to fix this problem due to problem being in a data transfer mechanism between sub servers. Banning didn’t discourage duping at all – player just got really proficient in using fake accounts and ‘laundering’ resources by repeatedly re-stacking to renew object ID and overflow any kind of history that was tracked. When tracking was improved to compensate for this one group of dupers just started randomly spreading portions of incriminated gold to general population to increase ‘civilian casualties’ in carpet bans and another group came up with elaborate method to launder it.

I agree, but I don't think it's as simple as that.

You can never design a completely hack-proof system. The best thing you can do is try to discourage the average player from using a cheat if they happen to find it, but the ones who are determined to cheat will always find a way.  That's true in computer security in general as well as crime in general.  In this view, the "HackCatcher2000" has some merit.

But I think it's overbalanced by the fact that you are giving the players too much information.  If you report to the player each time they are detected as maybe cheating, you get two things: 1 - legitimate players contacting CS to report false positives in the fear they are being unfairly targetted, which doesn't save you any CS time/money; or 2 - hackers using it as a tool to see what can be detected and what can't.  Even if a few accounts get banned in the process, ultimately they find a hack that's untraceable.

[Typhon]

2 - hackers using it as a tool to see what can be detected and what can't.  Even if a few accounts get banned in the process, ultimately they find a hack that's untraceable.

I didn't even think of this.  I didn't think of it so much, in fact, that my gut reaction to reading it was, "henh?! that's ridiculous, who has the time or motivation to put that much effort into cheating".

I guess I suck at security.

[Strazos]

I didn't even think of this.  I didn't think of it so much, in fact, that my gut reaction to reading it was, "henh?! that's ridiculous, who has the time or motivation to put that much effort into cheating".

I guess I suck at security.

Catass, anyone? When you're unemployed, in your mid-20's, and living in mommy's basement, time is not a problem usually.

[Pococurante]

Because selling virtual items is forecasted to be a multi-billion dollar industry and even the game service providers (UO, EQ2) can't resist getting involved.

We're well past the time of a few shut-ins looking for new ways to get kicks.