Ancient history - AC2

[Arthur_Parker]

Not sure if there's interest enough for a thread on AC2 again, but I was entertained.

What’s a QA team without a spec?

Asheron’s Call 2 launched with over 3000 severe yet unrecognized bugs.
...
Many of the crafting recipes were about ten times harder to create than intended. The creatures that dropped the needed parts spawned incredibly rarely, due to an oversight.
Many of the quests could easily become broken if the steps of the quest were done in the wrong order. Fixing them then required the assistance of a customer service representative.
Almost every skill in the game was broken in some way. Some skills literally did nothing; others were too costly or too powerful; some started out super strong and then got weaker as you leveled up the skill.
...
It was the extremely talented Jesse Kurlancheek (a.k.a. “Devilmouse”) who was responsible for the skills in AC2. Because of the agreement Turbine had made with their publisher Microsoft, he was obligated to create at least 600 skills for the game before it shipped, broken into 30 different skill trees. Problem was, he only had three months to design and implement them all. Jesse famously told his boss, “I can either implement the skills, or I can document the skills, but there’s no time to do both.” And he was right. But they chose to implement instead of documenting, and the result was tragedy.

Without any idea of how skills were supposed to work, QA never looked at any of them. To be fair, QA should have at least poked around with them some, but they just weren’t motivated to spend any time on them because the designer would almost always say “no, that’s not a bug, that’s how it’s supposed to work!” So the QA team spent their time wandering around looking for objects that were hovering a tiny bit off the ground, instead. At least those were irrefutable bugs!

The lack of specs was devastating. When the game launched, the live team needed to get the game into a maintainable state. Without specs, we had no idea what Jesse had intended. Worse yet, a few months later, not even Jesse remembered what the intention of each skill was. So we created elaborate analysis software in order to locate skill deviations, and slowly reverse-engineered the intentions, like archaeologists exploring an ancient culture. The lack of specs cost us thousands of man-hours. It made us look like total dicks, too, when we fixed the outrageous bugs we found.

The “reap” abilities in AC2 stole the health of an enemy and gave it to you. However, as is typical of this sort of power, you couldn’t steal more life than you were missing. So if you were only missing 100 health, you could never steal more than 100 from the enemy. Tragically, there was an accidental minus-sign in the implementation. If you reaped somebody and you were fully healthy, instead of doing no damage, you did double damage. Reaps were the most effective attacks in the game, provided you hadn’t been hurt too much. It made PvP in particular a nightmare, and players were howling for us to fix it. When we fixed it, though, several classes became unplayable; it turns out they had only been fun at all because of that broken skill. So then we had to buff those in various ways, until the next major bug was discovered, which threw our rebalancing out of whack again. And on and on, over and over, a constant dance of chaos and confusion, for over a year, before we had created our own specs and could work towards stability.

If only the skills had been affected this way, we could have dealt. But the lack of specs permeated almost every aspect of the game. The quest areas were about half-documented; the monster spec was nothing but one extremely complex spreadsheet. It’s not like the AC2 designers didn’t know how to make specs, of course: they were given impossibly small time windows to do their work, and they did their best.

If our publisher had allowed us, it would have been so much better for AC2 to have launched with only half the classes, but with docs for all of the classes. Then the live team could just implement the docs, rolling out new things every couple months, and make Turbine look super productive instead of super incompetent.

[UnSub]

You posted those other AC2 messages in the WAR thread by mistake?

EDIT: however, all of that was interesting to read. Explains AC2's current location in MMO history very well.

[Arthur_Parker]

No, I just thought there might be some general interest in what went wrong given Citan has now gone on the record.  But I'll repost the earlier ones below.

I said before that WAR in some ways reminds me of AC2 and the balancing that went on after release.

Found Eric "Citan" Heimburg actually discusses AC2 balance here which I found interesting because years later I still think he missed the point.

Of course, it doesn’t just happen that you hop onto the Live Team and suddenly you’re making game-design changes. At first there are a lot of smart and talented people at the helm, helping you learn the ropes, making the hard decisions for you, keeping you from doing stupid things. But inevitably they are pulled off to other projects, and somebody relatively junior gets the helm. That’s how I got to be in charge of balancing AC2’s classes.

Fortunately, I had a decade of engineering experience and understood how to tune complex systems. I wrote analyzers, modeled usage patterns, and made corrections.

Unfortunately, my approach did not take the “human equation” into consideration very well.

Learning to Balance the Human Equation

I found that the Feral Intendant class was 30% overpowered, and that’s why so many people were playing a Feral Intendant. Yet somehow, reducing the power of the Feral Intendant to the correct level did not suddenly make the game more fun… thousands of players were complaining and nobody was telling me they were happy about the change. Weird! I double checked my calculations. They were correct. So what had gone wrong?

Turns out that the people who played the other classes available to that race had taken on an “underdog” mentality. The people who played Claw Bearers liked that they were woefully underpowered compared to Feral Intendants. It was like playing the game on Hard Mode. And the people playing Feral Intendants liked playing on Easy Mode. In balancing the game I had failed to understand the needs of the people playing it. I just ham-handedly fixed the equations, instead of solving the problem with the finesse it needed. It was one of my more serious missteps. (And it’s a great example because I think it’s pretty obvious in hindsight. Most mistakes were much more subtle.)

But man, what a fast way to learn! After just a couple years of that, I became a good game balancer. The constant feedback loop helped me learn from my mistakes in a matter of weeks! Compare that to developers on traditional games, who must wait until the sequel ships before they get to try their hand at balance again. That’s why working on a live team is such a fast way to learn your craft: the feedback is so much faster than any other gaming platform, that it accelerates learning by dozens of times.

But AC2 cost millions of dollars to create. Turbine didn’t create it as a tool to help me hone my design skills, that’s for damned sure! How did I get to do it? Simple: the designers who would have done it were burned out of working on AC2, and were called away to work on the important New Project. AC2 wasn’t a blockbuster hit, so it didn’t make sense to use the rock star designers on it. Better to let the B team step in.

To contrast from my memory, Lugian's were overpowered but fun, got nerfed, Feral Intendant were overpowered but fun, got nerfed.  As one of the other classes you start to wonder if you are now the "strongest class" and start to consider if you are actually having fun, because you just know what's coming.

More on AC2, Taming the Forum Tiger

On Asheron’s Call 2, we determined that about 10% of the playing audience read our website or forums (it spiked on patch days, to a whopping 15%).

That's interesting.

This small percentage of people are not randomly pulled from your userbase. They tend to be very similar to each other and not very representative of the rest of your player base. That means you can’t use them to judge the quality or merit of ideas or implementations. But of course, the ones who do post are your most vocal users, and it would be foolish to ignore them.
...
The pretend quitting: “This is the last straw. The new updates on the test server are a mockery of everything this game was supposed to be about. If this change goes live, I will be forced to cancel all three of my accounts forever. Since I am guild treasurer, I’m sure most of my guild will quit too, and we’re the only decent PvP guild on our server, so PvP on our server will entirely die out. I wish I didn’t have to do this, but they’ve forced my hand.”

This is a classic. On AC2, I did my best to correlate people who said they were quitting to people who actually quit. Almost nobody who said they were quitting actually quit, and the few who did didn’t stay gone long: they entered a rebound cycle and came back pretty quickly. Most often, they didn’t leave at all.
...
Being exposed to the forums causes real damage to developers and moderators. For instance, the QA employee in charge of WoW’s test-server forums is so badly burned out that he routinely deletes threads full of information and bug reports because users just sicken him.
...
Players tend to whine and moan most loudly on the game’s official forums, because they are putting on a show to try to convince developers to change things. Behind closed doors in guild forums or fansite forums, posters tend to be considerably more upbeat. It’s very surprising to see the change of tone. It helps put things in perspective.

Don’t force your developers to read the forums. Instead, have the community send a weekly digest of posts to the team. Make sure the digest has as many upbeat or informational posts as it has complaints (even if the actual ratio on the forums is much different). Remember that forums are not representative of the user base so there’s no reason to expose developers to all the hate and anguish there. A taste is enough to get the idea.
...
Never post while drunk
...
The early AC/AC2 forum moderators at Turbine had a drinking game they’d play sometimes when reading forums late in the evening: drink a shot for anybody who says “slap in the face”, chug if someone says their guild is quitting, and if somebody predicts the game will be dead in a month, the whole room has to drink. I’m sure versions of this game exist at many companies.

Some of that is just 

[Xilren's Twin]

While i dont think any of what he said is neccessarily surprising, what continues to amaze me is that this pattern seems repeated in one form or another for almost every failed/troubled mmorpg made (which lets face it, is MOST of them).  Considering how many retread developes are in this business, why doesnt anyone actually seem to learn from these experiences?  Is it only because these things are "games" so that they are not thought about they way you would a commercial application?    Is it the scale?  The large amount of users but small amount of revenue per user?

Inquiring armchair-developers-who-always-think-they-can-do-better want to know!

[Reg]

The games industry has lots of prima donna superstar developers but very few decent project managers and MMOGs are too big to be carried by one or two talented people in this day and age.

[chargerrich]

I thoroughly enjoyed the read!

Brought back memories. Played AC2 from launch to about 6 months from close and did go back for the last month. A flawed game to be sure looking back, but I had fun (will always remember my groups first Lord Deimos kill... wow).

Thanks for the link!

[chargerrich]

Almost every skill in the game was broken in some way. Some skills literally did nothing; others were too costly or too powerful; some started out super strong and then got weaker as you leveled up the skill.

AHA! I knew the bounty hunters skill "lucky charm" did nothing. For literally years there was rumor and speculation that ranged from "improves hit or crit" to "increases drop rates" to "does nothing" and "is a developer joke".

Now we know... i think 

[Mrbloodworth]

While i dont think any of what he said is neccessarily surprising, what continues to amaze me is that this pattern seems repeated in one form or another for almost every failed/troubled mmorpg LARGE SOFTWARE PROJECT WHERE HUMANS AND TEAMS ARE INVOLVED made

FIFY.

I think turbine clearly learned from this. AC2 became LOTRO.

[Jamiko]

As someone that won a lifetime account (finally win something and this is what I get?), I was there through the whole thing. The game obviously had some huge issues, but none I think were worse than the Microsoft no-chat fiasco. Completely changing most of the classes did not help retain members either, but after all was said and done it was actually a rather playable and fun game at the end. Still, too little too late.

As players, we could tell the live team had virtually no clue what the design team had done. It was clear that they first had to figure out how everything worked (or didn't work) and then make it so it worked as they wanted. It was a long and difficult process for the players to go through, and probably was not a lot of fun for the live team assigned to it either.

[UnsGub]

The games industry has lots of prima donna superstar developers but very few decent project managers ...

Agreed.

It also pays way below what is need to get consistent abilities across all the function groups involved in developing large software projects.

[sam, an eggplant]

I like how he's abrogating all responsibility and completely blaming the publisher for their terrible game. He claims it was impossible to work within their publisher's unreasonable constraints and produce a game that actually functioned. It's very clear there's a great deal of bad blood with microsoft, but ultimately it's their job to work within those constraints and they should have scaled down design and content creation requirements accordingly.

[Arthur_Parker]

He made a lot of terrible decisions when he gained control of AC2, so I disagree with a lot of what he saying but I'm linking it for the historical facts and also because I found a lot of it funny.

But here’s the weird thing: WoW is exhibiting the same symptoms as AC2 did when I was doing the designing. The B team is in charge.

"Never post while drunk" indeed.

[K9]

Interesting read; I particularly liked his comments on forums, and could easily believe the comment about the WoW Q&A.

[kildorn]

He's projecting a lot of the whole "nerfing fun classes" thing onto other games.

If you're having fun, you're not nerfbait. If you're blatantly better than the rest of the game in some way, you are. Some people just find it fun to be playing the "best" class/spec/whatever.

That said, there's a lot to be learned about trying to implement something without documenting it. That goes for any industry, and at least everywhere I've worked has been the Bus Rule. Essentially "what if the guy who deployed X gets hit by a bus, who knows it enough to maintain it?" If you can't answer that question properly, it ain't getting deployed.

[Hayduke]

Interesting posts.  I've never used official forums in any of my time in mmogs, but I was surprised the number was that low.  The concept of a community digest being produced sounds more interesting.  But I think companies like SOE and Funcom have already done things like that by appointing advocates from the community.

[kildorn]

Interesting posts.  I've never used official forums in any of my time in mmogs, but I was surprised the number was that low.  The concept of a community digest being produced sounds more interesting.  But I think companies like SOE and Funcom have already done things like that by appointing advocates from the community.

Community digests to the developers are, I think, common. I liked the  of providing equal amounts of happy posts in order to not get the devs all down and such.

[Calandryll]

It's a very good article and illustrates the need for good documentation. When I was on UO I was part of a team that added ongoing content and fiction and let players make changes to the world. We did game updates once a week for 8 weeks straight, then took a break, and then did another 8 week string of publishes. Once we started the first publish we HAD to hit the date for each subsequent publish otherwise the story would get off track. There was no turning back once we got started.

The designers were required to work with QA to help build test plans and even write scripts to help QA test. So for example if a publish included a whole bunch of new items, we created a script for QA that allowed them to create a bag full of each item so they didn't have to loot it off a monster or create each item one at a time. We added scripts that "barked" text when a monster kicked off part of it's AI so QA good see that it was working. Etc. We did whatever we could to help QA test our work and give us good feedback.

The funny thing is, yes, this did slow down development at first. I got a lot of flak for that at first. But at the same time, the designers on my team spent less time dealing with issues that weren't bugs and were able to test their own work better, resulting in less rework and bug fixing. By the time we hit a stride, I guarantee we were developing faster using this method than we would have if we had just left QA to their own devices and designed/coded as fast as we could.

During the 52 weeks that we ran the Scenarios, we had 26 game publishes. Many of them were pretty big and included new systems like Virtues, crafting updates, new monster AI, etc. Once we started a string of publishes, we never missed one. Of those 26 publishes, we had one instance where we had to bring the servers down to fix a major bug (it was something I did sadly :P) and other than that, we had no server crashes, no client crashes, and no hot fixes due to content in those publishes. That whole experience convinced me that developers and QA need to work closer together and that while it does slow things down at first, the long term benefits are astounding once you get rolling.

[WindupAtheist]

I was playing during that whole period and it was indeed awesome. I remember I was on Great Lakes for the Yew scenario, and we were one of the few shards to manage to save the city. Good times. And yeah, it was huge world-changing stuff delivered consistently, largely devoid of technical problems, and all things considered pretty rapidly.

[Jayce]

That whole experience convinced me that developers and QA need to work closer together and that while it does slow things down at first, the long term benefits are astounding once you get rolling.

This seems like a fundamental problem that developers and their management of all areas hit over and over.  As long as we've been doing this, it's still somtimes a hard sell to convince people that doing additional work up front saves time on the back end, and reduces the number of times you look stupid for releasing something broken.

At first there are a lot of smart and talented people at the helm, helping you learn the ropes, making the hard decisions for you, keeping you from doing stupid things. But inevitably they are pulled off to other projects, and somebody relatively junior gets the helm.

I don't think the situation is as black and white as he's stating.  Sure it happens, but it supposes that the B team are not senior, smart and talented.  There is no fundamental reason the B team has to be less good at their job than the group that's off to work on the next-gen thing. It seems to me that the reason they might be less good would more likely be poor hiring, premature promotion, the Peter principle/halo effect, or something of that type.

[Rasix]

That whole experience convinced me that developers and QA need to work closer together and that while it does slow things down at first, the long term benefits are astounding once you get rolling.

Developers, QA, and information development need to all work together.  Wait, do you guys even have ID teams (or a single person) in game development?

The game obviously had some huge issues, but none I think were worse than the Microsoft no-chat fiasco.

Single largest reason that I quit.  Having that stupid chat server go down all of the time was probably the single most aggravating thing I've experienced in a MMO next to "Turbine brand rubberband lag".  Of course, if the game wasn't so broken, I likely wouldn't have played it as long as I did.  The grind in that game would have been absolutely nauseating if you couldn't exploit pathing and quest issues.

[Khaldun]

It's a really interesting article, and of a kind that MMO producers and consumers both desperately need more of: honest, interesting and public postmortems.

On the forums point, though, it's almost a cliche, and I want to point out three things that somehow don't seem to come up whenever devs take refuge behind this idea that forum readers and participants are totally unrepresentative of the general population of players:

1. If you looked at late 19th/early 20th Century societies with relatively low literacy rates and noticed that there were relatively few newspapers in many places and that they had low circulation numbers, you'd sometimes be making a big mistake to assume that very few people were aware of the content of newspapers in those societies. Because what often actually happened is that one literate person would read a paper and either tell his entire extended family and circle of acquaintances about what was in it, or would literally read it aloud to people. I think it's that way with a lot of forum or online content. Maybe not that many people read offical forums, but the person in a guild who does often tells a lot of people in-game about its content.

2. Similarly, forum content disseminates into lots of spaces where forum managers aren't directly aware of it: links in non-official forums, into guild boards, and so on. If you really want to know how many WoW players are in some sense participating in discussions of the future prospects of the game, you'd need a global readership census that would include the official forums, WoWInsider, Elitist Jerks, and so on. Combine 1 & 2 and way more than 15% of the playerbase is in some sense participating in and aware of conversations about the current state of a MMO.

3. Devs who comfort themselves by saying that drama queen posters who threaten to quit are just full of hot air can blind themselves to the verdict of a "silent majority", which may be indirectly influenced by online discussions as per 1 & 2. I mean, think about AC2: there's something really dumbly smug about a live management team patting itself on the back with a drinking game that dismisses drama-queen forum complainers while the game is bleeding out subscriptions. But reading the postmortem, it may be that the live management team knew from the very first moment that they were dealing with a dead duck that couldn't be saved anyway, I dunno.

[Nija]

Of course, if the game wasn't so broken, I likely wouldn't have played it as long as I did.  The grind in that game would have been absolutely nauseating if you couldn't exploit pathing and quest issues.

I must say... We made, absolutely, the most out of what we were presented with in AC2. There were definitely some fun times, but none of the fun times in AC2 were intentionally caused by Turbine. It was a lot like UO in that regard.

[Soln]

I really want to read that kind of article for SWG.  Cal?

[Morfiend]

I really want to read that kind of article for SWG.  Cal?

I second this.

[PalmTrees]

No one knew what to do because nothing was written down? Man that's hilarious in a sad way.

[Stormwaltz]

But reading the postmortem, it may be that the live management team knew from the very first moment that they were dealing with a dead duck that couldn't be saved anyway, I dunno.

"We're being shipped into 'Nam with no weapons."
-- Elliot, contemplating Live

[Calandryll]

I really want to read that kind of article for SWG.  Cal?

I wouldn't be the right person to write something like that. I was only on SWG for three months, well after launch, and not on the development side of things. I've heard a lot of stories, but I wouldn't feel comfortable writing them because I don't have any direct experience with pre-launch and early launch. I also left before the NGE was even first drawn up. Heck, I was on AC2 longer than I was on SWG. Although my experience with AC2 was being told to shepherd the shut down because nobody else would do it. Annoyingly people (even some at Turbine) actually thought it was my decision (it wasn't) just because the Community Person and I where the last people standing.

That's why I didn't comment on the specifics of AC2 in Eric's update. Although I don't quite agree with his impression on forums. I agree with a lot of it, but disagree with some of the nuances. Khaldun has it right. It's very dangerous to look at that 10% as only 10% (and actually 10% is high compared to what most products get on their sites) and it's also not quite accurate to say that the people on the forums are similar to each other. They actually are more of a cross-section (once you get past the edges of the bell curve) than he thinks. It's better to look at forums as a cross-section, albeit with a slight hardcore tint, than not representative. That said, overall I do agree that forums are at their best when they are used as a means for players to communicate with each other rather than a way to scream for Dev Team attention. But that has more to do with your Community plan and Community Manager than anything else. It's all about setting and maintaining expectations.

In a more general sense though, when a game under performs it's usually for a lot of reasons. There are management issues, technology issues, poor design decisions, publisher relationships, not solving potential issues (leaving things TBD) for too long, poor project management, and a whole bunch of other things we never hear about. People like to point to one or two things, but more often than not there are a lot more.

[Soln]

thank you Cal.  I remember you pre-NGE, I think it was about the time Tyrant (Walton) joined?  Thx.

[Arthur_Parker]

I always wondered how "lets not have npc's" managed to fly at an early design meeting for AC2.

Edit I should maybe add I hadn't done more than glance at Elder Game before.  But having read quite a bit now, maybe my selective quoting was unfair.  I don't agree with him on forums and still think he's overly obsessed with balance, but there's a lot of interesting stuff there.

[Khaldun]

I always wondered how "lets not have npc's" managed to fly at an early design meeting for AC2.

Edit I should maybe add I hadn't done more than glance at Elder Game before.  But having read quite a bit now, maybe my selective quoting was unfair.  I don't agree with him on forums and still think he's overly obsessed with balance, but there's a lot of interesting stuff there.

Let's put it this way. If I were a smaller development house working on an MMO, I'd hire him to take a look at a pre-alpha design and give his opinions about how to proceed. Even though I think he's wrong about some things, or coming to the wrong conclusions from correct insights.

I can totally see how the "no npcs" thing would have sounded like a great idea. The problem being, of course, that the AC2 devs didn't really seem to realize all the other things that they'd need to do to make a good idea (like, a much fuller suite of player-content creating tools, or a dynamic world design that generated procedural content in response to player actions, or whatever). So it was just one of those dumb things in the end that too many MMO designers think actually distinguishes their product from someone else's. (Decapitation and boobs does not make WoW into not-WoW!)

[Jayce]

That said, overall I do agree that forums are at their best when they are used as a means for players to communicate with each other rather than a way to scream for Dev Team attention.

Interesting.  One thing that comes to mind reading this is that the DAOC model of official communication (The Herald) is better than the Blizzard blue-post style.  The Blizzard style leads players to believe that their drama queening on certain official forums will get their favorite change made, where the DAOC model - blog-style - gives no such illusion.

I wonder what DAOC's official forums would be like if it had them.

[tmp]

I wonder what DAOC's official forums would be like if it had them.

Well, they have official WAR forums now

[UnSub]

You can have both your announcements and your dev forum comments.

Of course, this sometimes backfires, but it can also work very well. In general if the dev comments are regular, there tends to be less preening / tantruming for their attention.

[Stormwaltz]

I always wondered how "lets not have npc's" managed to fly at an early design meeting for AC2.

It didn't, when the decision was communicated to the design department.

[Arthur_Parker]

Ah, I guess that explains things a bit.