Mark Jacobs Interview Regarding AoC and Hellgate (Why They Tanked)

[Calandryll]

Have the devs actually play the quest and they'll probably figure it out.  Does Quest A take 30 minutes and give 5K xp, does Quest B take 5 minutes and give 4K xp?  Is the item reward for Quest A crap compared to Quest B?  Is it broken?  Does it just flat out suck as a quest to begin with? 

Don't need too much in metrics to determine that...

That only works if your devs have the same opinions on the game as your players, play the game the same way as the players, and know the game as intimately as the entire playerbase combined. When you have maybe 50 developers and 100s of 1000s of players, that's almost never the case.

[Tmon]

Wouldn't a poll at log in get just as much feedback as digging through the forums?  Set it up so the player can skip it if they want but give a small reward or something if they take the poll.  I suppose you could even add a comments section and use that to identify players that give consistently well thought out feedback who could then be included in more in depth polls or what not.

[Calandryll]

Wouldn't a poll at log in get just as much feedback as digging through the forums?  Set it up so the player can skip it if they want but give a small reward or something if they take the poll.  I suppose you could even add a comments section and use that to identify players that give consistently well thought out feedback who could then be included in more in depth polls or what not.

Not to be argumentative, but no. :)  Polls can be a good tool at times, but like metrics they are only one tool of many. Polls only work if you know that the answer is one of X number of issues. If you don't know the issue, then the poll won't help. I personally hate polls as a means of gathering feedback. They're like the pun (lowest form of humor) of community tools imo.

Honestly, message boards aren't that hard. You just have to commit to doing it and hire the right people. Also, message boards are about a lot more than just gathering feedback, there are messaging and community building benefits too.

Keep in mind, none of my comments so far are for or against official forums. What I've written so far holds true regardless of whether you host forums or not. If you are genuinely interested in what your players think and how they are playing the game, you have to talk to them.

[Ratman_tf]

Have the devs actually play the quest and they'll probably figure it out.  Does Quest A take 30 minutes and give 5K xp, does Quest B take 5 minutes and give 4K xp?  Is the item reward for Quest A crap compared to Quest B?  Is it broken?  Does it just flat out suck as a quest to begin with? 

Don't need too much in metrics to determine that...

That only works if your devs have the same opinions on the game as your players, play the game the same way as the players, and know the game as intimately as the entire playerbase combined. When you have maybe 50 developers and 100s of 1000s of players, that's almost never the case.

That's what QA is for. 

[cevik]

That's what QA is for. 

But Calandryll's point, and it's a good one, is that QA will never (no matter how good they are) ever get the coverage of 100,000 eyes looking at it.

[Khaldun]

Polls and surveys can never tell you what some version of qualitative social science can tell you: how things feel, what experience is like, what the general attitude is, how players are communicating with each other about the game's character. If your MMOG is doing well for the moment, polls, surveys and hard quantitative data are only going to tell you what people are commonly doing, not what exactly about what they're doing is making them satisfied. If your MMOG is bleeding customers, none of that data will tell you why exactly that's happening, only perhaps what people are and are not doing in the game. When you have to decide what systems of your game to build up or use as your foundation, or which failures are costing you precious legitimacy among your potential customers, good luck using polls, surveys and hard metrics to understand that.

[Ingmar]

Polls also only give you answers to the questions you actually asked. There is a good chance that the questions you are asking when you set up a poll aren't the ones you really need to know the answers to.

[Calandryll]

That's what QA is for. 

But Calandryll's point, and it's a good one, is that QA will never (no matter how good they are) ever get the coverage of 100,000 eyes looking at it.

Take it a step further - QA (and for that matter developers) won't even know what to look for all the time. Players will play the game in ways we never expected or intended. That's the nature of the beast and it's what makes these games so much fun. The only way to know how they are playing the game, outside of the intended mechanics, is to talk to them.

A quick example:

Years ago when I was OCR manager for UO, we decided to make a change to Runebooks. I forget what the change was, but it was a good one and if I recall, one that also fixed a bug. We designed it out and then posted the change for feedback. Everyone liked it. A day or two later another player, (I think his name was Goodman) posted that the change would ruin Rune Libraries. Now, this was not long after Runebooks had been released, we had never heard of Rune Libraries before. There were probably less than 10 of them across all servers. Most of us played UO a lot at the time and we still hadn't heard of them. They were a new idea. So our response was "what the hell is that?" to which Goodman replied telling us what they were. Our response was "wow, that's a great use of Runebooks, we'll change our design so we don't ruin that" - and we changed it. It wasn't a major change, probably took the designer less than an hour to do it. A few months later there were hundreds of Rune Libraries all over Britannia on every shard.

No poll or metric or survey on the planet would have allowed us to know about Rune Libraries at that time. The only way we found out was because we were talking to our players.

That may seem like a small thing, and maybe it is, but I can give you dozens of examples like that and they all add up. :)

[tmp]

Wouldn't a poll at log in get just as much feedback as digging through the forums?  Set it up so the player can skip it if they want but give a small reward or something if they take the poll.

You might get the feedback if there's a reward, but how will you know this feedback is actually any good? Because when i want to play the game (or quit the game for the day) and the obstacle pops up, i'll just click through any random option just so i get my reward for 'participating', and be done with it... i want it to go away as fast as i can make it and go on with my fun, not sit and read questions and/or ponder answers.

[Ratman_tf]

That's what QA is for. 

But Calandryll's point, and it's a good one, is that QA will never (no matter how good they are) ever get the coverage of 100,000 eyes looking at it.

And yet having 20 million eyes on a problem doesn't mean much if none of them see to the heart of the problem.

(Sorry if that came across as pretentious. I really had trouble putting that thought down right. Probably failed.  )

 

Take it a step further - QA (and for that matter developers) won't even know what to look for all the time. Players will play the game in ways we never expected or intended. That's the nature of the beast and it's what makes these games so much fun. The only way to know how they are playing the game, outside of the intended mechanics, is to talk to them.

A quick example:

Years ago when I was OCR manager for UO, we decided to make a change to Runebooks. I forget what the change was, but it was a good one and if I recall, one that also fixed a bug. We designed it out and then posted the change for feedback. Everyone liked it. A day or two later another player, (I think his name was Goodman) posted that the change would ruin Rune Libraries. Now, this was not long after Runebooks had been released, we had never heard of Rune Libraries before. There were probably less than 10 of them across all servers. Most of us played UO a lot at the time and we still hadn't heard of them. They were a new idea. So our response was "what the hell is that?" to which Goodman replied telling us what they were. Our response was "wow, that's a great use of Runebooks, we'll change our design so we don't ruin that" - and we changed it. It wasn't a major change, probably took the designer less than an hour to do it. A few months later there were hundreds of Rune Libraries all over Britannia on every shard.

No poll or metric or survey on the planet would have allowed us to know about Rune Libraries at that time. The only way we found out was because we were talking to our players.

That may seem like a small thing, and maybe it is, but I can give you dozens of examples like that and they all add up. :)

This I can't argue with. I mean, if you get some good feedback from any source, of course you're going to be grateful for it.

[tkinnun0]

Like I said, all I've got to go on is that it's got PvE

Yeah, I don't know about that. All I've heard is that they have some group quests with an automatic joining mechanism. Perhaps that's the endgame.

[Lantyssa]

That may seem like a small thing, and maybe it is, but I can give you dozens of examples like that and they all add up. :)

My personal favorite was proving the correct values of BE tissues in SWG.

Yes, I could have bugged that it was broken, however it took a long analysis involving linear algebra to show the results weren't matching the expected data and why.  It was laid out all nice an neat so any dev could come by, look it over, and go, *tweak* *tweak* "Done!"

Otherwise they would have needed to first believe something was broken, without any supporting evidence beyond "trust me, your math is wrong", dedicate someone to analyzing it who was actually capable of doing so, then impliment the fix.

I don't know if they relied upon it when it was fixed, however it most likely did lead to it in some fashion as the BE community kept it as one of their top five to fix until it happened with the CU.

[Fordel]

committed TLs who were good at collecting and summarizing information and providing useful suggestions, and they were completely blown off for years. One that comes to mind in particular was the wizard TL, who did a fantastic job and was constantly ignored.

Testers, that are paid to test, typically have 50% of their bugs result in a change.  The other 50% are not ignored but it could be viewed by some as being ignored since no change was made.  Dupe, No Repro, Works for Me, Already fixed, 3rd Party, Postponed, By Design, etc. just do not have visibility outside of the organization.  The speed of the feedback loop is also important.  Do bugs sit for 7 months, 7 days, 7 hours, or 7 minutes?  There are software service that do have minutes to hours turn around on issues and released to production\customers, just most companies are not there yet.  There is a movement to make all software products document all known issues at the time of each release but the industry is resisting.

IF Ing is refering to Therrik (Wiz TL) then the issue wasn't just non-transparency.

Wizards were like most casters in DaoC, except they had no real utility. Just nukes and more nukes. Therrik asked for utility for a long, long time. He would show why they needed it, he would show how they were inferior with out it, would even suggest entirely reasonable things to provide it. The people reading the feedback would constantly tell him 'no, Wizards dont get utility, just pure damage'. Of course this was entirely flawed, since Mythic wouldn't/couldn't give *any* casters any kind of DPS upgrade, let alone give Wizards enough of a upgrade to 'define' them or counter their lack of utility. So Therrik, for a long time tried to find non-overpowered ways to improve the 'damage' aspect of wizards, without actually increasing their DPS OR giving them utility, all the while pleading for someone to see reason as to why this wouldn't work.

This is a time frame of YEARS mind you, not weeks or months, YEARS of reports and player feedback being eaten by a logic loop. *Error can't give damage, Error can't give utility, further suggestions required* Just flat out told him "No, no utility, stop asking for it/commenting on it/suggesting it etc."

Then finally, one day after years of Wizard community brainstorming they actually did come up with a way to make wizard DPS... 'wide' instead of a flat increase. Clever ways to do more damage, without giving utility spells and keeping within the wizard theme. The response to that?

"No, try suggesting some ideas revolving around utility."

 

[Ingmar]

Yeah, Therrik, that was his name. And that's the cycle I was talking about. There were other classes that went through that too (I PLAYED A THANE, PLEASED TO MEET YOU) but the wizard one is the best example because of how long it went on and how obvious the problem was.

[Fordel]

Hero's didn't eve have a TL most of the time. The Warrior TL would just copy/paste his report for us too, since 98% of our issues were identical.

[Venkman]

Two days away from the thread... unleash the SBing!

That's why MMOs are a genre and not a medium.

This doesn't even make any sense. "Western," "Sci-Fi," and "Fantasy" are genres.

Those are settings. Videos game genres are about the experiences. FPS, RPG, RTS, Sports Sim, and so on for however long a list your particular publication/marketing group needs it to be. Within this comparison, "MMO" as a term doesn't work well imho. Because you can't just say "MMO". In any substantive conversation, you have to say "MMO like <game>", where "game" defines the actual genre (WoW for RPG, Saga for RTS, Planetside for FPS, etc). MMO by itself is just another way of saying "24/7 persistent state world", which is a technical foundation not unlike "canvas" or "film". Ergo: medium.

Just because the execution of entrants in the medium is shitty, derivative schlock peddled on the ass of a bump-mapped skank whore doesn't mean MMOG's aren't a medium.

I know that (per my prior post, and above). But the MMO medium is still young, and still overly defined by one playstyle. Heck, how long did it take the world to go from "Castle Wolfenstein 3D" to actual coining of the term "First Person Shooter"? We know that MMOs existed before "MMORPG" was coined (though there's still apparently some debate on who coined it and when).

You're skipping over a lot there.  WW2O, MCO, TSO, E&B, Majestic, UWOO (UO2), I think there were one or two more, not to mention a rash that got cancelled  in the summer before our launch but after those.

Point taken about WWIIO, as that launched near AO and had big technical problems as well. However, MCO launched in the same month as DAoC, TSO and E&B launched a year later, UO2 never launched at all and Majestic was not an MMO (more an ARG/service 'til they gave up and packaged it).

DAoC deserves its slot in history as having proven you CAN launch a game that CAN be played on day one. But I don't recall it doing much to turn investor's heads to this genre in general. And things only got worse after that through the subsequent flops of "how can anything go wrong" titles like TSO and SWG.

[Fordel]

I know that (per my prior post, and above). But the MMO medium is still young, and still overly defined by one playstyle. Heck, how long did it take the world to go from "Castle Wolfenstein 3D" to actual coining of the term "First Person Shooter"? We know that MMOs existed before "MMORPG" was coined (though there's still apparently some debate on who coined it and when).

I always called them 'Doom Clones' 

[UnsGub]

But Calandryll's point, and it's a good one, is that QA will never (no matter how good they are) ever get the coverage of 100,000 eyes looking at it.

True, but that is outside the scope of QA\Testing unless you hire 100,000 and tell them to cover it like 100,000.

There are different types of feedback systems depending on what you want, automation, manual, feesability, usability, focus groups, customers, publisher, press, peers, legal, management, ESRB, etc.  Each generates different types of feedback and require different expertise to collect and turn into action.  Most places are not setup to even collect it so it is no surprise it comes across as ignore and no action.  Fortunately there is lots of room improvement just by collecting it all rather then some and also in refinement to each of the feedback systems.  There is all high level stuff determine by management way before people actually make and use feedback systems, which is hard in itself.  The most common mistake on execution has nothing to do collection, processing, or even determining actionable items.  Its finding the time for people to actually do that changes in the time available, if any.

[UnSub]

Wouldn't a poll at log in get just as much feedback as digging through the forums?  Set it up so the player can skip it if they want but give a small reward or something if they take the poll.  I suppose you could even add a comments section and use that to identify players that give consistently well thought out feedback who could then be included in more in depth polls or what not.

Polls are only as good as the options provided. If it is a clear cut issue (e.g. Yes / No, Option A through Option E, etc) then they have a place. However, I've seen so many bad polls (and every poll with a comedic option is of extremely questionable value) that it is obvious they are poorly thought out and probably mis-used.

Often top level poll data is used as a bludgeon against anyone who might disagree, but (unless the response is account linked) you can't work out the type of respondents who answered the poll - is it the hardcore PvP community who are anti the change while everyone else seems positive? Are players who log on less than 5 times a week even answering the poll?

If you want less biased community feedback, you need to random sample your player base. However, it will only give you answers to questions you ask, not raise and explore issues unthought of by the devs - that is what forums can do (and I believe centralised community in official forums can do it better than a split community over several fansites).

[Khaldun]

Basically, that's it: a forum can produce discussions of issues and knowledge about feelings or experiences that the developers cannot ask about in a poll and survey because they themselves may not know about or anticipate those reactions. It doesn't matter that this will be surrounded by a lot of noise, or that it it coming from a probable minority, maybe even an unrepresentative minority, of players. It is a place to start, it is something new, it is information. Given that all commercial MMOGs to date have struggled badly at times, visibly, because the developers clearly did NOT know something vital about the experiences, feelings or problems that players were having, what developer would scorn or sideline a source of information like that, unless they had a very systematic alternative plan for obtaining even better high-value information (e.g., systematic qualitative research on players and their experience).

[cmlancas]

It's true. All these metrics we keep harping on generally don't mean anything because they don't have clearly defined objectives.

I think it ironic that the more I read about instructional design, the more I find it applies to fields other than training.

[IainC]

So basically what we're saying is that 'multiple data points are good' and 'forums are a valuable data point'? Which is what we had 4 pages ago.

[Venkman]

I think it ironic that the more I read about instructional design, the more I find it applies to fields other than training.

Video game user experiences can be broken down into three main categories: Learning, Practice, and Mastery. You could throw grokking in there between Practice and Mastery depending on the system. The first part of all of the (good) games in the last two decades is instructional design. Some could argue that includes the instruction manual, but who reads those?

[Khaldun]

Well, Iain, there have been some other points about why it's important to have an official forum, for a MMOG live management team regard forums as important (commitment to open-ended communication, sign of respect for the players, multiple weaknesses of external forums as a source of information and communication). But yeah, we had the "multiple data points are good" four pages ago, and hence, when Jacobs says, "Listen to your customers, gather data on what they're feeling" and "Forums suck", many of us said, "Those don't go together at a really basic level".

[UnSub]

So basically what we're saying is that 'multiple data points are good' and 'forums are a valuable data point'? Which is what we had 4 pages ago.

See, when you just skim one fan forum because you've got 5 others to check, you miss the real meat of the discussion. 

The issue came up if forums had any value at all compared to compiled system data. It does, because forums + compiled system data end up providing both qualitative (deep exploratory information, but not from a wide group of people so it may not be representative) and quantitative data (shallow exploratory information, but from a wide group of people so it is probably representative). Both are important and provide both sides of the picture. Forums may overstate the importance of something, or hold a perception that is incorrect. Compiled system data won't throw up issues if a system isn't working as it should or sometimes masks a system that is working correctly but 'feels' wrong to players.

I'm not going to pretend that forums are automatically bastions of wisdom even if they are official. In reality they are groups of people standing around a common point of interest, often with a complaint to make or issue to raise. But Jacobs going, "We think listening to our community is important" was pure spin compared to how he actually feels about about forums and because Mythic is the exception in terms of how they try to actually "listen". If Mythic did something exceptionally different - and I don't think the Herald is really that exceptional from what I've seen - then I wouldn't have as many issues with Jacob's original point.

[Ratman_tf]

So basically what we're saying is that 'multiple data points are good' and 'forums are a valuable data point'? Which is what we had 4 pages ago.

Getting to the point is overrated. Bullshitting over a cooler full of beer is more fun.

I think forums can be a valuable source of data, but I don't think they're intregal and necessary to the success of an MMOG.

I think that means I agree with Mark Jacobs. Does that make me a sychophant? 

[tmp]

I think forums can be a valuable source of data, but I don't think they're intregal and necessary to the success of an MMOG.

I'd guess it's like most of other game components/systems -- while not absolutely necessary, the lack can lead/add to negative impression. On its own that might not be enough to make people quit, but give enough problems (something that every MMO has) and it can become the proverbial straw. At the bottom of it, it's adding some extra risk to game well-being to save some money ... but given the game is already investment measured in millions, are these savings really worth the risk?

[Ratman_tf]

I think forums can be a valuable source of data, but I don't think they're intregal and necessary to the success of an MMOG.

I'd guess it's like most of other game components/systems -- while not absolutely necessary, the lack can lead/add to negative impression. On its own that might not be enough to make people quit, but give enough problems (something that every MMO has) and it can become the proverbial straw. At the bottom of it, it's adding some extra risk to game well-being to save some money ... but given the game is already investment measured in millions, are these savings really worth the risk?

They can also turn out to be a huge liability. Some thread on rpg.net, a fellow was talking about how much distaste he had for the posts on message boards after visiting the offical boards for Mercs 2. A game he was having a blast with, and then saw the barrage of armchair developers bitching about every nuance of the game.

One of the reasons I stopped visiting offical message boards is because while I was having fun playing the game, the tone and content of most posts were so negative, they made me question wether I was having fun "The right way".

I must say, my gaming is much "happier" now that I limit my visits to those boards. (Unless there's some really juicy drama or teh funnays going on.)

[SnakeCharmer]

One of the reasons I stopped visiting offical message boards is because while I was having fun playing the game, the tone and content of most posts were so negative, they made me question wether I was having fun "The right way".

This.

Also, I find the less aware I am of bug #127836 for some class or quest or item that I will never play and will never affect me, the happier I am. 

And even though I'm taking part in my last beta (unless something really freaking cool comes around - which is unlikely), beta forums are a bit different.  But I do restrict my reading to bug reports (so that I don't double up on it) and developer comments/posts.  Everyone else is on ignore.  Besides, I'm not looking for friends in the social community forums space.  I've got my handful of friends that I game with, and that's all I need.

[Ratman_tf]

It's been noted that betas tend to be a bit more productive, a bit more tight knit, than a post-launch community.

[Khaldun]

They can also turn out to be a huge liability. Some thread on rpg.net, a fellow was talking about how much distaste he had for the posts on message boards after visiting the offical boards for Mercs 2. A game he was having a blast with, and then saw the barrage of armchair developers bitching about every nuance of the game.

One of the reasons I stopped visiting offical message boards is because while I was having fun playing the game, the tone and content of most posts were so negative, they made me question wether I was having fun "The right way".

I must say, my gaming is much "happier" now that I limit my visits to those boards. (Unless there's some really juicy drama or teh funnays going on.)

Doesn't this apply then to reading anything about games? Wouldn't gameplaying be enhanced if you knew nothing about the games industry, games production, the reputation of specific games, or debates about whether or not there should be forums? I'm curious about when not knowing stops paying dividends of fun.

[Ratman_tf]

They can also turn out to be a huge liability. Some thread on rpg.net, a fellow was talking about how much distaste he had for the posts on message boards after visiting the offical boards for Mercs 2. A game he was having a blast with, and then saw the barrage of armchair developers bitching about every nuance of the game.

One of the reasons I stopped visiting offical message boards is because while I was having fun playing the game, the tone and content of most posts were so negative, they made me question wether I was having fun "The right way".

I must say, my gaming is much "happier" now that I limit my visits to those boards. (Unless there's some really juicy drama or teh funnays going on.)

Doesn't this apply then to reading anything about games? Wouldn't gameplaying be enhanced if you knew nothing about the games industry, games production, the reputation of specific games, or debates about whether or not there should be forums? I'm curious about when not knowing stops paying dividends of fun.

http://en.wikipedia.org/wiki/False_dilemma

[Slyfeind]

If I knew anything about Prey before playing it, I would have thought it boring. As it was, I went in knowing nothing about it, other than the title. I thought it was awesome. WoW was a freaking blast with my first three characters; NE rogue, human mage, tauren druid. Now it's the most boring game I own.

I'm replaying Bard's Tale right now, and I'm really bored with it because I know where to go and what to do. I replayed Ultima 3 a while ago, was a bit bored with it, then decided to get all the treasure from all the dungeons; something I never did before. I went into them with spoilers and maps at hand, and I had a bit of fun doing that. Before that, I replayed Ultima 2 and went dungeon delving, without the maps. I got stuck a lot, and sometimes I had to restart the whole game because I died. I had a lot of fun doing that.

Yeah, it's different degrees, but I'd say it depends on the person. I'm sure there's only a handful of people who think dying in a dungeon and restarting the game is "freaking awesome." It depends on who you are, and why you play games.

[Venkman]

I'm curious about when not knowing stops paying dividends of fun.

Depends on how well the game is designed. The mass market successes out there ARE that way because they have hit a segment of the market that a) found it fun, b) found it fun unto itself; and, c) probably didn't visit a single forum nor metacritic nor maybe even didn't read about the game itself.

That combination of success factors is as rare as WoW is to MMOs. Very few can rely on making a good game that is so intuitive as to not require any pre-knowledge nor research while at the same time providing an ongoing enriching experience players continually seek without feeling the need to research better ways of playing it.

[Sunbury]

I wish devs would STOP talking on message boards, and games STOP posting release notes, both take all the mystery out of these games.  I thought the whole point of RPGs is to 'figure things out', yet message board postings and release notes explain it all.  The only comment a dev should make is 'not a bug' or 'its a bug' and stop there if a big issue comes up.

Devs can't stop players talking between themselves out of game, but they don't have to join in, but they can 'listen'.

RPGs should not be 'chess' where the rules are fixed and known - its part of the game not to know the rules - but to figure them out, and realize they could, and will, change.