Alpha, Beta, Gamma -- public testing the right way

[Stephen Zepp]

Disclaimer: I'm not in any beta programs currently, and don't have any insider info on anything--this is all my opinion, mostly gathered from f13.

Ok, this is a bit of a soapbox, but I've seen a trend about (specifically with recent thread regarding the Warhammer beta, but others as well), so I wanted to throw out some points from a developer's perspective. They are going to be against the grain (no one does it this way anymore), and possibly inflaming, but hey, that's the purpose of f13!

Suggestions to the "Public" : Make up your freaking minds!

Do you want developers to listen to you and change/adapt to user expectations and desires, or do you just want to play free games and bitch about them publically?

Here's what I mean by that: it's a general consensus here at f13, and on the internet in general, that 90% (use whatever number you want, doesn't matter) of games that get released suck. Theoretically at least, alpha and beta testing should be the times where the public (or whatever subset of that class is in the testing pool) gets to assess, discuss, and recommend about a product. It might be just some features, it might be core mechanics, it might be simply saying "it would be really cool if", but it's the first time you (the testers) get to actually communicate with the developers at a serious level regarding the game they are developing.

Many times it's pointed out that the first time the public gets to take a look "for real" at projects is in the final stages of beta--and in fact, it's not actually testing, it's marketing. This is a self-fulfilling prophecy. Just look at the Warhammer beta thread--no one knows exactly why the beta period was shut down, but most of the reactions are negative--but why? These responses generate heat around the project, in a very negative way--giving negative feedback to the developers: don't listen to our feedback if it's negative, and certainly don't change your development cycle simply because we say so.

Why isn't there a strong feeling being expressed of "wow, they got some negative feedback, decided it was enough to pull back a step in the dev cycle and re-engineer, and come back to beta with a freshly developed product--and that's awesome!". Instead of applauding a developer's decision to re-time the development cycle for positive benefit, it's treated almost uniformly as extremely negative--spelling the doom of the product, etc, etc. In effect, you are "proving" to the developers (and other developers watching) that actually listening to their testers on more than watching bug reports is a very bad thing.

If you, as a testing public, want to actually be able to affect a game project, you'll need to stop bitching about companies actually listening to you, and taking the time, effort, and risk associated with re-designing when appropriate if your collective feedback indicates. If you keep blasting developers that honestly want your feedback and take the risks of listening and adapting to it, stop kicking them in the nuts when they actually follow up on that--otherwise, all a beta will continue to be is what you are bitching about--marketing, and possibly load testing.

Suggestions to the developers : Go early, or go home.

It's too freaking late in most development cycles to expect a "beta testing" period weeks/few months prior to scheduled gold to be worth anything at all. Mythic is taking huge heat for it, but I personally applaud the decision, for whatever the reasons may be, to take a step back, consolidate any new design/implementation requirements around the collective feedback from a beta period, and actually use that feedback in a positive way.

But why not plan ahead for this? Ignoring business/IP risk for a moment (and that's a big stipulation I admit), and given that you can have a testing pool from the public that will actually work with you instead of against you, getting feedback on your project after being deep in production (or even post-production in some cases) is way, way, way too freaking late. Get it early, and get it often. Agile development talks about prototyping--but the biggest strength of a prototype is telling you what will not work, not what will--and to get an honest assessment of that, the bigger the audience in some cases, the better.

Plan, from the very beginning, to have an alpha testing cadre that will take your completely unattractive, totally feature poor, buggy as hell prototype and evaluate it for fun. At the worst, you'll be able to back out of a project before it's already cost you multiple millions and is too late to fix, and at the best, you'll get valuable direction and feedback for success.

Testing Phases

This isn't new, I'm not proposing the newly born robot jesus of game development here--but there are at least three key places where you can gain valuable feedback for a project:

Alpha Testing -- pretty much as soon as you can get a working sub-set of your game into someone's hands, do it. Programmer art, text UI's, no persistence, hacks and cheats abounding--get it into someone's hands for evaluation. Of course, it's critical that expectations are managed properly for this to actually work, and it's also critical for the testing pool to understand the nature of the test period, and work with it instead of against it.

• Devs: make sure the testers understand expectations, and if you can't show it, describe it and say why it's not in yet. You should be demonstrating core mechanics and possibly key game systems here, not bling-ing up the latest multi-pass procedurally generated shaders.
• Testers: If you fail in actually performing your role here (similar to the things I describe above in regards to the WH thread), you're shooting yourselves in the foot. You are providing feedback on potential, not HD resolution modes and pixel magic. If you do things right and in the right way, you have the power to fundamentally affect both the quality and the fun of a game you are (hopefully) interested in--but if you do it wrong (bitch, whine, complain and  ), the devs will clam up and ignore you.

Beta Testing -- most major systems developed, core mechanics finalized, prototype assets (not production quality yet, but step up from programmer art), most kinks worked out. Bang on that sucker, because once we're in production it's really too late to be making broad sweeping changes.

• Devs: listen to the feedback, but listen wisely. Big risks here of losing the vision of the game and allowing the testing pool to focus on bushes and shrubs instead of the forest. If it does turn out that massive sections of the forest need to be burned down, then do it--but do it wisely, and not as a knee jerk to "omg they don't like it because it's ugly".
• Testers: You've got less power here than in Alpha, but you still have quite a bit of input to the final game. Use it wisely as well--if you continue to bitch about every mob using the same animation rigging, when it comes to gamma and the core mechanic is still completely un-fun, it's your fault, not the developers.

Gamma Testing -- this is actually what most of "beta testing" really is--just about completely post-production, all systems are finalized and implemented, core mechanics are unchangeable, by god this game is shipping. Effectively, all you have left is actual bugs (as opposed to design issues), and polish, so focus on that.

• Devs: You're pretty much stuck in regards to actually changing things, so keep the team focused on what's important to a quality ship: bugs, bugs, and more bugs. It's too late to change mechanics and systems, and if you've managed expectations and process effectively, you won't need them.
• Testers: see above. We're well past changing the game itself, so focus on polish and robustness in your evaluations.

Keep in mind that what I'm describing in the "Gamma" period is what we see in the current methodology as "beta testing"--testers want to change the game in core ways, developers can't without taking huge risks--so why set ourselves up so this conflict always occurs?

Summary

With the current paradigm, developers and public testers are in direct conflict whenever they interact. By the time the feedback is available, it's too late for the feedback to be effective--and when companies like Mythic take the plunge and actually use the feedback and modify the development time line, they are getting blasted for it.

Stop working against each other, and start working with each other, and we'll all benefit.

[Stephen Zepp]

I wanted to post this as a separate post because it's of a more direct nature, but I will also say that if a development comes to you as a person, or you as a community wanting to set up this type of testing cycle, it's not in your interest at all to start suddenly wanting compensation, control, or royalties. With how testing works today, you don't get compensated.

If a developer is willing to give you more influence and earlier access so your feedback has the potential for better integration, that's not a good time to start demand more--getting an alpha test exposure and all the benefits that may entail can't be equated with being part of the development team--but it may lead to it if individuals are consistently strong and have good vision.

[Kaa]

Suggestions to the "Public" : Make up your freaking minds!

Do you want developers to listen to you and change/adapt to user expectations and desires, or do you just want to play free games and bitch about them publically?

Both, of course :D

However, Stephen, consider that the "public" is not a single homogenous entity, but rather a messy mix of all kinds of people. Some like cockstabbing, some like crafting, some like tactical combat, and some like cybering with elves. You can't satisfy all of them.

The public's job is to bitch and moan about the game -- in many different, usually contradictory, ways. The developers' job is to listen to that bitching and moaning, but listen to it critically, realizing that your users actually comprise a collection of very different subcultures and each subculture wants something different. And, of course, what the users say they want is not necessarily what they mean they want.

If you, as a testing public, want to actually be able to affect a game project, you'll need to stop bitching about companies actually listening to you, and taking the time, effort, and risk associated with re-designing when appropriate if your collective feedback indicates. If you keep blasting developers that honestly want your feedback and take the risks of listening and adapting to it, stop kicking them in the nuts when they actually follow up on that--otherwise, all a beta will continue to be is what you are bitching about--marketing, and possibly load testing.

I don't think anyone was kicking Mythic in the nuts for stopping the beta (possibly for other reasons, though ). It's not like we thought the developers are bad people and/or made a bad decision. The reaction on these boards was that the stopping of beta is an indication of problems with the game. It might be a correct decision, and the one showing that the developers listen to feedback and are willing to modify the game. but it still is an indication of problems with the game.

Think of it this way -- you just bought a new car and the dealer said you could pick it up tomorrow, but then he phones you and says that his mechanic needs a couple of more days to tune your car up. Wouldn't you take this as an indication of problems? That doesn't necessarily mean the dealer is a bad guy out to screw you, but it might mean that some innards of the car are not as shiny and perfect as you hoped they would be. Of course, if the dealer fixes the car and delivers it to you fixed, you'd be glad that he took the time to fix it. But really, you'd prefer that the car didn't need fixing to start with.

Kaa

[Salamok]

I agree with Kaa on his response regarding "the public making up it's mind", sort of a wizards 1st rule type of thing, individuals can be intelligent but the mob is a retard regardless of it's make up.

I wasn't in their beta so I don't know if mythic did the right thing.  I will say that I saw some figure of 400k+ beta accounts for Warhammer, if they managed to send out that many invites prior to figuring out they needed to retool to this extent then WTF.  If a majority of those 400k accounts were active they probably should have just friggen released and recouped what losses they could.

As far as dev expectations and player input it is my opinion that trying to decipher player input past a certain stage is counter productive.  I personally think that point is in late alpha (definately not late beta 400k accounts).  Your alpha testers should be carefully chosen, encouraged to be outspoken and probably paid.  Your beta testers should comein 2 phases, the 1st is more of a focus group to see if you have appealed to your target audience.  A minor amount of game design tweakage can come out of that.  Your second is more of a final debugging, load test and PR event. 


edit: oh yea, in case it didn't come through strong enough - I am a firm believer in "the vision", dictatorships seem to produce better games than democracies.  Listening to the playerbase too much ruined EQ and is ruining WoW.  In both cases the devs/producers forgot that the majority is usually silent and more or less content.  If anything in an MMO I think the production team is better off paying attention to the statistics (aka what the players are doing) than what the player base has to say (aka the forums).

[sam, an eggplant]

The gamma period you describe is not analogous to the traditional MMO closed beta. During closed MMO betas, major systems are added or changed, giant swathes of content are staged, etc. It's what would normally be considered alpha testing. Your "gamma" testing is actually the open beta or stress test period immediately preceding release, and it's not used for squashing bugs or polish, it's exclusively for marketing and performance tuning.

[Salamok]

it's not in your interest at all to start suddenly wanting compensation, control, or royalties. With how testing works today, you don't get compensated.

This is the problem, most of these games are diku's and the people with the most experience with this type of game are all grow'd up now and don't have the time to thoroughly play and test a diku.

So either hire them full time to thoroughly test and play or thow them aside and quit trying to please them at all as they haven't been your target audience for quite some time now.

[Nebu]

This is the problem, most of these games are diku's and the people with the most experience with this type of game are all grow'd up now and don't have the time to thoroughly play and test a diku.

So either hire them full time to thoroughly test and play or thow them aside and quit trying to please them at all as they haven't been your target audience for quite some time now.

It's funny that you'd mention this.  I've been playing diku since there was a diku and yet I still never get invited to beta tests anymore because I'm outside of the target demographic. 

On topic, I can understand the frustration that many development houses must have with the public.  Their desires are a moving target and their complaints are often unwarranted.  I think the key is finding a subset of people that can articulate what they see in the game during the testing phases but have diverse enough tastes in gaming to bring a variety of perspectives to the table.  Making a game for the masses is certain to piss off everyone in one way or another.  As long as you satisfy 80% of their needs, the other 20% is little more than white noise. 

[Venkman]

You get what you pay for :)

MMOs have been allowed to be all over the place because people continue to like that it's still got a form of "Wild West" feel to it. This is fast becoming less the case though, as more money and IP come in. I fully expect the day to come when there's only a few hundred people in a close beta through launch and they only leverage thousands when it's time for stress tests.

On the one hand, someone might say "you can't only have a few hundred people testing in games that support thousands!". But on the other hand, consider the games themselves. How many games really leverage the "thousands" beyond Eve? These games have only gotten smaller in scope, larger worlds with activities for smaller groups to fight against. 24-man raids, 10-man, 5-man. The "economy" part, well, I wonder how relevant that data is anyway when core systems of a game can change so radically in beta anyway.

Limit the scope of your testing base. The larger you go, the more likely you'll lose that many more box sales.

Further, ingame reporting tools. All over the place and intuitive. If you don't want to spend the time making these reporting tools the right way, then it's you have no-one to blame when you're suffering under the deluge of plain English reports you're receiving, most of which say the same thing but slightly different enough you need to eyeball each one.

Finally, targeted events and required inputs. You need to direct your testers as you would any QA group (not to the same degree of course), and compel some sort of participation. They're not on salary, but they can be kicked out to make room for people who'll actually provide feedback.

The days of letting this stuff continue willy-nilly are over. There's just too much money coming to the genre now.

[Aez]

Suggestions to the "Public" : Make up your freaking minds!

Do you want developers to listen to you and change/adapt to user expectations and desires, or do you just want to play free games and bitch about them publically?

As a small part of "the public" who can make is on mind on a game, a quote like that scares me.  I see allot of whining on beta forum, I often think the poster doesn't understand the game, is unjust or simply want an Holodeck for 12$/month.  I can deal with it has a consumer, I certainly hope the developers can do the same.  This kind of quote is akin to a director saying  the public doesn't get his movie because they're not smart enough to like it. 

Why isn't there a strong feeling being expressed of "wow, they got some negative feedback, decided it was enough to pull back a step in the dev cycle and re-engineer, and come back to beta with a freshly developed product--and that's awesome!". Instead of applauding a developer's decision to re-time the development cycle for positive benefit, it's treated almost uniformly as extremely negative--spelling the doom of the product, etc, etc. In effect, you are "proving" to the developers (and other developers watching) that actually listening to their testers on more than watching bug reports is a very bad thing.

You are forgetting the history of mmorpg.  What you are asking actually existed back in 2000.  Do you remember the Shadowbane fans?  Now however, with all the games who let us down, you are asking a beaten wife to believe that it's over, this time you'll be nice, you'll never hit her again.

In today's market, mmorpg are guilty of sucking until proven otherwise.

[Pennilenko]

In today's market, mmorpg are guilty of sucking until proven otherwise.

You got that right!

My big thing isnt how developement works out. My problem is what they do to us as players during live. Nerfing is clearly a developer failure. Its a scary domino effect leading to game play experiences changing drastically and alienating parts of the playerbase. Never take power away from the players. If a developer screwed up and they need to make changes, knuckle up and make the tough choices that involve not screwing your players over.

[Venkman]

The word "nerf" is used far more often than it should. MMOGs are fluid systems. Their only absolute is that they will change. No system yet has been so future proof that some change doesn't affect some player's perception of How Things Work(tm).

The basic problem is that these games are systems designed laterally but games experienced vertically.

At any give time, a player is conducting activities in one narrow set of features. Their entire perception of the game is based on how that one narrow set of features feels and plays out within the limited time they have to play, sometimes against another narrow set of features. Few out there that can garner the entirety of interacting systems as a pure end user. And then there's few of them that can discuss it dispassionately. Those are the ones who should be hired for QA rules :)

Unfortunately, they're also usually the dedicatd players than ascend to community leadership roles.

[LC]

I have been in the beta of almost every MMO since UO, and it's very rare to see the developers care about tester input. In fact I could probably count on one hand the number of betas that had any interaction between developers and testers. Most beta forums seem to be a fancy frontend for /dev/null.

Visual Aid:




Other testers are an obstacle as well. If you post something negative about the game, fanboys will attack you with "DUDE ITS BETA ITS NOT SUPPOSED TO BE FUN". Which usually results in the thread getting derailed and locked.

[Talonus]

I have been in the beta of almost every MMO since UO, and it's very rare to see the developers care about tester input. In fact I could probably count on one hand the number of betas that had any interaction between developers and testers. Most beta forums seem to be a fancy frontend for /dev/null.

On the flip side, I'd say that it's quite rare to find testers actually worth listening to. The vast majority, as you mention, are fanboys or better ignored. Most testers are useful for little more than data mining and stress testing unfortunately. The better betas I've been in had developers that would read and occasionally interject, but knew to ignore the idiot testers most of the time.

[Righ]

The public's job is to bitch and moan about the game -- in many different, usually contradictory, ways. The developers' job is to listen to that bitching and moaning, but listen to it critically, realizing that your users actually comprise a collection of very different subcultures and each subculture wants something different. And, of course, what the users say they want is not necessarily what they mean they want.

The most difficult aspect about accepting input from a group of alpha/beta testers is managing expectations and dealing with signal/noise ratio. Some ideas are expressed poorly and it takes discussion to bring them to the surface, while at other times the discussion obfuscates the ideas. Its not always ideal to manage notification feedback, idea exchange, social networking through a single web forum. Its not always ideal to have all testers working on the same things. Most testers have something worth listening to. Most web discussion forums are great for making sure that you never hear that. :P

[Trippy]

Summary
With the current paradigm, developers and public testers are in direct conflict whenever they interact. By the time the feedback is available, it's too late for the feedback to be effective--and when companies like Mythic take the plunge and actually use the feedback and modify the development time line, they are getting blasted for it.

Stop working against each other, and start working with each other, and we'll all benefit.

My issue with them shutting down the Beta for two months is that they are doing it in a game that has a publicly announced Q1 2008 release date. That's very late in the cycle to have figured out that the game is "unfun" and needs significant reworking. Of course it's better that they do it now rather than after release but why didn't they figure this out back in, say, June?

[Margalis]

Just look at the Warhammer beta thread--no one knows exactly why the beta period was shut down, but most of the reactions are negative--but why?

Most delays and schedule changes happen because of significant problems, not because devs are making a good game even better. A proper development cycle has time for responding to feedback built in. When the schedule changes that means that time was inadequate or the problems were much greater in scope than expected. The vast majority of the time projects that are delayed a lot end up sucking, because the delay itself was evidence that the project had problems.

IMO the problem Mythic hit was that the beta shutdown was unexpected. If it was announced in advance it was not communicated well. So it looks like "OH FUCK ABORT ABORT!!"

Why isn't there a strong feeling being expressed of "wow, they got some negative feedback, decided it was enough to pull back a step in the dev cycle and re-engineer, and come back to beta with a freshly developed product--and that's awesome!".

Because enough negative feedback that you have to re-engineer is a sign that things are going poorly. It's good that they would rather delay than release crap but at the same time the delay is evidence of problems.

Shorter version: It smacks of Vanguard.

Plan, from the very beginning, to have an alpha testing cadre that will take your completely unattractive, totally feature poor, buggy as hell prototype and evaluate it for fun. At the worst, you'll be able to back out of a project before it's already cost you multiple millions and is too late to fix, and at the best, you'll get valuable direction and feedback for success.

This is the key point, I agree 100%. The problem is that a lot of the testing at this stage is either not done or ends up with bizzare results. The Magic Online "friends and family" testing is a good example, they collected feedback and based on it went in a crazy, opposite direction of anything the players wanted.

Finding the appropriate testers for this is very difficult. Few people are willing to spend time with a true alpha-level product. At some point the responsibility of the developer is to figure out what fun is.

I think the problem many devs hit is that they *know* the game is not fun but they figure it will become fun at some later point in time, and that never happens. Moo3 is the example I love to use here, they just kept working on it knowing it wasn't fun and the fun never magically appeared as they thought it would.

One thing about Nintendo games is that a lot of the time just moving the character around is fun. For example in Mario64 a lot of people spent ten minutes just clowning around outside the castle, before they even got to the real game. That's an example of something that was fun very early on.

[KyanMehwulfe]

It should depend on the development style of said game as well. You can approach everything at once or have it more segmented.

Consider something like Star Wars Galaxies, or maybe Shadowbane (though I'm a bit uncertain). Those games, being more sprawling and dynamic, really need to be brought up all at once because so much is interconnected*. It's hard to segment the game too much and test each segment well when each segment depends on the others so much. So because you're trying to get everything into the game ASAP, in rough form, to see how everything functions together, at X point its going to be very unpolished and bugged. Meaning, expanding open beta testing isn't very viable yet.

Where as in contrast, and this was part of its beta success, WoW was easily developed in very segmented pieces. Instead of bringing in the entire world at once and slowly bringing it all up, they worked on tiny segments at a tiny, polished them somewhat internally, and only then added each segment to the beta. So they not only could bing in the masses earlier to beta test each segment (since regardless of the amount of world in the beta, the quality of that amount was safe for a larger number of testers), but they could even benefit from the no NDA and open discussion premise. It's very much part of their beta mantra, even, one must assume. However, you could never do the same thing at the same point (Y days from release) in, say, the aforementioned development cycle because instead of having piece_01 at a great quality, you have pieces_01-20 all in at once but at bad quality.

I don't see it as anyone near a case of being one-method-suits-all in terms of handling public testing.


*Such a design should suit faction war games as well. Case in point: DAOC. As referred to in the other topic. Hibernia got absolutely jacked in terms of development effort. Midgard less so, but still somewhat compared to Albion. Same with Horde in WoW. But with aforementioned approached, instead of doing one faction totally, then the next, you instead bring them all up together at once to ensure they're kept on par. Now, in a perfect cycle, one at a time could accomplish that as well, but as we've seen many times before, often the last race/faction/whatever in development gets the short end of the stick.

[Margalis]

Waiting until near the end to flip everything on at the same time is asking for trouble, part of the job of the developer is to not let that happen.

In SWG just leave out most of the classes. Or only include certain weapons. Or include crafting but give everyone resources on a fixed schedule instead of having them harvest.

Part of the job is figuring out how to make your stuff testable.

[SnakeCharmer]

Part of the job is figuring out how to make your stuff testable.

I've often wondered why they didn't cap levelling, then at the end of a set time, bump up everyone to the next "tier".  For instance, in a game with 50 character levels, for two to three weeks, everyone is capped at 10.  Then after whatever time frame, everyone is bumped to 11 if they made 10 or not.  Then played/tested to 20, then bumped to 21.  Rince repeat.  After reaching 50, do a character wipe, and start over.

I doubt we'll see a game with a crafting system as unique as, say SWG, ever again.  So, that's a relatively easy set of testing criteria, for the most part.

[KyanMehwulfe]

Waiting until near the end to flip everything on at the same time is asking for trouble, part of the job of the developer is to not let that happen. In SWG just leave out most of the classes. Or only include certain weapons. Or include crafting but give everyone resources on a fixed schedule instead of having them harvest. Part of the job is figuring out how to make your stuff testable.

For sure. By bringing everything up together, it's in a very basic form with everything you can exclude not implemented yet. I didn't mean to suggest a more finished state than that i it came across as such. Rather, just the most basic interdependant features that cannot be segmented (a la most of WoW's "pushes" of Level/content chunks).

[UnSub]

Stephen, WAR is getting a lot of negative attention for the beta break in part because 2007 was an incredibly mediocre year for MMOs (*insert your own joke here!*). If you compare what was expected to launch this year with what actually launched, you've got a big gap. WAR is just continuing this trend and is suffering slightly for it.

But to cover the issue of betas in more depth: right now they have the grace and control of the running of the bulls. Alpha / beta opens, the testers rush in, the devs close the door behind them (or try to stay out of their way, lest they be trampled). External MMO testing in my experience has little to no structure and no targets, so it shouldn't be surprising that the effectiveness of them is all over the map.

If the devs really, really want quality results from their external testers (in alpha, beta, whateva) then they need to:

1) Communicate constantly with the players. This is not just the CM saying a few words here and there - it's the actual people who make the decisions getting into discussions with players about why certain (possibly contentious) decisions were made and being able to accept that perhaps the players have a better way / solution, or that the original solution needs modification.

2) Focus the testing. Every week place a focal point for testing. It may be the first 10 levels, or an area, or an event, but that's all the players should do for that period of time. If they go outside of the boundaries, they risk getting kicked from beta. Following their experiences, they are expected to provide feedback - the good and the bad - in a structured way. What did they like about the areas they tested? What did they dislike? What was the most effective tactic they found to get through the area (which can then be duplicated / checked via datamining)? etc

3) Be honest with the players. G&H's "hey, we let go of half the design team and that's a good thing for the game" was already a punchline before today. Acknowledging that certain things need attention, but they aren't a priority right now, will actually gain the devs respect with most of the playerbase. Spinning certain things so that they look good rarely fools anyone outside of the forum warrior tragics who accept every line as truth. You can tell players what areas are the most important to the devs at this point and they will accept it (or do enough to get banned). If WAR / Mythic knew that a break was coming up, tell players a month before, not 3 days (or whatever).

4) Only call it a beta when it's 90% feature complete and what you really want is load testing. In the minds of lots of players, alpha is the buggy, development-focused phase and beta is the "we're almost ready, let's go see if this MMO is worth playing" phase. Keeping a game in alpha longer will scare off the beta players and only attract those willing to do more testing than playing.

5) Work out if you actually want player feedback. If you aren't going to change a system regardless of what the players say, say so. Otherwise you just end up with players angry that you aren't responding to them.

[Trippy]

I've often wondered why they didn't cap levelling, then at the end of a set time, bump up everyone to the next "tier".  For instance, in a game with 50 character levels, for two to three weeks, everyone is capped at 10.  Then after whatever time frame, everyone is bumped to 11 if they made 10 or not.  Then played/tested to 20, then bumped to 21.  Rince repeat.  After reaching 50, do a character wipe, and start over.

I doubt we'll see a game with a crafting system as unique as, say SWG, ever again.  So, that's a relatively easy set of testing criteria, for the most part.

Turbine did something like that with LOTRO, though not as systematically as your example, where in the later stages the early testers could bump up their level so that they could test the higher level content as it was being added.

[Margalis]

My biggest pet peeve is that most companies try to attract the biggest fanboys to test instead of the best testers. Making people jump through hoops or enter contests and acting like testing is a priviledge is a bad plan. A small number of articulate, dedicated people makes a lot more sense than a mass of drooling fanboy board warriors.

[LC]

My biggest pet peeve is that most companies try to attract the biggest fanboys to test instead of the best testers. Making people jump through hoops or enter contests and acting like testing is a priviledge is a bad plan. A small number of articulate, dedicated people makes a lot more sense than a mass of drooling fanboy board warriors.

I was recently reading a post in the "Suggestion" section of a game's beta forums. The tester complained about limited resolutions and total lack of wide screen support. Here are a few of the better fanboy responses (paraphrased):

"STFU its still beta why are you complaining about something that's free?"

"HIT ALT ENTER YOU DUMBASS"

"Nobody else has a problem with it. Go play some other game if you don't like it."

"buy a smaller monitor and quit whining"


The thread was locked for getting off topic and is buried on page 3 of the forum now.

[Mrbloodworth]

My biggest pet peeve is that most companies try to attract the biggest fanboys to test instead of the best testers. Making people jump through hoops or enter contests and acting like testing is a priviledge is a bad plan. A small number of articulate, dedicated people makes a lot more sense than a mass of drooling fanboy board warriors.

I was recently reading a post in the "Suggestion" section of a game's beta forums. The tester complained about limited resolutions and total lack of wide screen support. Here are a few of the better fanboy responses (paraphrased):

"STFU its still beta why are you complaining about something that's free?"

"HIT ALT ENTER YOU DUMBASS"

"Nobody else has a problem with it. Go play some other game if you don't like it."

"buy a smaller monitor and quit whining"


The thread was locked for getting off topic and is buried on page 3 of the forum now.

I see that a lot.

[SnakeCharmer]

My biggest pet peeve is that most companies try to attract the biggest fanboys to test instead of the best testers. Making people jump through hoops or enter contests and acting like testing is a priviledge is a bad plan. A small number of articulate, dedicated people makes a lot more sense than a mass of drooling fanboy board warriors.

The biggest fanbois will accept shit and thank you for it.  Trust me, I know:  I played SWG for 3 years.  Those same fanbois will generally be your biggest viral marketeers, and giving the devs forum blowjobs to take back to their bosses showing their doing the right thing.  Fanbois cause less problems.  Fanbois also generally feel a greater sense of loyalty to the NDA in the sense they'll say good things, but rarely say the bad.  The thinking is that if they only say good stuff, they'll be less likely to get in trouble.

Prebeta forum favorites and early adopter 'community leaders' that start their own fansites generally are the first to get invites.

[ajax34i]

Suggestions to the "Public" : Make up your freaking minds!

Do you want developers to listen to you and change/adapt to user expectations and desires, or do you just want to play free games and bitch about them publically?

If you, as a testing public, want to actually be able to affect a game project, you'll need to stop bitching about companies actually listening to you, and taking the time, effort, and risk associated with re-designing when appropriate if your collective feedback indicates.

I don't know what our "collective feedback" indicates.  It would be nice, you know, if whoever collects the feedback would summarize it for us too, just so we know that for example this week 80% of the comments seemed to indicate problems or displeasure with the paladin class, or whatever. 

You guys aren't giving an indication of bug report progress to the testers. We put in the effort, send in test reports, they go to /dev/null as far as we can see.  Who the hell has the patience to wait a month to see if their little suggestion had any impact and is in the next build version?  There's no "wow, that was particularly useful, thank you tester x, you're better than average", there's no "you guys have caught and reported 8209 bugs this week, thank you, we'll get them fixed."  You might need a tester community manager.

And yes, we're also not one person but many, each one of us knows what they want, but as a collective, no.

[Alkiera]

My biggest pet peeve is that most companies try to attract the biggest fanboys to test instead of the best testers. Making people jump through hoops or enter contests and acting like testing is a priviledge is a bad plan. A small number of articulate, dedicated people makes a lot more sense than a mass of drooling fanboy board warriors.

I was recently reading a post in the "Suggestion" section of a game's beta forums. The tester complained about limited resolutions and total lack of wide screen support. Here are a few of the better fanboy responses (paraphrased):

"STFU its still beta why are you complaining about something that's free?"

"HIT ALT ENTER YOU DUMBASS"

"Nobody else has a problem with it. Go play some other game if you don't like it."

"buy a smaller monitor and quit whining"


The thread was locked for getting off topic and is buried on page 3 of the forum now.

I see that a lot.

Me too.  IMO, the job of beta-era Community Managers is not to just lock the thread, but to remove all those people who respond like that from the beta.  Betas don't need people like that.  Odds are those people have never reported a bug, despite spending more time in the game than most others.  They are the beta equivalent to yes-men; the absolute last thing an alpha/beta project needs.

I agree with others that these kinds of things need to be much more rigid, especially in the early stages.  In the 'everything is done marketing beta' you can just let people play, but in the actual testing beta, you need to set testing goals, altering people's characters/accounts so they can test those things.  Like unit tests.  Players log in, and get an in-game message "Today we're testing leatherworking, your inventory has been replaced with a jillion leather pieces and other items required.  Go here to find recipies, tradeskill machines are there; if you run into any problems, /bug has special options for the leatherworking test.  Comment thread for todays test is up in the beta forum."

Track the number of items crafted; legit bugs reported, comments in the appropriate thread, etc.  Build some kind of beta-exp program that is filled by that.  Do the same thing with every scheduled test; sure, not everyone can make every test, but if someone is creating very little beta exp, they are either not contributing to the tests, doing what they want instead of what you want, or they're no longer interested in testing your game.  You can give their slot to someone else.  Other people have mentioned such things to be used to reward the beta players; IMO, the reward should be not getting kicked from the beta.  Heck, keep track of those useful tester accounts, and invite them to your next game's beta FIRST.

This way you can be sure that everything gets tested without requiring 20,000 beta invites, where even after inviting a ton of people, you still end up with a case where only two people tried alchemy, both had problems, and just gave up without reporting anything, as 'surely someone else has reported this already'.  You may work your way through a ton of invitees by weeding out people who don't do anything, or who do nothing useful; but the people you end up with will help you more than the rest of the sycophants ever would.

--
Alkiera

[Merusk]

Quick note before I actually read the thread:

People bitch about the unexpected, that's a guarantee in all facets of life.  One of the first lessons in SUCCESSFUL marketing & sales is, "Set expectations early, and meet if not exceed them."   If you promise something, deliver on it.  If you forsee a problem, point it out don't obfuscate it and hope nobody notices.  Someone will, and then you just look like you were a shitty planner or simply incompetent.

Mythic is taking it on the nose because it's an unexpected and sudden pause.  Things like that are automagically lumped in with part 2 of the above.  If they'd said months ago, "it looks like we might have to pause because things are getting backlogged." or "there will be a pause between beta 2 and 3 to implement ideas" at the start, nobody would think the less of them.

This is why Blizzard gets good graces for their (otherwise) intolerably slow creation process.  That expectation is set UP FRONT.  Thus 2 year expansion cycles are hand-waved away with, "Oh that's just Blizzard" rather than, "wtf man do you all just drink and game rather than doing your job all day?"

[Kaa]

This is why Blizzard gets good graces for their (otherwise) intolerably slow creation process.  That expectation is set UP FRONT.

Actually, Blizzard gets a pass on slow development because they proved that at the end of it excellent games emerge. Everybody knows that they polish, and polish, and polish, and the end result is worth it.

Kaa

[Righ]

2007 was an incredibly mediocre year for MMOs (*insert your own joke here!*)

A carefully considered and precise analysis given that the difference between a good year and a bad year is infinitesimal.

 

[Aez]

2007 was an incredibly mediocre year for MMOs (*insert your own joke here!*)

I don't know, it's not over yet and there is at least one really good hybrid mmo to be released before 2008.

[Morfiend]

I cant contribute much to this thread at this time due to  on what I want to address and examples I want to use. I will say this in regards to the outcry over the WAR shutdown. Also, no I am not in the WAR beta.

I think people take this as an indecation that the game is really sucking. I think part of that is because not as you say, we arnt happy if the devs actually do listen, but because most devs in "beta" would never think of doing this.
Now, if this is as they say, they are taking tester feedback and retooling a few thing that sucked. More fucking power to them. Release it "when its done". But since so many companies would never consider doing this, the general public (well, as general public as it gets here) see it more of a sign as "ABORT ABORT ABORT...".

I honestly hope that WAR is just doing minor retooling like Mark said. The more quality MMOGs we get, the more we will get in the future.

[sam, an eggplant]

It's probably worth noting that it's not necessarily significant that the many warhammer NDA violations over the past couple of days are universally negative. Anyone breaking NDA self-selects himself as an asshole. We get that. There's no need to respond to it and break NDA yourself.

[Merusk]

2007 was an incredibly mediocre year for MMOs (*insert your own joke here!*)

I don't know, it's not over yet and there is at least one really good hybrid mmo to be released before 2008.

What, besides TR, is still supposed to come out this year? I've seen and played TR, and if it's what you're speaking of, I'll disagree with you on the "good" part.