Alpha, Beta, Gamma -- public testing the right way

[Venkman]

Huxley, but they've been silent.

Otherwise, probably a bunch of animegrinds that even Signe misses :)

[Righ]

2007 was an incredibly mediocre year for MMOs (*insert your own joke here!*)

I don't know, it's not over yet and there is at least one really good hybrid mmo to be released before 2008.

What, besides TR, is still supposed to come out this year? I've seen and played TR, and if it's what you're speaking of, I'll disagree with you on the "good" part.

E's 'avin a larf, inni? The clue was the *insert your own joke here*.

[CharlieMopps]

Unfortunately, I've rarely been in a beta in which the Devs gave a crap what the community was saying. It usually gets to the point where the devs are pissed because all the community is talking about is how they can't get above 10FPS in the game and they'd rather they helped them balance the new dungeon they made. Listen, if I'm getting 10FPS, I sure as hell am not spending hours in your dungeon so I can get a migraine. You gotta fix the basics before people are going to want to test your more complex content. Even if that means optimizing the engine multiple times. That's to expensive? Well you can't afford to make a video game then.

[Salamok]

2007 was an incredibly mediocre year for MMOs (*insert your own joke here!*)

I don't know, it's not over yet and there is at least one really good hybrid mmo to be released before 2008.

What, besides TR, is still supposed to come out this year? I've seen and played TR, and if it's what you're speaking of, I'll disagree with you on the "good" part.

you are forgetting Hellgate.

[ajax34i]

Me too.  IMO, the job of beta-era Community Managers is not to just lock the thread, but to remove all those people who respond like that from the beta. 

[...]

Track the number of items crafted; legit bugs reported, comments in the appropriate thread, etc.  Build some kind of beta-exp program that is filled by that.  Do the same thing with every scheduled test; sure, not everyone can make every test, but if someone is creating very little beta exp, they are either not contributing to the tests, doing what they want instead of what you want, or they're no longer interested in testing your game.  You can give their slot to someone else. 


Other people have mentioned such things to be used to reward the beta players; IMO, the reward should be not getting kicked from the beta. 

I agree with your points about how to organize the testers so that they do useful work, but I disagree with the "reward" part.  I wouldn't beta test if the only reward is not getting kicked.  Actually, I should rephrase:  I don't beta test, since a. all the effort I put into reporting bugs seems to go to the void, and b. there's no reward.

[Aez]

2007 was an incredibly mediocre year for MMOs (*insert your own joke here!*)

I don't know, it's not over yet and there is at least one really good hybrid mmo to be released before 2008.

What, besides TR, is still supposed to come out this year? I've seen and played TR, and if it's what you're speaking of, I'll disagree with you on the "good" part.

you are forgetting Hellgate.

touche

[Quinton]

Most delays and schedule changes happen because of significant problems, not because devs are making a good game even better. A proper development cycle has time for responding to feedback built in. When the schedule changes that means that time was inadequate or the problems were much greater in scope than expected. The vast majority of the time projects that are delayed a lot end up sucking, because the delay itself was evidence that the project had problems.

I have never worked on a commercial software project/product that had a schedule that had enough time built in to react to external tester feedback that would require larger changes.  If *anything* significant goes wrong, you're almost guaranteed to slip the schedule because the schedule barely had enough to to finish the engineering work and basic testing in the first place.  If anything, gamesdev seems to be even more broken than the spaces I work in as far as "aggressive" schedules go. 

Nothing is sadder than getting great user feedback but knowing you're far too late in the cycle to make any changes beyond absolute show-stopper bug fixes because the product is in final certification testing and the factory is waiting on a golden master image to start the line and you can't slip the hardware schedule because you'll miss the product launch window and that means missing christmas or the carrier freaking out or whatever.  Some of these issues obviously don't apply to software-only projects, but the general problem is still there.

Few major development projects operate in a space where they have the resources to take all the time they need.  Even if they did, external factors (competing product launches, changing markets, etc) still intrude on the schedule.

I think Stephen's initial point is a good one -- what most companies describe as Beta is a much later stage where you're far closer to ship than allows for significant reworking of things.  More focused testing earlier on, before things start getting locked down and the change is less disruptive obviously would help.  The downside is at that point it is usually *really* early, management is worried about leaks or bad press early in the cycle, etc.

- Q

[Viin]

The comment that 'you get what you pay for' is very true.

I'm already busy, why would I spend a lot of time finding problems with your app?

I need some incentive to plug along, especially when I hit something that makes me want to stop playing.

[Venkman]

As I understand it, Beta is "content complete" with some bugs, mostly of the QA variety (it works, it doesn't, it needs to, it can be waived forever or for future refresh). But Beta (and any other stage) is largely an internal label. Basically, the owner of the project decides the stages, more to understand schedule and resource requirements. But the term "Beta" is pretty standard. You're testing the executive of your intention.

What we've seen over time is a wide range of "betas", some that are barely pre-Alphas and others that are shippable product. But even the latter isn't the perfect the average end user expects. It is enough for the company though.

It all comes back to clear communication. You don't need to tell the public the same stuff you tell your internal staff, but the concept of communication is the same. Like I said earlier, you get what you pay for. But you also get answers to what you ask for. The less you ask for, the less you define the feedback you're looking for, the less you can expect it all to be useful, good and non-duplicative.

This is what folks pushing betas-as-PR haven't ever gotten. And they wonder why more betas lead to negative public impressions than the positive buzz they're supposed to.

I'd rather they just drop public betas altogether. The odds of converting people to a box purchase just aren't good enough in my opinion. But I think the horse is too far from the barn at this point. Not having a public beta is almost as bad as merging servers in the eyes of your audience.

[Stephen Zepp]

I admit I was partially letting the horse out of the barn before the sun came up--when I wrote that little soapbox, we hadn't as a company announced either our next engine (Torque 2), or the fact that we were moving to a transparent development model.

Effectively, we are "opening a window into the engine development" with our next engine, at a point in the development life cycle where it's early enough to actually factor in responses to the designs, milestones, goals, and architecture from our transparent development community--and well before the engine is even capable of doing much more than compile and add a new object or two.

We plan on of course maintaining control over the development of the engine itself, but after literally resource-years of research and strategy/planning, we've decided that the earliest feedback possible, and most importantly continuous feedback will be a fundamental contributor to success. Watching the responses to the announcement I referred to in my original post, it made sense to me that a less transparent, but still "very early" set of phased interaction with the player base should be extremely effective if handled well and managed properly for most games as well.

It all comes back to clear communication. You don't need to tell the public the same stuff you tell your internal staff, but the concept of communication is the same. Like I said earlier, you get what you pay for. But you also get answers to what you ask for. The less you ask for, the less you define the feedback you're looking for, the less you can expect it all to be useful, good and non-duplicative.

Extremely well said, and exactly what I mean :)

[SnakeCharmer]

It all comes back to clear communication. You don't need to tell the public the same stuff you tell your internal staff, but the concept of communication is the same. Like I said earlier, you get what you pay for. But you also get answers to what you ask for. The less you ask for, the less you define the feedback you're looking for, the less you can expect it all to be useful, good and non-duplicative.

Extremely well said, and exactly what I mean :)

Interesting twist on feedback...

Developer A asks for feedback on <whatever>.

The responses from Alpha / Beta testers are hidden to everyone but the devs/mods?  What DQ posts, I can't read, and vice versa, but YOU can.  That way, it's possible the responses are more subjective than the testers being influenced by other testers in their answers which would help to elimiante forum clique bandwagoning?

[Cheddar]

The public is your best QA.  BUT... you need to have a plan to pull back when they figure out what is wrong (there will be more than one thing).  I have nothing but respect for the WAR Beta... recent events have taught me how much random people can teach you about their product.

[Quinton]

The public is your best QA.  BUT... you need to have a plan to pull back when they figure out what is wrong (there will be more than one thing).  I have nothing but respect for the WAR Beta... recent events have taught me how much random people can teach you about their product.

I sure as hell hope these shops also have decent in-house QA -- external feedback is super useful but having a team in-house to qualify builds before they go live, verify that the bugs supposedly fixed in the release really were fixed, check for regressions, etc is pretty essential to any level of success.

- Q

[CharlieMopps]

It all comes back to clear communication. You don't need to tell the public the same stuff you tell your internal staff, but the concept of communication is the same. Like I said earlier, you get what you pay for. But you also get answers to what you ask for. The less you ask for, the less you define the feedback you're looking for, the less you can expect it all to be useful, good and non-duplicative.

Extremely well said, and exactly what I mean :)

Interesting twist on feedback...

Developer A asks for feedback on <whatever>.

The responses from Alpha / Beta testers are hidden to everyone but the devs/mods?  What DQ posts, I can't read, and vice versa, but YOU can.  That way, it's possible the responses are more subjective than the testers being influenced by other testers in their answers which would help to elimiante forum clique bandwagoning?

Yea, sounds neat, but the shit people are going to tag team your ass on prior to release is the same shit they will afterward. You do, after all, want to know what the bandwagons going to be after release right?

If you want everyone to stop harping on the same issue, my suggestion would be to fix the issue so it isn't a problem anymore.

TesterA: XX sucks!
TesterB: Whats XX?
TesterA: Go over there.. yea, click there...
TesterB: Holy crap, that DOES Suck! XX sucks!

[Venkman]

I think even before fixing you, you dull the noise of a thousand people reporting the same thing with slightly different words. There's been some excellent reporting tools for players over the years. There's been some horrific ones too, amounting to little more than a text field.

Basically, treat your testers like a QA group, within reason. Those that don't like it can leave. When you get hundreds of thousands of people in a beta, it's either a PR exercise or a Stress Test. So  either the feedback you're going to get is going to other forums anyway, or the only feedback you need is them hitting your servers from a buncha different directions. This stage is way late in the process. Everything prior was more valuable and should come from a much smaller group, one you can manage.

Wisdom of the Crowds only works when there's a clear objective :)

[CharlieMopps]

I think even before fixing you, you dull the noise of a thousand people reporting the same thing with slightly different words. There's been some excellent reporting tools for players over the years. There's been some horrific ones too, amounting to little more than a text field.

Basically, treat your testers like a QA group, within reason. Those that don't like it can leave. When you get hundreds of thousands of people in a beta, it's either a PR exercise or a Stress Test. So  either the feedback you're going to get is going to other forums anyway, or the only feedback you need is them hitting your servers from a buncha different directions. This stage is way late in the process. Everything prior was more valuable and should come from a much smaller group, one you can manage.

Wisdom of the Crowds only works when there's a clear objective :)

Building a decent ticketing system is not that hard. Hell, you can do it in Microsoft Access for Gods sakes. Next you need a "Known Issues list" in your beta forums, and what you're doing to correct those issues.
i.e.
Known Issues:
Rendering Bug - All weapons appear as Staves <-- We're waiting until the new models are redone to avoid an even worse bug.  about 2 weeks or so...
Crash at login <-- we think this is related to anomalies in end user PCs. Please send us your DX logs so we can look for commonalties.
Class imbalance <-- we aren't going to start balancing for at least 3 months. Just go out and kill stuff, we'll worry about this one when all the classes are in.

Then you need a decent full time moderator to link to the list and lock threads. With $30 million budgets it's drop in the bucket.

[sam, an eggplant]

Building a decent ticketing system is not that hard.

You're right in that it's not difficult to setup a decent ticketing system, but building a great one is an incredible amount of effort on both design and implementation tiers. You can't grab remedy or RT off the shelf and expect it to work efficiently; you need to create and polish highly usable task-based workflow, knowledge, and asset management systems around it. Otherwise you spend all your time in QA managing timelines, documentation, and project plans instead of actually completing tasks.

I'm looking at it from a MSP perspective, because I work at one and deal with this stuff every day. From the development side Bugzilla may cover all your needs. But you can't come out and say "ticketing is trivial" and expect everybody to nod their heads. It's not.

[Venkman]

None of this stuff is trivial, and I don't think anyone thinks that. For me at least, it's not a question of how easy or hard something is to do. It's a question is whether is something is worth doing. So far we've seen a lot of priority be given to development and fixing combines with a general expectation that the answers will come from the testers. Even those companies with great internal tracking tools and processes don't seem to, by and large, apply that to the playerbase. It's nowhere near "easy" to do that, but it's absolutely worth it.

[CharlieMopps]

Building a decent ticketing system is not that hard.

You're right in that it's not difficult to setup a decent ticketing system, but building a great one is an incredible amount of effort on both design and implementation tiers. You can't grab remedy or RT off the shelf and expect it to work efficiently; you need to create and polish highly usable task-based workflow, knowledge, and asset management systems around it. Otherwise you spend all your time in QA managing timelines, documentation, and project plans instead of actually completing tasks.

I'm looking at it from a MSP perspective, because I work at one and deal with this stuff every day. From the development side Bugzilla may cover all your needs. But you can't come out and say "ticketing is trivial" and expect everybody to nod their heads. It's not.

I think I'm trying to say "effort" is the important part. The last beta I was in was Vanguard, and there was no "effort" at all put into the ticketing system, finding bugs, or listening to the players. They basically were building a game they wanted, we were only in beta as a marketing ploy. In fact, most of the Devs treated us like an annoyance (there were some exceptions)

Would remedy work off the shelf? No... I actually use it every day... lol But, with some modifications I think it would work. As with most business today I think a lot of these company's are short-staffed, putting together budgets that are very unrealistic, and working on schedules that are just plain silly. Conan pushed out 6 months? WTF? 6 months off schedule? That's just plain ridiculous.

[Mrbloodworth]

I admit I was partially letting the horse out of the barn before the sun came up--when I wrote that little soapbox, we hadn't as a company announced either our next engine (Torque 2), or the fact that we were moving to a transparent development model.

Effectively, we are "opening a window into the engine development" with our next engine, at a point in the development life cycle where it's early enough to actually factor in responses to the designs, milestones, goals, and architecture from our transparent development community--and well before the engine is even capable of doing much more than compile and add a new object or two.

O really?

[Stephen Zepp]

I admit I was partially letting the horse out of the barn before the sun came up--when I wrote that little soapbox, we hadn't as a company announced either our next engine (Torque 2), or the fact that we were moving to a transparent development model.

Effectively, we are "opening a window into the engine development" with our next engine, at a point in the development life cycle where it's early enough to actually factor in responses to the designs, milestones, goals, and architecture from our transparent development community--and well before the engine is even capable of doing much more than compile and add a new object or two.

O really?

Really! (new discussion thread in a more appropriate forum here).