WAR beta test idea...

[Shapechanger]

I know we have a WAR thread up, but this thread is about special beta testing - and it could actually be applied to any PvP based MMO.  I decided to put it up at F13 because of the breed of posters found here...  So here goes a long ass post about a not new type of beta idea for Warhammer:

Summary for the lazy:
old beta - dump testers in game see what they do launch the game and watch em find all kinds of new shit to make you nerf classes.
new beta - dump testers in competitive focused mandatory organized pvp testing with maxxed out characters so they are acting like they would 6 months after live so you find out the ways they are going to abuse class-sets and abilities before launch...


Mark Jacobs once held forth that:

Racial imbalances in an RvR game are always going to be a problem no matter what the races or lineup look like. These problems are exacerbated when the groups that are taking part in the game are soooo different from each other. This has been one of our main concerns from the beginning and getting it right is crucial to this games success. A few simple rules:

1) The sides/realms will never be balanced in terms of population/ unless the developer forces the issue. OTOH, no smart developer would ever force the issue if they want their game to succeed. Once you force players to play in a certain realm/race, you will allienate your players.

2) An RvR game that features lots of complex systems can never be perfectly balanced unless all the systems are exactly the same across all the realms. Using that setup for WAR would be dumb, dumb, dumb and we are not doing it.

3) We are making a WH-based game not a 100% faithful reproduction of the WH material (nobody could do that, there is way too much material that has evolved over decades and so much of it is different).

4) No one race can be uber. The game has to be fun for all players or in the end, it will be fun for no players. This is an RvR game not a solo, PvE game. Races/Realms will have their strengths and weaknesses, advantages and disadvantages but the game system has to work 1:1 or it won't work 10:10 or 40:40.

5) We must have complex systems that are capable of helping to ease the imbalance between the realms remembering that no matter what we think the players will like to play, we will not be right all the time.

6) Remember rule #1.

Mark

Now that is a fantastic quote and it shows understanding of something he should understand.  It gives confidence.  Yet something puzzles.

It's beta.  Why?  Because Warhammer is using an old-fashioned beta.  Like EQ style, or DAoC style.  It is not even computerized but spread-sheet based, from what I've heard, so it's going to take a lot of man hours to select good testers, which means it'll just be any old pile of testers - who has the time?  Really, it should be automated.  Are there other styles of betas that are more successful at approaching PvP balance? Turns out, yes.


I'll quote something I say later about what beta should do, "Ultimatley, the competitive element of beta drove outside testers to come up with creative strategies, and this focused pvp beta (with the ability to create mature pvp characters on the spot), simulated a mature post-launch experience."


Old beta:
In an old style beta, players are invited to the game and they experience the content. They level from 1 on up and eventually some group up and go pvp against some others.
This is a great and necessary type of beta for understanding how new players will develop in the game world.  It has it's uses, particularly during marketing or stress betas.
However, this type of testing doesn't do jack or shit for your pvp balance.


The reason why is that type of organic levelling has nothing productive to do with PvP.  PvP is all about competition, and dog eat dog.  It is about players obsessing over perfectly built groups and designed gear, ala DAoC 8 mans. Aka, PBAE groups, for instance.

Why didn't DAoC beta discover PBAE groups?  Because when players are forced to start up a game they have no experience in, they just level up whatever and get together with whomever to scratch ass on the PvP field.  But it's just for fun, not for blood.

Once the game is live, the blood starts flowing. Players discover how to combine classes and abilities.  They begin premaking groups.  They begin experimenting with groups.  They discover the "we didn't anticipae this so we have to nerf" flaws.  All of this takes time, however, because these different classes must be combined in ingenious ways to take advantage of game mechanics. Game mechanics that were never abused in this old fashioned beta.


Now what I think would be a great idea, would be to use two kinds of beta tests for Warhammer. 

• In one beta type, you have the standard lemming launch - organic levelling. It has it's uses but it is no good for getting PvP ready for launch.  If this is your only type of beta then you entirely depend upon in-house to prepare your PvP for launch, which is foolhardly but I'll get back to that.
• In the other type, you have PvP focused testing.  In this type of testing, you'd simply emulate what Guildwars did in their Alpha testing program, which was an awesome experience if you were lucky enough to participate.

My understanding of the way this worked is there was a special beta, forgive me if I am wrong about how they did it.  This beta had mandatory attendance twice a week for PvP, on Tuesday and Thursday evenings. If you couldn't commit to the times, you couldn't be in this beta.  DXDIAGs were mandatory as well so tech could quickly troubleshoot.

The beta was guild-based.  Each guild was headed by a company employee.  The employee would summize a report once a week, and the reports would be consolidated by one person for senior dev review.
The employees guild was small at first, with chosen people.  Through participation in all calls and other PvE testing events, the guild would earn points by which to invite more testers.  The tester had to apply and someone in the guild had to know them and vouch for them.  If the new person missed 3 all-calls they were booted, no exceptions.

This kept beta tight. People knew each other, and were accountable to show. No free riders.  Serious and devoted testers only.  Forums were based upon guilds, private for your guild, as well as larger general and bug.

People were kept motivated by the ladder.  The results of the guild vs guild matches were tallied and put up on a ladder for people to see.  You wanted to do good, so you'd be higher on the ladder.  Bringing in that competitive element really made people work hard.


What would happen test-wise is someone at the company would either put out a message to the guild leaders that on such a day they want to test 'these types of groups', or allow them to make their own.  Because all testers had access to a range of dev commands such as /level /make gear /learn spells etc, they could create custom PvP characters on the spot suitable for any prebuilt group idea.

In this fashion, devs would then pit guilds built in certain ways against other guilds.  Then during the matches, the devs could hover and watch all the options and numbers the competing guilds were using.

It doesn't seem a reasonable use of time to simulate 10 vs 30, as could be found in open skirmish RvR.  It would probably be most productive to keep these tests group VS group, or in some cases 2 groups VS 2 groups.  After all, if 10:10 then 40:40.


Ultimatley, the competitive element of beta drove outside testers to come up with creative strategies, and this focused pvp beta (with the ability to create mature pvp characters on the spot), simulated a mature post-launch experience.


This is important.  A pvp game, any pvp game, is going to have a great deal of complexity.  In a game where PvP is the endgame, how this PvP comes out post-launch is incredibly important because changes will be seen as nerfs.  Dealing with the majority of these early in beta is paramount.

In an old style beta, it is the responsibility of in-house to do this.  Yet 20 or 30 people cannot possibly approximate what will happen live.  They are the same group of people who played the game from infancy - they are bored with it.  They know what each other will do.  They are immersed in office politics and will not stab their managers idea in the back like an outside tester will do.

Outside testers bring many fresh new ideas to the PvP puzzle.  They will find the flaws because it is all new and fresh.  They will challenge designs because they are not trying to get a promotion or being nice to their co-worker.  It will create a healthier more vibrant PvP experience with less flaws and exploits.

Racial imbalances in an RvR game are always going to be a problem no matter what the races or lineup look like. These problems are exacerbated when the groups that are taking part in the game are soooo different from each other. This has been one of our main concerns from the beginning and getting it right is crucial to this games success

PvP balance is an ongoing process that will never end.  Yet in a game such as Warhammer, it is crucial.  Why not do as much as is possible?

This idea isn't terribly new, Arena Net did it for Guildwars and they had arguably the most balance launch PvP we've seen in an XORPG to date. 

Why not take that and run with it?  What do yout think?

[Margalis]

Magic: The Gathering has an in-house thing called "The Future Future League" (no, not just the future, the future future!) that tests upcoming sets in a competitive environment well before they are released. I would compare that to what you are proposing in that those testers don't have any rareity issues, they can use any number of any cards right off the bat.

I agree with your general point. In a competitive environment all the levelling and experience along the way becomes irrelevant. What matters is what happens when players have access to all the spells, items, classes, etc. It really is basically a different game.

[Shapechanger]

Exactly, and it only seems common sense to prepare for that game, because that is the serious game people will be competitively playing.  Yet MMO betas in no way address it.

It's a recipe for having a pissed community six months out.

[Venkman]

For balancing, a good idea.

However, you also need to assess who's going to bother getting there. Only then can you know if there's some class ineffectiveness, absurd level requirement, insane failure penalty or general lack of interest. For example, rolling max-level in DAoC would never have shown the buffbot issue that only came up later due to players finding that, for the most part, the class was both required for RvR and boring as heck to play.

Many betas do have focused activities. They just do them in phases before they invite the general public into "open" beta, which is really little more than a marketing exercise, or "stress" beta when they are looking to simply testing zillions hitting login and world servers at once.

[Johny Cee]

A focused beta with a small group of testers would not necessarily pick up any more issues then the present beta system, simply due to the shear complexity of the system.

How long in DAoC before the PvE pbaoe groups became the dominant method?  As I recall, months of live.  Players were still wrapped in the EQ mezzer/healer/tank strategy.

How long did it take to develop the classic "tank train" setup for 8 v 8?  A year or so into live?  And by and large,  it didn't become the effective tactic until changes to determination, charge, and purge RAs made tanks virtually immune to CC.

Besides,  a system as complicated as DAoC RvR ended up being,  a couple small changes to class abilities could end up changing the balance of how a realm performed when used in concert.  Speedwarp + sorceror/cabbie lifetap changes + buffs to main healers + increased caster survivability/damage through ToA gear = caster assist groups

[garthilk]

In terms of Beta and WAR here's some quick quotes.

We do have testers with testing tools and 24/7 access to devs. That's our product quality division. We will be hiring more of those people as there is more content to be tested.

The "open beta" is a chance for us to find bugs that required a critical mass of players to find, and to help us prioritize the bugs that just got backburnered during the rest of the development process. And yeah, it's to build buzz. That's not exactly a secret.

The early closed phase will require participation in a private forum with a hardcore NDA.

The middle closed phase will include guilds and websites, where leaders are responsible for reporting the feedback and opinions of their groups. Everyone is responsible for sending in bugs in-game.

Older MMOs used bug reports as basically a "to do list." It was possible for a lead person to not be aware of every item on the list, and for items to move to the top of the list once a lead person was made aware of the problem. Tools are more sophisticated these days.

If you aren't employed by Mythic or our various partners - GOA, EA, etc - you're testing the experience, the fun, the gameplay. If you happen to encounter and report bugs, that's GREAT, and we're very grateful because the job of testing an MMO is too big for a paid staff to do fully. But you should not expect to be treated as a professional, or get angry when you don't get immediate feedback for your emails and posts. You essentially must trust that your material is being read.

For example, every feedback form sent to the Heralds is read and counted, and Richard does data crunching on it all. But you will only rarely get a reply, because it is far more important for us to read it all than to respond, and we MUST choose between the two.

I've run a few betas in the last five years

Yeap, that's where banning comes in. Unlike DAoC, we are going to be much, much stricter this time in enforcing the rules for the beta. So, if somebody is treating the beta as a demo, buh-bye, see ya and don't let the Orc hit you on the way out.

Folks,

Some beta info:

First, the no-nos

1) You cannot work for another game company

2) You cannot work for any of the item farming/selling companies

3) Constantly asking for invite will not get you anywhere (nor will bribes)

Now, the yes-yes part

1) Good, solid posting here is a plus. It can't be of the butt-kissing kind but nor can it be of the "U Suxxor" type. We pulled in people from the sites we frequented for DAoC and we'll do the same here. What we are looking for are people who can communicate their thoughts in a clear and concise manner.

2) The willingness to write reports (short of course) and respond to questions from the team are necessary. If you are looking to just play the game early, wait till the final stage (playtesting) but if you want in and want to help make this product great, be prepared to talk to us/write to us about your experiences (which is why #1 is so important).

3) You must be willing to do what we ask when it comes to testing. If we say we need certain things tested, you must test them. Completing quest #2 when we asked you to do scenario #1 will not keep you in Beta.

4) You must be enthusiastic about the game (enthusiasm != blind devotion). If you are not psyched to be part of the Beta, give your slot to someone who is.

5) You must remember that you are in a beta test and as such, bad things will happen in the game. Things will break, things will be changed and while we expect it to be fun, there are many things that will occur during beta that will be annoying. You must be able to deal with it responsibly and not go all Raging Bull on us.

So, just some information to keep in mind.

Mark

Now the reality is that there is already about 600+ people in beta. Mostly EA, GOA and Mythic people. With the more public phase of beta around the corner they've stated they expect to have about 1000 people by the end of the next phase. Leaving about 400 hand picked people to get in.

[tazelbain]

I'd be interested in knowing how they are going to test RvR.  DAaC's RvR didn't seem tested beyond it's basic functionality.  Very costly to have redesign it and by that time you have already blown the first impression.  Testing and getting feedback on the design isn't a part of testing?  You don't want to be like AutoAssult right?

[AlteredOne]

The difference from DAOC would be that Camelot RvR was an endgame activity.  Originally there were no battlegrounds, and everyone felt that they needed to be at least in their 40s to attempt RvR.  In truth I do remember some very early raids where even lowbie 20-something characters tried to participate, but the code prevented such low level characters from literally even touching high-level toons. 

By contrast, Warhammer claims to integrate RvR at every level of the game, from the newbie zones onward.  Hence it should be feasible to test the general gameplay and RvR even from low levels, and EA-Mythic will not rely on a few hardcore beta-testers to grind their way to high levels just to attempt RvR.  That's the plan, anyway.

[tazelbain]

I hope so.  Nothing has me more excited about WAR more than the idea that one could make a nooblet, run off to fight with/against the other nooblets, and never worry about leveling again besides spend xp that I earned while having fun.  Its tall order, but I think back to the days when I first starting PvPing in the BGs.  At first I was very skeptical.  PvP was the domain of the elite min/maxers and uberguilds.  But BGs hinted that under the correct conditions everybody can participate. Making PvP accessible to everyone exactly in the same way PvE is accessible.  And even more people will flip over like I did.

(then you add a shadowbane expansion, where people can go to a separate area and burn each other cities.)

[slog]

Why people are so excited about Beta testing a WoW clone from EA is beyond me....

[Strazos]

Um, if the PvP works as advertised, it's not even going to be Close to WoW.

[garthilk]

Exactly, and it only seems common sense to prepare for that game, because that is the serious game people will be competitively playing.  Yet MMO betas in no way address it. It's a recipe for having a pissed community six months out.

The reality is though in order to be played competitively there is a certain learning curve. Even when given artificially inflated characters, players won't know what to do with those characters. There's no learning curve. The best way to really test competitive PvP is to allow the knowledge and strategies evolve on their own rather than artificially introduce them. Much of that takes time, and evolves as part of the games release. PvP at the beginning of DAOC, for example was nothing like the PvP three months later. Evolution of gameplay at it's best.

[Nonentity]

They're right, for the most part.

When you stop hand picking people and start spinning the wheel of fortune, you're not going to get good testing feedback from the people that were randomly picked. At that point in time, they just want people in the game playing, and they just observe and see what happens with a critical mass of people.

It's when they start spinning the wheel and letting people into the ALPHA (read: me) that I get somewhat nervous (read: DDO).

For all intents and purposes, WoW, from Beta 1 on, was purely for the critical mass of it. Sure, big gameplay exploits were found by some people, but those people were NOT the majority. The majority were just running around like dillweeds to be in the mass of it.

[tazelbain]

I would think it'd be hard not to get a chorus to tell you what you want to hear by hand picking beta testers.  I'd bet the EA internal beta-ers would be more diverse.  How do you get a good cross-section of gamers in the beta so you can get feedback to give you a broad overview?  Stuffing your beta with people who want to "break into the biz" so they write long-winded reports about armor mitigation rates isn't going to do that.

[slog]

Um, if the PvP works as advertised, it's not even going to be Close to WoW.

eeheheheheeheheheheh 

[Chenghiz]

eeheheheheeheheheheh 

Are you trying to be an ignorant smartass, or merely annoying?

[tazelbain]

I do not begrudge daslog's right to giggle like schoolgirl nor his skepticism for games in development.  The MMOG industry and EA, in particular, has earned it.

[Lt.Dan]

Um, if the PvP works as advertised, it's not even going to be Close to WoW.

eeheheheheeheheheheh 

Are you trying to be an ignorant smartass, or merely annoying?

Here, let me spell it out for you

S H A D O W B A N E

Sins of the father, doomed to repeat history and all that.

[tazelbain]

Wait... are you guys damning WAR because it is too much like SB or because it is not enough?

[Signe]

They are damning slog for his charming girlish airs. 

[Xanthippe]

Those beta expectations posted by MJ look very much like expectations written by someone who is filling out a paycheck; which is well and good if one actually receives a paycheck but what do beta testers get exactly besides getting to look at a game early? 

Something about having the "privilege" of beta testing somebody's game like that rubs me the wrong way.  I'd rather just wait until launch to beta (which somehow is what all MMOs to date have felt like upon launch anyway).

[Coplann]

I think Mythic has enough experience by now to at least have a certain idea how class/balance/design changes can affect the dynamics of a MMORPG.

as long as they ensure that there is no "perfect" setup for a pvp/rvr-group at any time it will be decent in the end. In DAoC the fotm classes for gank groups change every now and then, but mostly its 2-3 healers, speed/CC and 3-5 raw DPS classes depending on realm (be it a non-CCable tank train or a lifetap caster train with body resist debuff...). in the end of like 15 classes per realm only maybe 5 are consistently grouped in the competitive RvR-environment. they need either way way less classes or simply allow every class to perform decently in a group vs group setting.

heh, as long as they dont do a "Trials of Atlantis"-move in WAR I will be a happy camper anyway, lol.

[Xanthippe]

heh, as long as they dont do a "Trials of Atlantis"-move in WAR I will be a happy camper anyway, lol.

Yes, if they do something like that, I won't play it.  It drove me from DAOC.