Mechwarrior Online

[Falconeer]

No, it's quilt. It's a snow map, you know.

[Zetleft]

Full body fishnet

[Falconeer]

Long writeup about the squashed HUD bugs. You might like it if you can understand theyr language.

Hey everyone,

With the HUD bug hotfix coming out soon, I’ve been asked to provide a write up about what we found, and explain a bit about why it took so long to resolve these issues. I was brought on to help resolve these issues relatively late in the process, once we suspected that the problem did not reside within Piranha’s internal code, but was in-fact a pre-existing issue either in the game engine we were using, or in some other external system used by the game engine in turn.

As you all are probably aware, MWO is built on CryEngine. CryEngine uses the very popular Scaleform flash library for all of its UI, and additionally has replaced the default windows implementation of the standard template library with a third-party implementation called STLPort.

Our first challenge in solving this issue was simply finding a reliable process for reproducing and detecting HUD corruption issues to begin with. The issue appeared to be affecting random bits of the HUD UI with extremely random behaviour. The CSR’s, QA, and community testers spent a lot of time scraping through community posts trying to collect common themes and patterns, and testing what they found to see if there was any discernible correlation that might help us out. By the time this process was completed, the QA team had run hundreds, if not thousands, of games in the testlab under varying circumstances, visually inspecting each screen for any signs of HUD corruption and collecting any common factors, such as Windows versions, common hardware configurations, GPU driver versions, etc. Several leads were investigated during this process, including 64-bit issues, alt-tabbing, and low memory conditions, but unfortunately none of these were capable of providing us with a 100% reproduction rate.

When I was brought on, Jin, our senior graphics engineer, and Sean, our UI engineer, had determined that there was a major issue somewhere in the CryEngine to Scaleform layer that was basically causing C++ calls made by the game to invoke incorrect flash functions in Scaleform. This was after significant effort was spent tracking down and investigating other suspected causes, such as random heap memory corruption, or threaded race conditions.

We first spent a considerable amount of time attempting to reproduce the issue with a debug build and breakpoint a failure case, in the hopes that the error was obvious, but also to get an idea where the issue truly resided so we knew where to focus our efforts. Eventually we did manage to reproduce one of these issues with a debug build, and what this effectively confirmed for me was that there was almost certainly an issue somewhere in CryEngine itself. Starting from here, it became important to understand as much about the interactions between CryEngine and Scaleform as possible.

What followed was about two days of line by line debugging and auditing code in order to figure out what the intended behaviour of these systems were. I’ve got several pages of notes from this process, outlining general flows and several class structures. Once I had a good idea of how this system was intended to work, I spent some time building up a set of validation and verification routines to detect when these subsystems entered an invalid state. I was specifically interested in a convoluted set of lookup structures which mapped string handles to Scaleform object instances. The end result was an extremely slow build, but which would crash and generate stack call data whenever it detected that the CryEngine scaleform layer had entered an invalid state.

This immediately produced results for us. CryEngine has built a global string table subsystem, which is used to accelerate specific string operations as well as save memory, similar to the string tables kept by some other languages. Additionally, users can create handles to this global string table in the form of CryName instances.

Internally, we found that this global string table is built upon a case-insensitive string hashing mechanism, which it used to determine whether or not an entry exists within the global table or not. This case insensitive compare is most likely a result of the case insensitivity of the windows filesystem, and the probable roots of this CryName system in resource handling. In the case of Scaleform it was actually an error. This meant, if the UI team had ever added two flash elements to a single player which differed only in case, one of those elements would have been essentially inaccessible to us.

Unfortunately, it turned out that the case-insensitivity issue was not related to the current HUD corruption issues on production, so for now I revised the verification routines to similarly ignore case, and we continued with our debugging.

Again, we reproduced a real instance of the HUD issue with the revised verification code, and this time we managed to isolate a single function call as having put the system into a bad state. This was enormous progress, as we could now focus on this single specific function. We audited the function line by line, and unfortunately determined that while messy and initially confusing, it was actually safe, and should not have resulted in the behaviour we were observing. It’s important enough to note that internally this issue had an extremely low reproduction rate, and that this function was invoked hundreds of times every time a client loaded into a game. For this reason it was impossible to know ahead of time whether or not a given single call would produce the behaviour resulting in corruption. Instead, I went back and added extremely heavy logging to this specific function in order to determine the flow sequence that was resulting in the HUD corruption.

Once again we reproduced the issue, and we were left with baffling results. The logs indicated that the STL library itself was in error; but this was nearly inconceivable. The data structure from in question was an std::map, which was implemented as a red-black tree in STLPort. This is an extremely efficient self-balancing tree structure with excellent performance characteristics. STL data structures for user defined types take a user defined sorting criteria. This sorting criteria must guarantee a property known as strict-weak ordering. I had already examined this sort operator for correctness, which was one of the CryName structures in question, and determined that the comparison operator definitely obeyed strict-weak ordering. In fact, since the global name table guaranteed one instance for any given string in memory at the time, CryName used absolute memory addresses as their sorting criteria. Now, this sounds terribly unsafe, and I spent a good while verifying that a CryName would never be deleted from the global table and re-added at a different memory location while being stored as a key value in one of these mapping structures. CryEngine guarantees this by keeping reference counts for every CryName. However, what we DID find was extremely surprising.

STLPort has a whole category of data structure assessors which take member function template parameters, rather than template parameters defined by the data structure types themselves. This appears to actually violate the STL spec, as the function signature for a map access is definitely defined in terms of the key type for the data structure itself. The problem however manifested as follows:

CryScaleform layer attempted to lookup a structure which did not yet exist in their cache layer, this triggers a load and bind operation. This lookup operation involved passing a string key value as a const char * to the map operator[] function; however the operator[] function was templated in such a way that the key value provided actually expanded to a const char * at compile time rather than a CryName instance.

What followed was a terrible failure case, where a CryName instance would be constructed, inserted into the global name table, and compared against a given node within the map, and then destroyed and completely removed from the global name table; only to happen all over again at the next level down in the red-black tree.

This definitely threw away all concept of strict-weak ordering. Now, if the global name table happened to allocate the comparison CryName object at differing memory locations, what would result was essentially a random walk through the tree structure, and the final map node returned to us was completely unpredictable. The only reason any of this system tended to ever work was because of the tendency of the memory allocation subsystem to reuse the same memory address when reallocating one of these global name table entries. As soon as the memory subsystem deviated from this behaviour, the function in question would fall apart and generate completely unpredictable results. This was the root cause of our HUD woes, and once found was relatively easy for us to address.

Unfortunately this leads us to very serious bug number two. Once the mapping structures were repaired and this subsystem was put back into a good state, QA located another bug; this time relating to a race condition between precached texture loads. This manifested only as missing minimap textures. Jin, our senior graphics guy, very quickly located this issue and identified it as an incorrectly scoped lock during the precache phase in CryEngine. With these two issues finally resolved, the QA reproduction rates finally dropped to 0%.

Hopefully this gives you some sense of how deep these issues ended up taking us, and why it ended up taking so much time.

tl;dr: The HUD bugs were a result of undefined behaviour in CryEngines scaleform wrapper library, which took considerable effort for us to track down and fix.

[Ironwood]

I actually like it when companies do this.

[Phred]

I actually like it when companies do this.

The problem with it is the length of time it takes a highly skilled worker to write that up when they need him fixing bugs/coding. :)

[Merusk]

Good news: Missiles seem to be viable once more.

Bad news: Matchmaker is clearly fucked as one side will always have 3-4 LRM or SRM boats while the other is PPC/ Brawlers.   

At first I thought it was just my perception, but then I started playing chassis with no missiles and it was still true.

[Falconeer]

A very important little bit that surfaced yesterday on Ekman's twitter:

Player's question:
Anything in the pipe to adress all the convergence issues and topics that are rising more and more? Where do you stand on this?

Bryan Ekman 's reply:
it's on our radar for review. The design team is working feverishly on a big tuning pass for the 21at #mwo patch.

[Ironwood]

See, this is my perception also, but I'd like to know WHY.  They've never admitted to changing missiles, but you see more of them and you see them working, even on brawlers like me.

What gives ?

[Merusk]

They upped the speed on them, so that had some effect.  Beyond thati don't have a clue other than it seems more folks are using 20s, so that extra 5 missiles seems to assist a lot.

[Zaljerem]

They upped the speed on them, so that had some effect.  Beyond thati don't have a clue other than it seems more folks are using 20s, so that extra 5 missiles seems to assist a lot.

I had some excellent games last night with my AWS-9R missle boat .... I got a ton of XP from using Tag, 4xLRM-15s+Artemis coming down on your head really hurts, Sensor Decay helps big time. Just need to finish Mastering it for the other module slot (Sensor Range) and it will be all it can be.

[Arthur_Parker]

See, this is my perception also, but I'd like to know WHY.  They've never admitted to changing missiles, but you see more of them and you see them working, even on brawlers like me.

What gives ?

Dunno but some people get bored or want to check if a weapon has changed and then other people copy them, there's a sometimes noticeable cycle to it.

[Ironwood]

Yeah, that's fair enough, but it doesn't explain why they rained down death on my head when they're usually ignorable.

:)

I suspect shit's being sneaked in.

[Arthur_Parker]

Yeah could well be, I must have checked small pulse lasers 6 times in beta as I couldn't believe they wouldn't tweak them into semi useful and forgot to mention it, think for about 2 weeks once they were ok'ish.

[Arthur_Parker]

Link

Update: PPC/ERPPC cooldown bumped to 4.0s.  

Flamers re-configured to heat a Mech up to 90% of it's heat threshold and is capped at that point. If the target Mech fires, chances are they will shut down. The more flamers you have on your Mech, the faster it will take the target Mech to its 90% cap. We do not want the flamers to directly shut down a Mech which would introduce a very nasty stun lock.

To be included in the May 21st patch.

Forgot about the level of assumptions that would be made about the PPC change.

Let me clarify, this is not a change to "nerf" boating/high alpha builds/"poptarting". It's a change to put the refire rate of the PPC back in line with the rest of the large energy weapons.

Under investigation right now:
To curb boating with high alphas... we are testing a system that induces a heat scale when firing multiples of the same weapon within a specific time frame. The more weapons fired of the same type, the higher the scale climbs.

Possible internal damage on certain heat levels of the player's Mech. If you blast past your shutdown threshold and then some, you start to take damage internally.

Investigation items are not locked in and are exactly that... thoughts and tests. Do NOT go flying off the handle about how this won't work or that won't work until we make an official post. It will severely help your blood pressure.

Hey it's the old extra heat for boating shit idea they had over 6 months ago, let's fuck medium & small laser mechs because we can't figure out that letting people shoot with pinpoint accuracy in a game with massive damage, long range weapons might be an issue.

Bit of a worry that if you are recycling bad ideas, still never mind lets fiddle with machine guns and flamers 

[Pennilenko]

When I started playing at the beginning of all of this, I felt the game was really fun. Unfortunately the developers nerd fucked the fun right out of this game. That and they are really shitty developers. Oh yeah, they are also slow as shit.

[Goreschach]

Don't even care any more. Game is dead.

[Falconeer]

The whole circus of short-sighted changes is really annoying. Dead, not by a longshot. Sorry it's not fun for you anymore but the core is solid.

[Pennilenko]

Hehe, Like you have any type of credible objectivity.

If there is any sort of worthiness left in the core of the design, these developers are certain to continue squandering it.

[Falconeer]

Ok, game is dead.

[Ironwood]

Fuck that.  Game is still free and pew pew.  Hardly dead.

I am, like all the rest of you, amazed at the shortsightedness that comes with being INSIDE the company and in Silos.  It's why I don't do anything stupid, like drop tons of fucking money on games that aren't even out yet.  Pointing no fingers.

[Arthur_Parker]

For the entertainment I had it from it, it was worth it, that's factoring in a major pc upgrade that was needed too.  I played the crap out of this, more than anything else ever, played nearly 5k games just since the last wipe.

[Falconeer]

My "credible objectivity" is taking bets on the popularity factor and financial state of this "dead game" one year from now.

[Ironwood]

I'm not saying this didn't get money from me.  Just not tons of it.  Totally worth it.

 

[Arthur_Parker]

My "credible objectivity" is taking bets on the popularity factor and financial state of this "dead game" one year from now.

It's an uncertain bet now though Falconeer, true they might pull something good out of the hat, but they are just like any other "beta" game out there now.  Months ago while still in real beta they had a brilliant game.

[Falconeer]

I would be ridiculous if I claimed that it will be inevitably successful. Of course I could be very wrong, and very disappointed. But I think I have enough elements to bet it will be (For the record, I didn't spend any money of it after the initial Legendary purchase). As I said I think the core is so solid and so good that no matter how badly they try to screw it usually things get somewhat adjusted within a month or two, so their thirst for money will keep it not-dead for a long time. Seems so me there's still a fuckton of people playing it no matter how many burned out already on the little content available at the moment. Do you really think that leaderboards, achievements, community warfare, a new UI, more maps, more modes, and more 'mechs are gonna kill it? They are not necessarily gonna improve the gameplay, sure we'll see about that, but to call it "dead" is as ridiculous and extreme as calling it "perfect".

[Arthur_Parker]

I wasn't calling it dead, though I have zero desire to play with the way things currently are.   It seems they are going for a more tactical sniper game over a more brawling/skill based game.  Host State Rewind means any idiot can shoot effectively now and lights that used to be paper to the sniper rock, find that snipers are now scissors.

I think you are being optimistic, nothing wrong with doing that, but for me, I see PGI as a good couple of miles from shore, swimming in the wrong direction, when they used to be on the beach getting free cocktails from bikini babes.

[Falconeer]

Not you Arthur. Goreschach and Pennilenko.

[Pennilenko]

My "credible objectivity" is taking bets on the popularity factor and financial state of this "dead game" one year from now.

Im sorry if you read my post as hostile. I was really just trying to poke innocent fun at you.

After all I bought a legendary founders pack like a giant moron.

[calapine]

Well, we the game can't be that bad. We are on post #3003 in this thread!

More seriously. I agree with those (everyone?) that say the game went into the a bad direction lately...sniper boating, consumables, ECM, <insert pet peeve>, but think it's too early to write it off. The most egregious problems are balancing issues and not the game mechanics itself. So while there is no guarantee that PGI gets their act together it's not beyond point of no return yet.

FakeEdit: In the thread announcing the PPC changes that Arthur linked a dev did acknowledge there is an issue:

Let me clarify, this is not a change to "nerf" boating/high alpha builds/"poptarting". It's a change to put the refire rate of the PPC back in line with the rest of the large energy weapons.

[Yegolev]

I MUST POPTART OR MWO IS DEAD

[satael]

Somehow just not having that much fun in the game recently. Just played 2 matches on the ice map (poor visibility so a little less sniping there) with double-gauss muromets and both turned boring (partly due to my choices). In both games I was the only one (on either side) to use the tunnel; in the first game (conquest) I spent the game capping the 2 farthest points and not getting  any shots off (we were losing by points since the enemy held all the others but the game ended with all the enemy mechs down). The second game was assault and seemed to go better (for me) as I made it behind the enemy and started to gauss em down at the ship wreck in the middle only to see them move towards friendly base (I also noticed the game was 6-0 in their favor) and I couldn't catch em before they capped the base 

tl;dr: it's either a sniping fest or a really quick brawl in the middle these days

[Falconeer]

Ask the Devs 38 is out. Too much stuff to make highlights, but I'll just leave here the one that depressed me (my wishful thinking optimism does not prevent me from seeing what is wrong with this team).

DeathofSelf: What is your reason behind the decision to not reintroduce collisions/knockdowns until after launch?

A: Priorities mostly. The same people needed to work on Collisions are needed to work on CW and UI 2.0.

 

[Arthur_Parker]

Pencilboom: What do you guys feel about large weapons boating? And what are you guys planning to do to prevent them from creating game imbalances?
A: This is something that Paul is looking into currently and he will post a CC statement on his findings. We’re also exploring incentivizing players to balance out their BattleMechs better on a personal and team level in addition to small tuning changes.

Forcing people to mix weapons should be a red flag to them but they don't see the problem, it's the age old issue of devs not playing the game enough, they also killed the general forum a while back, which was the main feedback loop.

If it's not just me not liking the current gameplay, that's always difficult to tell, but it's so bad I don't think it is.  The fall in numbers of players should let them know fairly soon that something is wrong.

[Merusk]

I wonder how much of a fall in players there is.  I've certainly noticed hitting "Launch" is taking longer and longer, regardless of how heavy the mech I'm using is.

[Arthur_Parker]

Above is just for the mwomercs.com, maybe we'll get a better idea of player figures if they ever manage to put CW in.