f13.net forums - Latest Devblog... Hey did you know theres lag in Eve?

http://www.eveonline.com/devblog.asp?a=blog&bid=727

Quote

'we are aware of the problem'
reported by CCP Atlas | 2010.02.04 12:03:48 | Comments

Following the release of the Dominion expansion we started to get reports from players about an increase in "fleet lag." Before we go any further, I would like to assure everyone that a taskforce of developers has been working on this continuously, but tracking down the source has proven a very difficult task.
So, what is the problem?

"Lag" is an ambiguous term and can refer to many things. The graphics in the client might be slowing it down, the network might be congested or a host of other issues may be coming into play. We have spent a long time trying to reproduce the issue on the test servers as well as monitoring fights as they have been unfolding on Tranquility, where we have received help from the players in the form of testing and bug reports. You might see names like "CCP Atlas", "CCP GingerDude" and "CCP Warlock" poking about in Local. Don't be alarmed, we are simply monitoring the node, looking for glitches. There are senior software engineers spending their evenings and weekends on Tranquility monitoring and measuring since we have not been able to see issues when testing because the sheer number of people required to trigger the problems and the specific 'usage pattern' of the game systems in question are outside what can currently be tested in the lab (although this is something that is being worked on).

What we are looking at now specifically is the fact that the performance of the sol nodes (the computers in the clusters handling the solar systems) appears to be worse than before. Even though CPU across the cluster has not increased with the Dominion expansion, each reinforced node is no longer able to handle the same number of people fighting as gracefully as it did before.

Additionally, we have the issue of the grid not loading when jumping into a system. Being stuck on the source side and not getting through the gate is one thing since you're not in the battle yet, but if you actually get through the gate and arrive on the other side without managing to load the grid you could be at a disadvantage and possibly vulnerable because of your client's inability to act. This can be detrimental to the fleet jumping into the system.

What is happening is that the client is requesting information about the simulation from the server but it's not receiving it in a timely manner. There may be many minutes before the information is finally received as it is getting stuck in the lower levels of the sol node.
Anatomy of "grid not loading"

When the specific scenario of "grid not loading" occurs, its fingerprint on your client is quite specific (even though sadly we cannot see the effects in the server logs).

Refer to the following handy diagram to determine whether you have encountered the problem:

This is what it would typically look like if you're jumping into a system which exhibits the issue. However, if you are logging into such a system and encounter this issue you will only have a black screen and a "loading" bar.

If we look at the image a bit you will see that the local channel, the right click menu and the system information are all updated to the new system while the nebula is that of the old system. Your overview will also be empty and no brackets in space. If you open up the monitor (control-shift-alt-m) it will have (at least) one outstanding call. This is the request for the simulation instance.

When you see this, the worst thing you can do is to click any buttons or type in local. It will only exacerbate the problem. Relogging will also not help and might even make the situation worse. The best you can do is wait it out (see the "Fleet fight survival guide" below).
Light at the end of the tunnel

We were finally able to reproduce this with a special fleet test on our Singularity test server on Jan 27th where over 400 people participated in helping us out. That was really fantastic to see and I would like to thank everyone who showed up.

On Jan 28th, we had a 1600 person fleet fight on Tranquility which our team monitored closely, keeping the node alive using methods that make our system admins faint. This was one of the biggest, if not the biggest, fleet fight ever in EVE (at least where the node survived the ordeal). This event allowed us to identify what was causing some of these glitches and deploy fixes live.

We have added a stop-gap measure in place which keeps nodes from dying during these high stress scenarios and we have some more permanent solutions in the pipes that will be deployed over the next few weeks.
How can I help?

There is a mass test scheduled for the Singularity test server on Thursday Feb 4th at 16:00 Eve time. Those of you who wish to help out, please come and provide us with client logs when you encounter the "grid not loading" problem or other issues. We will specifically be testing jumping in several solar systems trying out a few of the software solutions that have been implemented.

This requires running Logserver.exe from the client installation folder before you start the client. Your logs are extremely important, and each bug report you submit with this information will go a long way toward bringing us closer to a resolution or fix for this issue.

There is more information on this fleet test in this thread.

Additionally, if you are planning to attack a system we also highly recommend that you notify us at least 24 hours beforehand through the Fleet Fight Notification Form so that the system can be placed on a reinforced node. This will also give us the chance to mobilize people to be in the solar system with measuring devices firmly attached to our Polaris frigates.
Fleet Fight Lag Survival Guide

I'd like to talk a little bit about the best tactics we've found to use during periods of extreme server-side lag and methods to make the experience a little more bearable. Note that I am talking about server-side lag here, not low FPS or bad network conditions (for example, if you find yourself in a fleet fight on your 9600bps cell phone modem up on a glacier somewhere). The lag that this refers to is when there are hundreds of players fighting in a solar system and everyone is experiencing delays, where guns could take minutes to fire and there's some gnarly rubberbanding. If you've encountered this type of lag you typically know the symptoms. It's under these circumstances that the "grid not loading" issue manifests itself for example.
Do not hammer that button!

One of the most important things to keep in mind when you are in a system with heavy lag is that you should not hammer any buttons and you should try to execute as few commands as possible. If you are waiting for the grid to load the worst thing you can do is click buttons on the interface or chat in local. These actions will have a negative impact on your client's ability to recover (more to the point, on your session on the server). If you have an outstanding call you should wait for it to complete, and it will complete eventually. It might take minutes but your gun will fire and clicking the button again will not help. Do not browse the market or look up contracts while waiting for the system to load. I know it's hard but you have to leave the client alone while it waits for the calls to complete or you will add to the problem and can even cause the session to get stuck.

Use the Monitor

The network tab in the ingame monitor debugging tool (control-shift-alt-m) can help you understand whether your calls are completing in a timely manner. If there is a number in the "Outstanding" row that means that the server is busy executing your command. Wait for it to complete before executing another one!
Calls do time out

What I said about calls completing eventually is true... from a certain point of view. The call will be handled eventually by the server but it's possible that the client has given up by then. Calls typically time out in 8 minutes and this is true of the request for the simulation instance (grid). If you have waited for 8 minutes for the grid to load you will not recover. If you have the monitor open and see the Outstanding number drop to 0 and your grid doesn"t load then you have no choice but to relog, and even that might not fix things if the server is still overloaded.

Note that if you get podded while in this state then the client should recover when your session is moved into your clone station.
Final words

We have come a long way in understanding the issues that have plagued large fights after Dominion and have gotten important insight into fleet fight lag as a whole. The problems we are experiencing post-Dominion are being worked on and they will be fixed soon, adding more space carnage to your battles.

I would like to ask that we keep the discussion thread attached to this blog productive and try to keep flaming to a minimum.

Thank you,

Jon Bjarnason
Technical Director
EVE Online, CCP Games

A Word from Customer Support

Support's intervention in large fleet engagements in even a remotely fair manner is very problematic due to the complexity of the situation.

Under circumstances where lag is involved or elusive bugs that are difficult to establish as the deciding factor, CCP cannot justify passing judgment on who should have won or who should have lost. While a problem such as the one in focus here is known to affect some players in certain situations, the evidence available in server-side logs, when available at all, can be haphazard and arbitrary. As such, intervention on CCP's part would risk being arbitrary as well.

What this means is that CCP will not be granting reimbursement for fleet fight losses.

Please understand that fleet fight reimbursement has always been very controversial and few issues have been discussed and argued in more detail within CCP. The conclusion has, however, always been to leave fleet battles alone rather than reimbursing them as a whole. We apologize for any inconvenience this may cause and we appreciate your patience as we work diligently towards resolving this issue.

Bolded the funny parts that leaped out at me.

Yet another bunch of :words:

Quote

keeping eve running smooth and fun to play - how you can help!
reported by CCP Tanis | 2010.02.10 21:39:29 | NEW | Comments

Wow, this has turned into quite a long blog, but it's chock full of much info-goodness which hopefully addresses many of the questions and concerns people have had about public testing and how we approach large-scale performance and feature testing for EVE.

What's in our anti-lag arsenal?

As EVE grows, more features are added, more players are online at once, and the whole system becomes more complex we find that our ability to weed out sources of lag becomes increasingly difficult. Throughout EVE's development we've used various methods to identify and fix sources of lag; server performance monitoring, special task-forces to ‘seek and destroy' lag, monitoring forums and petitions, public test servers, and intensive regression tests of all new code, for example. Though these are great tools, we don't feel that it's enough. Over the years, while creating all of our expansions and the new features and changes that come with them, it became apparent that we had a problem: We had no good way to simulate high-load situations such as fleet fights, factional warfare, Jita, etc. This is a particularly difficult situation for us. In any high-load situation there are always a very wide range of actions being performed simultaneously, with many different hardware configurations on the client machines, and many small factors quickly add up. But these factors are all relatively unpredictable and almost random in practice.

Consider Jita. Think of how many people are there; now think of all the different market transactions, combat, chatting, industry, and other activities that are going on simultaneously in that system - you can see that testing this scenario is not as simple as "load up 1000 clients and let them run." We needed a way to get as many different transactions and machine configurations as possible in order to confidently say that we've gotten a valid test for high-load performance.

Over a year ago, in order to address this issue, we began exploring several different options. We looked at everything, from specially built load-testing clients (bots), to hiring many more testers, to community-supported testing, to developing specialized test suites and scripts, and everything in between. After some prototyping, tests, and wracking our brains, we finally came up with a multi-prong approach that includes community-supported testing (i.e. Mass-Testing - more on this shortly), massive test-server upgrades and changes, load-test clients (still in the early stages of development), special debugging code, and more data logging than you can shake a stick at.

Mass-Testing

The cornerstone of our approach is an initiative, dubbed "Mass-Testing." Simply put, we figured that the best way to "simulate hundreds of players all doing many different actions at the same time" was to, well, get a bunch of real players together and run through proper test scenarios. This is such a vital thing because it is the only way we can accurately gauge how the new code behaves under such extreme conditions as fleet fights with several hundred pilots. This is because of various factors which only become apparent in such highly loaded situation, such as diminishing returns, and code scalability.

At its core, Mass-Testing is a very basic framework which allows us to invite hundreds, or even thousands, of players onto the test server to hammer away at various things. This framework provides us with an excellent venue to present new features or changes to people and not only see how well it performs, but also to get critical feedback about the new changes directly from the players who've taken the time to try them out on the test servers. By running these events regularly, we also gain the ability to track empirically the performance of the client, server, and network layer over time. This all comes together to give us a much clearer picture of where we stand now, where we came from, and where we should focus our efforts moving into the future.

Can't we do this another way, like testing live on TQ?

Unfortunately, testing is never as straight-forward as it seems on the surface. Even in fairly simple programs, the number of possible combinations of inputs, raw data, transactions, and hardware can add up quickly to the point where ‘testing everything' would literally take several human lifetimes. Because EVE is constantly being updated and changed, this adds another level of complexity to the issue overall. How we choose to address this for EVE is by prioritizing areas that need more attention; based on previous history, the complexity of the system or the changes made to it, risk to the cluster/game, player feedback, and several other factors.

When dealing with testing for EVE, we must always keep one thought in the back of our heads: "how will this affect the game, the cluster, and the players?" Sure, we could test directly on TQ, add all sorts of debugging code, join in every fleet fight, etc, but at the end of the day, TQ is for players, not for testing. Adding debugging code would kill the server's performance and make laggy battles much worse. Sending testers into fleet fights could very well lead to cries of favoritism or ‘DEVH4x'. Neither of those results would be desired. We in QA feel, very strongly, that anything that would negatively impact the performance of the live servers or people's enjoyment of EVE should be avoided like the plague.

What good will all of this really do for EVE?

We have been running a pilot of this program for roughly the last year, and it's certainly done its job in helping us identify certain performance issues on the server, client, database backend, and also in our networking layers. This program also provides us with a direct line of communication between CCP and the EVE player base to find how well the expansions perform for you, and also to get some critical feedback about new features and changes in those expansions. This feedback has helped developers tweak and tune their changes in an attempt to make sure that we're providing you with a quality user experience. Some examples of changes that have come about as a result of your feedback are:

Overview tweaks, such as the ability to toggle all brackets on/off easily
Improvements to fleet-finder
Fixes to improve lag in factional warfare
Cleaning up POS code
Fixes for EVE Voice
In addition to the feedback we've been able to gather, we've also isolated and killed several problems well before they ever made it to TQ. Some examples of these resolved are:

Server-side memory leaks
Runaway DB procs
Client FPS instabilities
Client memory leaks
Services being called too often (kills server and client performance)
I am the great and powerful OZ!

Just as it was with Dorothy and the great Oz; an unfortunate fact about Mass-Testing is that most of what's being done is all behind the scenes, which can lead to many folks getting the wrong impression about what's actually going on. But unlike with the great Oz, there is a lot of actual work going on behind the scenes here. We have staff from our Software, Quality Assurance, Operations, and Game Design departments, plus our awesome Bug Hunters, all working in concert to collect data and feedback, analyze the information, and fix what's wrong. I'll give you a brief summary of what we look at during these tests:

DB traces

We collect full traces from our databases. This information tells us what transactions are being made, what procedures are being called and how often, and how long they take to execute. In addition, we also track overall database's load, health, and performance.

Server-side metrics

We collect detailed metrics on the servers' performance, especially detailed statistics on CPU and memory usage for discrete layers of the server code. In addition, we collect data from every piece of the game code during execution to see how well they're playing together, where bottlenecks occur, and if there are any runaway tasklets in our code.

Client-side metrics

We also collect metrics on several developer clients during these tests that collect data on the client's CPU and memory usage as well as things like FPS, response time, desync, etc. This data allows us to pinpoint pieces of the client code that need to be fixed or streamlined in order to improve the client's performance.

Network-layer metrics

In a game like EVE, which use a single server with many thousands of simultaneous users, network traffic, bandwidth, routing, and load-balancing are very important. This is why we also gather full network traces for connected clients and collect data on how well the load balancers are coping with the new expansions.

All of this data is collected and kept from one test to the next so that we can compare, analyze and find performance trends, which in turn helps us identify those areas we need to spend more time cleaning up in order to ensure that EVE runs as smoothly as possible.

Growing up a bit

Over the past year, this program has really just been in its infancy, and we're proud of our baby, but it's time for it to become a toddler and start walking on its own. We need this program to get bigger, better, and provide more value to EVE as a whole. In that spirit, we are moving into the next phase of the Mass-Testing program, where we will be making improvements across the board, but especially in scheduling, backend support, and, later, reporting and information.

Calling all alliances!

In the past, we've had problems getting enough players to show up for these tests, and we will be addressing this issue directly. Ideally, these tests should have at least 400 participants, up to 1500 pilots per test; historically, we've managed to get around 100-200 pilots per test. In an effort to improve numbers, we will do three things. First, we will try to hold these tests on the weekends to avoid work/school/real life conflicts. Second, we have setup a moderated, public mailing list, "Mass Testing Info", which anyone can join, which we will use to send out dates and information about testing. Finally, we will also invite all alliance leaders to bring their members onto the test servers to participate in these tests. The idea being that, by working with alliance leaders, we can hopefully achieve the numbers we're looking for in these tests.

How will this work?

Before each test, we will put up a sign-up thread on the forums with details and an open invitation to any alliance with more than 150 members, as well as sending an in-game EVE-mail to the "Mass Testing Info" mailing list and also direct mails to the leaders of all alliances with 500 members or more. Alliance leaders can then sign-up and indicate roughly how many pilots they are confident that they can bring; we will take the first 1800 pilots, on a first-post, first-serve basis. If we don't get enough pilots signing up, we will cancel that test run.

Important caveats
Players not in alliances can still join in
You just need to show up at the correct time and follow directions
We will accept all alliances who sign up, until we reach a maximum of 1800 pilots
Do not sign up if you're not confident you can bring people!

First of a new breed

You folks have been saying that you want test events that are held at more opportune times for you, and we've listened! These tests do indeed require as many players as we can get, but holding events on weekends or too late into the evenings, on a regular basis, is not as easy to coordinate as one might think. Even game developers and testers have real lives, families, and friends that they enjoy spending time with too. But, we aren't afraid of change, so we've worked out staffing and, from now and into the future, we will be holding test events on weekends, close to peak hours. There will be exceptions to this rule, if critical issues come up that require special testing, but we hope that this new schedule makes participating in these events much easier for you all.

The first event in this new series will be held on Saturday, February 20 at 20:00 GMT

This first test will be a return to our standard "fleet fight and POS siege" format, with some tweaks and slight changes, look for an official test announcement thread in the General Discussion forums in the coming days. We will post full details and goals for the test there.

Information and transparency

One thing that we feel has been a shortcoming in our previous mass-testing exercises thus far is the lack of reports and information being relayed to you, the players of EVE. To remedy this, we will begin publishing the results of each test. Over time, we will add to what is being reported and, hopefully, even get a sub-site on eveonline.com to host the results from all these tests so that anyone can view results from recent tests, as well as historical results.

And I'm spent...

Testing for EVE is an ever-evolving process, one which we strive to constantly improve. One important measure of quality, at least for us, is determining how good of a gaming experience we have delivered to the players of EVE. Though this can be difficult to quantify, or even be a bitter pill to swallow, we still find your input and feedback to be vital to making EVE better.

To that end, I'd like to invite you all to give us your input about the overall quality and performance of EVE. Tell us what areas of EVE you think could do the most good from performance tweaks, what new things you think we could add to give you better control over your user experience, what changes or additions we have made that helped you, etc.

Mass-testing is an ongoing process and is intimately tied into the community of EVE; let's use this as another way to improve EVE together.

CCP asking us to beta test their game for free ITT.