CCP Fixing lag - system player limits implemented!

[bhodi]

During the downtime today, Sunday 21st September, we deployed a new feature to Tranquility. The Stargate will now check how many players there are in the system and deny access if you attempt to jump into it when too many players are present. In addition to this you will not be able to login on a character if the system has too many players. We have also increased the amount of GMs this evening and if you are unable to login, please press the ESC button and submit a petition in the Stuck category.

This is a temporary solution to this issue, we are working on a permanent feature which has the highest priority and will be implemented as soon as possible.

Yeah... we'll see.

No word on the actual system limit.

[NiX]

If they actually say what the limit is, that would be horribly stupid. All it takes is a bunch of bored people to clog up a vital jump gate.

[Der Helm]

Jita had 600 people online, right after downtime.

[Jayce]

Sooner or later they will figure out the number, then warfare will devolve into:

1 - get enough people to lock the system.  If < 5% are enemy, that's ok too.
2 - reinforce their POSes
3 - do not logoff for the stront timing
4 - profit.

I really don't see how this is not a ridiculously stupid idea.

[eldaec]

Jita had 600 people online, right after downtime.

My guess is that system limits will vary, and they could presumably exclude docked pilots from the count. Though 'you cannot undock because the system is full' wouldn't be much fun.


In the short term they could also exclude high sec from the limit, since I don't think anyone ever had 5 minute module lag in high sec.

[Sparky]

Sooner or later they will figure out the number, then warfare will devolve into:

1 - get enough people to lock the system.  If < 5% are enemy, that's ok too.
2 - reinforce their POSes
3 - do not logoff for the stront timing
4 - profit.

I really don't see how this is not a ridiculously stupid idea.

Even if you don't have enough pilots with all their alts, make trial accounts!  This will be horribly, horribly abused.

[Vedi]

There are actually lots of way of getting into system apart from gates or login. From the top of mye head:

• Jumping in (capital or black-ops) by cyno.
• Jumpclones.
• Normal clones (when you die you reappear in a station where your clone is registered).
• Gating in via a Titan.
• Jump bridges.

If this is a temporary thing, it wouldn't surprise me if they missed a couple of ways to circumvent the measure. Will be interesting to see what that devolves into, strategy wise.

For instance, say you got a system with only one stargate in it (dead-end). If you can get a sizable enemy fleet in there that is confident that they occupy majority of the slots, you can then move a large fleet into the neigbouring system, filling that up and trapping the enemy fleet. You could then jump-clone in a few hundred into the system where the enemy fleet is trapped, undock and engage. Or just go ahead and reinforce every POS in every enemy system.

[Endie]

This will make taking space even harder.  Want to be sure of a system defence?  Get 2/3 or 3/4 of the limit in (even if it varies by cluster people will quickly work out likely values by experimentation) and let your enemy trickle in, hutting each on the head then waiting for the replacement to be allowed in.  And for anyone who has flew in F-T or the like, you'll know that you can pop caps all day long like that.

[eldaec]

System capping will work just as well for the attacker as the defender.

If for instance you were attacking a Vanguard system, you'd be able to get in hours before, safespot up and lock defenders out whilst they hide in a station 58 jumps away.

What it will do is swap the lag battles for long term wars of attitrition with alliances logging out huge numbers of ships in a system days earlier, and then continually attempting to log in till the population shifts two to one in your favour.

Basically, much more theory-craft-faggotry driving strategy.

[Jayce]

If this had been in place when we took FAT and 25S, we'd probably still be there.  1000 strong alliance, logged in, in station.

[JoeTF]

Cap is supposed to be around 1k or something, so it shoudn't be an issue.

However, in course if implementing this CCP screwed up the servers ROYALLY. Killmails are not generating (not new, but yeah), people are getting double payouts from insurance, people who died ending up in brand new ship after relogging, people who got jammed, tackled by three guys and attacked by 15 fighters (hi! ) getting 1mln km form the gate in brand new ship after relogging, people getting new ship but only with half of the fittings...

I play this game for quite a few years, but I haven't seen mess like this even on the test server.

[Miasma]

Is this because of the EvE-O thread you guys linked about the factional warfare having the same problems as 0.0?

[Pax]

The next thing we know is, CCP releases "Premium Tickets" with "Preferred System Access" to enter capped out clusters. At a nominal fee, of course.

[Drugstore Space Cowboy]

The next thing we know is, CCP releases "Premium Tickets" with "Preferred System Access" to enter capped out clusters. At a nominal fee, of course.

Join the New World Order!

[Pax]

The next thing we know is, CCP releases "Premium Tickets" with "Preferred System Access" to enter capped out clusters. At a nominal fee, of course.

Join the New World Order!

It's not a conspiracy, it has already happened in Silkroad Online - they capped servers, figured paying customers could no longer log into the game, so they released Premium Tickets in different flavours (and with different price tags). The terms in ".." aren't just possible catchphrases, but actual item descriptions from SRO.
True story.

[TheDreamr]

Honestly, it's just Jita and this time we mean it!

Server hotfix applied Monday 22nd September regarding system player limit.
reported by: CCP Navigator | 2008.09.22 12:17:07

During downtime today, Monday 22nd September, a hotfix was deployed to Tranquility. The system cap applied to the server yesterday has now been removed from all systems with the exception of Jita. From today, Jita will still maintain a maximum player capacity which has been raised significantly.

[Mirochai]

Get 1000 farmers to log in to Jita when you don't want people trading. Then do it for any other system when it shifts.

[bosoa]

Sooner or later they will figure out the number, then warfare will devolve into:

1 - get enough people to lock the system.  If < 5% are enemy, that's ok too.
2 - reinforce their POSes
3 - do not logoff for the stront timing
4 - profit.

I really don't see how this is not a ridiculously stupid idea.

i would be willing to bet real cash that CCP didnt even think about this. give it a few months (if it can be done) and after all the petitions come in some proggie will get the idea and then the hamsters will get back on their wheel so he can undo the crap he did >.>

[lac]

Devblog - system avoidance and jita cleanup.

A Tale of Two Cities

Iceland's capital of Reykjavík is an interesting experience for me. It's big, it's busy and, as a result of being a convenient place to get most anything one needs, it can be a bit expensive; much like any other major metropolis that has over a hundred thousand inhabitants. It's a place with a little too much hustle and bustle for me.

Often times though, I do have to travel there. For example, if I find myself needing a new bicycle, I know that the stores in Hafnarfjörður simply don't have the same stock as the ones in Reykjavík. Therefore I hop on the bus and go through the pain of entangling myself in the fast-paced lifestyle and traffic congestion of the capital (we do have some bus priority lanes, which is cool).

I can't really complain since it's my own decision and I'm going back to Hafnarfjörður with a spanking new, albeit overpriced due to run amok inflation, bike. *ring ring*

What really grinds my gears to a halt is that I can't go visit my sister in Mosfellsbær without first going through the traffic-congested Reykjavík. If only there was a way to go around Reykjavík...

Where was I? Oh yes.

We have a similar problem in a little space game called EVE Online. The Jita system is a lot like Reykjavík. The biggest playability issue with Jita (if you don't count the lag and spam) is that people sometimes have no choice but to go there for various reasons:

* When navigating through this region of space the autopilot sometimes wants to send you through Jita.
* There is a constellation you cannot reach without going through Jita.
* There are some agents in Jita that new players sometimes have the misfortune of working for.
* Agents sometimes send you on missions into Jita.

We are (finally, I hear you scream) fixing all of these issues.


Autopilot Avoidance

We have a new feature in place that allows players to set a list of systems, constellations and regions that the autopilot should never send them to. This list is preconfigured to avoid the Jita system.

This works the same as the current "avoid pod kill" feature; the autopilot will act as if the system isn't even there and will rather return no path at all than a path that takes you through the undesirable systems.

What this means is that you can happily navigate all around the Forge region on autopilot without having to worry about getting stuck in the Jita blackhole for the better part of a Sunday evening.


Figure 1: Autopilot avoidance turned off
http://staff.ccpgames.com/codemonkey/avoidance6.png

Figure 2: Autopilot avoidance turned on, with Jita on the avoidance list
http://staff.ccpgames.com/codemonkey/avoidance5.png

Figure 3: A new map mode showing your avoidance systems
http://staff.ccpgames.com/codemonkey/avoidance4.png

This is a completely general system to avoid any number of star systems, constellations or regions. You will therefore be able to use this to avoid that annoying faction that keeps shooting you or that constellation that has those pirates in it.


World Shaping

There is a constellation called Ruomo which is famous for being a place where you cannot go unless you go through Jita. Citizens of the constellation have finally rebelled and gotten themselves a spanking new hyperspace bypass, courtesy of the Interstellar Bottleneck Bypass Initiative (IBBI for short).

Soon you will never have to go through Jita to reach any other system.


Agents & Missions

Agents in Jita have finally had enough and are hauling butt out of there. Jita will be completely agent-less soon.

Also, agents outside Jita will never send you into Jita to complete a mission for them.


When will we get it?

This system will be implemented incrementally but you should see this start to trickle in soon. Not the CCP soon(tm) mind you, but very very soon.


Conclusion

This is being implemented at the same time as we're aggressively optimizing our software backbone to provide a smoother playing experience in Jita. That is, however, outside the scope of this devblog.

Our goal is to create an environment where players don't have to go to Jita unless they want to. This is very important to the overall player experience.

Together, these two avenues of attacking the problem should provide the EVE denizens with a much smoother and more consistent playing experience. It's not a silver bullet, but every little bit helps.

Now, if only the Reykjavík Transit Authority would do the same...

By the way, "A Tale of Two Cities" refers to Reykjavík and Jita. Clever, ain't it? Jita isn't even a city... Hilarious!
- Nonni

[Simond]

Conclusion: People will find a new hub.

[Pax]

This... and united. will need a new spot.
The second thing after setting IBBI to avoid Jita will be setting IBBI to avoid Rancer.

[Thrawn]

Conclusion: People will find a new hub.

Why?   

[Moosehands]

Conclusion: People will find a new hub.

Why?   

Because people are social, because the convenience of a centralized market translates to profit on the part of merchants, and because the contract system blows goats and the best way to get stuff sold through it is to find a huge group of players and spam local.

They removed all belts in Jita, the population continued to climb.  Prior to that, they rearranged the whole universe.  People just moved from Yulai to Jita.

Every decently populated MMO is always going to have at least one completely player driven gathering spot.  Trying to find ways to reduce the reasons people will want to go there is counterproductive.

[Thrawn]

From what I understood most of what they are doing is just trying to keep people that have no reason to go to Jita from having to pass through it, not trying to force people out of it.  The only change that seems to be trying to keep people from actually using Jita is the removal of agents, but I can't imagine any sane person would try to run missions out of Jita to begin with.  I don't see why this would make people look for a new trade hub.

[Jayce]

I think this has a chance of working, certainly better than arbitrarily capping the number of people.

The agent system has that ridiculous penalty for turning down a mission, so people are torn between contributing to the Jita problem when t they get a mission there, or losing standing.  IMO the mechanic is sort of busted, but that's another topic.

What they are doing is trying to make it a trade hub ONLY, no other activities, even passing through, encouraged.  I think that will help some.  Next they should consider some incentives to move to Amarr or Oursalaert or Rens.

[eldaec]

Conclusion: People will find a new hub.

Why?   

Because people are social, because the convenience of a centralized market translates to profit on the part of merchants, and because the contract system blows goats and the best way to get stuff sold through it is to find a huge group of players and spam local.

No, he means why would this mean they have to find a new hub that isn't Jita?

Nothing in this post makes Jita a less desirable place to trade.

Quite the reverse.

[Endie]

Indeed.  This will remove a few score people who didn't need to be in Jita, and weren't trading.  It'll improve lag a little.   Making Jita a nicer place to trade,  Marginally.

[MahrinSkel]

Yeah, they already tried the "break the hub" method, in Yulai.  It's why everything in Genesis region is so many more jumps from everything than normal, they cut so many of the links to and inside that region to get people out of Yulai.  For a while it pushed people to Oursalaert, but eventually things stablized on the Rens/Amarr/Jita triangle, with Jita by far the highest volume. I used to go to Ours and Yulai for cheap T2, people who set up their production lines there early and didn't move away when the markets did would often price war each other to 20% discounts off the Jita price fighting over a trivial local market base, and I'd swoop in every few weeks and buy the market out.

--Dave

[lac]

More lag fixing dev blog goodness.

StacklessIO

For the past two years we have been developing new technology, called StacklessIO, to increase the performance of our network communication infrastructure in EVE. This new network layer reduces network latency and improves performance in high-volume situations, e.g., in fleet-fights and market hubs such as Jita.

On 16 September we successfully deployed StacklessIO to Tranquility. We noticed an astounding, yet expected, measurable difference.

Normally Jita reaches a maximum of about 800-900 pilots on Sundays. On the Friday following the deployment of StacklessIO there were close to 1,000 concurrent pilots in Jita and on Saturday the maximum number reached 1,400. This is more than have ever been in Jita at the same time. Jita could become rather unresponsive at 800-900 pilots but on Sunday it was quite playable and very responsive with 800 pilots. It should continue to be snappier and more responsive in the future.

http://staff.ccpgames.com/explorer/devblogs/images/playerpopinjita.png

The Measurements

This spring we saw the fruits of our R&D work when we deployed StacklessIO to Singularity and began measuring the difference.

Confirming suspicion we had had for a long time the Core Server Group team, lead by CCP porkbelly, proved that StacklessIO vastly outperformed the old network technology. They also demonstrated that the old technology could sometimes, under extreme lab conditions, delay network packets in an arbitrary manner for a significant amount of time.

Later CCP Atlas of the EVE Software Group showed that those symptoms also happened in wild with the old technology; although on a smaller scale, then network response to client requests could in some cases be delayed for a few minutes on highly loaded nodes in the cluster. In particular we measured client network communication to the node that hosts Jita.

Since the client and server clocks are synchronised then we called a remote service on the server from the client, the server responded with the global time and we measured the server and client deltas. We also called that same service directly on the server node to measure the service call's processing time, which turned out to be negligible.

What we discovered in our tests is that the server delta was almost identical to the client's received delta so the delay was due to the remote service call taking a long time to reach the server-side service, most likely somewhere in the network layer on the server. The values on the graphs below are seconds.

http://staff.ccpgames.com/explorer/devblogs/images/delta_jetbyte.png

This is a Sunday profile and is very specific to Jita. This was one of the primary reasons why Jita could sometimes become fairly unresponsive on Sunday evenings. It was not uncommon that client requests could take up to 1-2 minutes to reach the service layer on the server, and the requests would be delayed seemingly randomly since for two requests in succession then the first one could be delayed for minutes while the second one would get a response almost immediately. From a player's perspective this would manifest itself in lag and strange client behaviour as requests were delayed and completed by the server much out-of-order.

By comparison, here is Jita with approximately the same number of players, around 800 pilots in local, after the deployment of StacklessIO.

http://staff.ccpgames.com/explorer/devblogs/images/delta_stacklessIO.png

It's very apparent that StacklessIO does not demonstrate any of the earlier issues. There is only one small spike and two small bumps but we must keep in mind that such isolated occurrences could be caused by general network issues on the internet. Since the client/server network communication has to travel through the internet then some delays would be expected depending on general internet health and the particular ISP.

There are no systemic issues anymore as with the old network technology and StacklessIO provides all-around superior performance.

One of the other measurements we did was to ping all nodes in the cluster from a single node to measure network latency within the server cluster. The values in the tables below are seconds.

Ping Pre-StacklessIO
Time Minimum Maximum Average Stddev
16:00 0.00065 3.22 0.042 0.032
21:00 0.00064 4.36 0.068 0.056
22:00 0.00065 1.21 0.027 0.027
23:00 0.00064 4.36 0.027 0.028
00:00 0.00065 1.01 0.020 0.017

Ping StacklessIO
Time Minimum Maximum Average Stddev
16:00 0.00064 2.00 0.014 0.021
21:00 0.00064 1.02 0.014 0.018
22:00 0.00064 0.25 0.009 0.011
23:00 0.00064 1.93 0.014 0.021
00:00 0.00064 1.06 0.010 0.014

From the table we notice that the minimum values are the same before and after. The lowest maximum is approximately the same but overall the maximum values are lower with StacklessIO by approximately a factor of 2 and they are more consistent.

The average values are lower overall with StacklessIO by a factor of 3 and the standard deviation is lower by a factor of 2. Below is a visual representation of the average values.

http://staff.ccpgames.com/explorer/devblogs/images/ping_avg.png

At 1,400 pilots on Saturday the node hosting Jita ran out of memory and crashed. As crazy as it may sound this was very exciting since we had not been in the position before to be able to have that problem. We immediately turned our attention to solving that challenge and are making significant progress. I will provide information on that specific effort in a dev blog later.

But we have already made good progress on memory optimisation as a part of the StacklessIO technology effort, e.g., memory usage on the proxy servers in the cluster reduced significantly.

http://staff.ccpgames.com/explorer/devblogs/images/proxy.png

The two tall peaks are memory issues we encountered in the first days after deploying StacklessIO. A task force was put into action and it reduced the memory usage by 50% compared to pre-StacklessIO values.

The graphs and measurements above show primarily statistics for Jita but the benefits of StacklessIO apply everywhere. We measured Jita in particular because we could rely on activity and regular load in Jita for measurements. StacklessIO should have a positive impact on your playing experience, no matter where you are in the EVE universe and no matter what you are doing.

[Amarr HM]

Actually CCP seem to have done it right this time, can't see anything but positive from these changes. Though if they do decide to move the system cap limit elsewhere then it might be exploitable. The AP feature is interesting, Jita was always easily avoidable if you setup waypoints but there's lots of dumbass or lazy pilots out there so I suppose it's circumventing them more than Jita 

CRITICAL

    * A new autopilot feature has been added: Avoid System. This will allow you to set systems, constellations or even entire regions to be ignored by your autopilot. By default, all autopilots will avoid Jita unless you configure it otherwise.
    * Dynamic System Cap: We have implemented a system that allows us to set a maximum number of players that are allowed into a given solar system. At this time only Jita has this cap set. The cap can be changed during runtime and we will be adjusting it in the coming days as we make software improvements and add more hardware to Jita.
    * An Auto-Move feature has been added to the login process. If you are trying to log into a system that currently has more people in it than the dynamic cap (see above), you will be given the option of moving yourself to a neighbouring system.

FEATURES

World Shaping

    * 3 new jumps have been added around Jita:

    * Muvolailen - Maurasi
    * Maurasi - Perimeter
    * Veisto - Sarekuwa

[Predator Irl]

This sounds very promising, the idea of having no mission runners in Jita long overdue. Bypassing Jita completely, well I think this will take a decent amount of traffic out of the way also. Who knows, Jita of the future could be lag-free. I've already noticed improvements in the past few weeks.

[Amarr HM]

They really took their time but this is a welcome change to everyone I'm sure.

[Moosehands]

From what I understood most of what they are doing is just trying to keep people that have no reason to go to Jita from having to pass through it, not trying to force people out of it.  The only change that seems to be trying to keep people from actually using Jita is the removal of agents, but I can't imagine any sane person would try to run missions out of Jita to begin with.  I don't see why this would make people look for a new trade hub.

Looks like I completely misunderstood your point then.  Sorry about that.

[bhodi]

You know what else broke today?

Outposts (stations) no longer show up in the right click menu.

That's pretty rad.

[Endie]

We've got a pile of bugs: people moving while cloaked in battleships without cloaks fitted.  People in fleet appearing as members of NPC corps.  Drones travelling into pos shields.  And so on...