The end of POS warfare: a CCP draft

[Stephen Zepp]

You also can't ignore things that the client can inspect if they want, but normally don't--such as standings per player (god it would suck if they are delivering ALL the standings of a particular ship in the initial update, but I could see it happening), simulation data on things like bounding boxes, camera limits, all sorts of things that should already be client side, but bad networking strategies might require being sent every ship, per ship.

I assume they don't do that because, A) it would be really stupid B) when you inspect someone it always has a delay. It feels like the client is waiting for data from the server. But if they somehow manage to indeed get 8 megs sent during the initial loading, then they must be doing something really weird.

Overview and local all have to have standings data...so it is dependent on if they pre-calculate relative standing and transmit that as part of the initial update, or just send all the data and let the client determine. Smart way would be the former, but the "easy way out" would be to send all the data, since the client could be asking for it on a user inspect.

Honestly we're just grasping at imagination--but like everyone agrees, 8 meg for a high load grid is ridiculous, and especially that it dominates client processing for so long.

One more thing to keep in mind however is disk loading of assets--they almost certainly force the client to load the ship models and textures on the initial update, and by definition a blob will have a huge amount of ship types, so that's a lot of blocking access to the disk. Smart move here would be to have a background thread asset loader, so they can at least continue input and rendering while the resources are loaded from disk.

[Endie]

This certainly gels with what happens in all but the 1000+ person fleet battles: you sit around with a blank screen for several minutes then bang the grid loads virtually instantly and the lag is then a fairly constant module lag issue of up to 20 minutes.  More on at least one occasion in the Feythabolis campaign.

The really big battles are a complete lottery.

[MahrinSkel]

I've been to a lot of big battles in a Covert Ops.  More than once, I've been unable to steer away from something, been completely uncloaked in the middle of the enemy fleet, stayed that way for 5-10 minutes, and not been killed simply because by the time anyone saw I was there, my recloaking or warp-out had kicked in and I was safe.  The mass battles are simply too damned hard on the *server*.  A battle is an N^2 problem, when N = 200+, the server just can't handle it.  Theoretically, if they could dynamically rebalance and transfer systems between server nodes, they could keep a handful of baby supers on hand to deal with big battles and the limiting factor would become client framerates and network throughput.

--Dave

[Krakrok]

Are they still running only dual-processor 64-bit AMD Opteron-based IBM BladeCenter LS20 blade servers? I would think they would have upgraded to 4 or 8 cores by now. You can get 8 core 18Ghz machines for dirt cheap. I would think their coding would be multiprocessor compliant if it works anything like web servers do.

[Quinton]

Are they still running only dual-processor 64-bit AMD Opteron-based IBM BladeCenter LS20 blade servers? I would think they would have upgraded to 4 or 8 cores by now. You can get 8 core 18Ghz machines for dirt cheap. I would think their coding would be multiprocessor compliant if it works anything like web servers do.

Multicore doesn't help you much if you're using an scripting language that has a big 'ol lock around the interpreter, which has been a big limitation of python forever.  I don't believe stackless fixes that issue.  If they have to have everything in a node in the same process (would not be surprising) there's no benefit to be had from more cores except possibly density (running more server processes per box).

[MahrinSkel]

Are they still running only dual-processor 64-bit AMD Opteron-based IBM BladeCenter LS20 blade servers? I would think they would have upgraded to 4 or 8 cores by now. You can get 8 core 18Ghz machines for dirt cheap. I would think their coding would be multiprocessor compliant if it works anything like web servers do.

They went through a couple of upgrade cycles to stuff like that.  I believe Jita runs on a dedicated 8-core blade running as an SSI.  Sometimes if you give them advance warning of a battle, they can give it a dedicated high-end node the DT before and things improve.  But that just raises a workable N to about 600.

--Dave

[lac]

Wait, doesn't stackless python mean they only can run one node on every core? It used to be like that, don't know if changed.

[Quinton]

Wait, doesn't stackless python mean they only can run one node on every core? It used to be like that, don't know if changed.

I'm less familiar with stackless, but with regular Python, the interpreter itself is not threadsafe/reentrant so there is basically one big lock around it.  Thus, in a multithreaded python program, only one thread can actually be running python instructions at a time.   This means that there is little advantage to running a multithreaded Python program in an SMP environment, unless it's doing heavy native computation without the lock held (typically pretty rare).

My assumption was each node is a single python environment, which would at best use a single core even if mulithreaded.  Presumably it *is* multithreaded -- otherwise I'd fail to see the advantage of stackless which brings inexpensive microthreads (aka greenthreads).  Of course cooperative, non-kernel-level multithreading is an enormous nightmare if you want to actually take advantage of multicore systems.  Oops.

This is pretty much the biggest advantage java has over most of the other popular scripting languages -- it actually *can* take advantage of multiple cores.  Sadly it's got tons of other issues... don't get me started ^^.  I'd love to see a lightweight scripting environment that is actually multithread/smp friendly.

[Morat20]

My assumption was each node is a single python environment, which would at best use a single core even if mulithreaded.  Presumably it *is* multithreaded -- otherwise I'd fail to see the advantage of stackless which brings inexpensive microthreads (aka greenthreads).  Of course cooperative, non-kernel-level multithreading is an enormous nightmare if you want to actually take advantage of multicore systems.  Oops.

I understand that's their current problem. If I understand their development thinking, they decided zoning was a huge exploit area and designated each solar system as a single zone, no exceptions. No breaking up zones. So "zoning" is a full wipe and load as you jump, which means it's hard to play zone-boundary games -- you can't wander back and forth like you can in, say, EQ. (Remember, this was all designed in like 2000 or 2001).

So they decided, based on projections of the universe size, number of players, and the sorts of numbers a "busy system" should encounter, that this would be easiest if each solar system was a single, unique thread. So a processor could run one busy system, or several empty ones. But you couldn't split a solar-system over multiple servers, as this would require some sort of zoning.

They used stackless python, which was perfect for what they wanted -- each system a seperate and totally independent thread, and they could microthread the crap out of each system. It worked really well -- until people wanted to start 300v300 fights, or pack 1200 people into Jita.

Their current problem is to fix it requires either zoning busy systems (which either means some serious rewriting of code, and the probably introduction of lots of godawful bugs -- especially duping bugs, which will fuck them hard if one gets out of hand), or rewriting stackless's interpreter to handle multiple processors (also another bitch job). Short term, they've been altering gameplay to try to (1) cut down on server calculations -- things like drone reduction and now the garbage disposal they're planning, (2) Thrown hardware at it, and (3) Altered gameplay to try to cut down on the blobs and get people the fuck OUT of Jita.

[JoeTF]

There is one thing that was keeping me thinking recently:
Would it be possible to implement "Bullet Time" for entire solarystem?
IF load gets twice as big as CPU power available to given solar system, make everything (reloading, acceleration, itp) in the system happen take twice as much time and slap a nice graphic effect on it. Problem with lag isn't with battles being slow, but with everything fucking up - waiting 30 minutes to load anything (and more frequently not loading at all), messing up of command execution queue and having to wait 15 minutes for your commands to be acknowledged. Gun firing once a minute instead of 6 times a minute would be much better than waiting for 15 minutes for them to start firing - extra time and no command lag would also make focus fire easier so people would die much, much faster, possibly balancing out slowed down game time.

Other than that, they should fix their design - people blob because blobbing give them advantage - both in firepower (having more guns makes winning easier) and strategic value (ie. being able to take down cynojammer in 4 minutes, faster than the enemy can form up and reach the system). There should be a hardcoded limit to how much dps POS Tower can take/heal and every other POS thingy should be balanced from it.
Some penalties for blobbing (in one grid, game should encourage attacking in several places at once) or crowd control weapons other than doomsdays would be nice.

Oh and while we're at fixing the game, they should make constellation chat behave like local and ditch the local althogether. With tools like BEACON, removing local would actually help carebears, probably more than it help gankers.

[ajax34i]

Would it be possible to implement "Bullet Time" for entire solarystem?

I doubt it, for the simple fact that the client is single threaded, and if your chat gets lagged the client will stop the graphics and wait for it.  I.E. it cannot play a "nice graphic effect" while it waits for data, it simply freezes as it waits for data.

[JoeTF]

Would it be possible to implement "Bullet Time" for entire solarystem?

I doubt it, for the simple fact that the client is single threaded, and if your chat gets lagged the client will stop the graphics and wait for it.  I.E. it cannot play a "nice graphic effect" while it waits for data, it simply freezes as it waits for data.

Uhm, nope? Client is microthreaded and uses prediction quite heavily.

[Endie]

Joe, your idea about a max dps that can affect a tower, either inflicted or healed, is brilliant. If there is simply no point in bringing more than (say) 20,000 dps plus reserves, then there is a real point in multi-pronged attacks.

[lac]

I suppose you would have to make the max_healing equal to the max_damage (what about hardeners, they lower max_damage so they should also lower max_healing so they would become useless) and limit the number of guns on a pos.
This means the attackers would still blob when there are carriers repping a pos and be at liberty to attack many more posses at once. So I think this would lead to multiple posses getting reinforced but the same blobs we have now when posses come out of reinforced. 

[JoeTF]

Right now very important part of blobbing is ability to finish the damn POS/module faster. After the change you wouldn't need everyone you can possibly get online, but merely enough people to kick other team ass. It's a huge difference. You're right - it won't fix blobbing completely, but it will limit it, and at least make alternative strategies worth considering (after all, the most optimal way of reinforcing would be going after several POSes at once and most optimal mode of defense would be simultaneously repping multiple POSes; in worst case it means that after every large attack at least one POS will change hands, and all other cases look...interesting;-)

Max healing would have to be equal to max dmg, otherwise we would create quite lame timelimit penalty for one of the sides. After all both sides have to be outside of shields to do their thing, so eventually the better one will get their way:)