TMA: moving a server cluster

[Stabs]

As the Eve servers are currently offline I thought I'd pick your brains to understand a little more about the reasons why.

What exactly is involved with moving a server cluster and what can go wrong? Are they just saving the data and restoring it to new computers or is it just a physical move of the machines? Why do it, is it a response to the game's steady growth? Will it cure lag?

[TheDreamr]

Wow ... still offline, neat.

Not entirely sure what they're moving as it could be boxes or it could be switching to their DR systems, assuming they exist.  Either option has many interesting ways they could overrun, mostly of the "we made the switch and something small but critical aint working" variety.

There was a devblog recently explaining that the idea was to consolidate all their machines into an aisle / section of the DC so they'd have faster connectivity between the machines in the cluster as well as the latest power and heat management options.

[tgr]

There are tons of things that can go wrong. If they've moved the hardware from the old to the new datacenter, then they can have broken some of the hardware while in transit, they can have misconfigured something which worked fine on the old datacenter but is triggered on the new datacenter, or they can quite simply have connected some wire incorrectly. I expect the last issue to be something they can solve quickly, however.

My interpretation of the move was that it was a physical move, i.e. shut it down on the old centre and haul it to the new, but it doesn't really explain the "during final testing of the database configuration" message on their site right now. They might've gone for the copying data way of doing things, and fucked up something, somewhere. I dunno.

I would've probably either setup a dedicated line between the datacentres (the network latency between the sites is probably not going to matter much if it's in the same city) and moved the nodes piecemeal, but that still leaves room for error.

If I had done this, and I had sufficient funds and an absolute desire to make sure this would go smooth as a baby's bottom, then I'd probably just setup a whole new cluster, have it properly tested, and setup the official SQL server to replicate its data to the new SQL server in real time, so they could verify the new site worked flawlessly and had the updated data. Then, when that's done, set the TTL for the hosts the clients use to 5 minutes, wait for the required DT, change the IP to the new frontend nodes/proxies, and just get to work dismantling the old centre. Although, there's still the case of murphy's law, so things could've gone horribly wrong even if they had done this. vOv

As for why, it might be that the old centre just isn't providing them with sufficient temperature stability or space. I remember the old datacentre we used to live in at the old workplace. If we got a temperature increase of about 2-3 degrees, some of the Netfinity 4500R's would die with a problem on some component that was on the motherboard and distributed power to the CPUs (I usually solved this by either moving the CPU to the other slot, or just rip the disks, disk extension thingy, CPU, memory and network cards to a new chassis with a non-dead motherboard). The new datacentre CCP is moving to probably provides them with exactly this, more stable temperature, and more space. They are going to add Dust514 soon after all, and I believe it's this (along with consolidating the website servers etc? not sure) that's the cause for the migration.

This will not cure lag in any way, shape or form, as I believe there were no hardware changes whatsoever when dominion was deployed, and CCP claims the new code is using less CPU than ever before.

[lac]

There are hardware changes, they are using new switching equipment. They are moving to a dedicated room in the same datacenter.
Here is the devblog:

What makes up the live EVE Cluster and how it's all done is something of a mystery to many who have speculated what it's made of and how it's all connected together.

As you may know the Tranquility (TQ) cluster will be down for maintenance on Wednesday, June 23, 2010 from 0900 to 1500 UTC.

With a migration and a bit of a redesign on the way I thought it was about time to deliver the facts on what it is and will be.

Step one: A cozy new home

TQ has morphed and adjusted over the years as much as EVE Online has. It's gotten to the point were a couple of cabinets simply don't handle it anymore. So, this first step is to move TQ to a bigger place. We'll still be in the same datacenter and connecting to you from multiple networks to ensure the best performance, but this time with a lot more space and power and room to grow.

The new space is a whopping 79kW of power across 12 cabinets. With the larger space and added power, we can now aggregate TQ, Singularity, and the ancillary EVE Services (web, forums, account management, etc.) into a single location in the datacenter. This will provide better network connectivity, fewer intermediary devices and increased capacity.

As with any dense computer solution like our blade servers, heat is always a major concern. Sure, we get great management tools and reduced physical space requirements, but we still have to cool the servers. To do this we've moved from an ambient cooled system (basically the open room temperature is managed but not funneled direct to server intakes) to a completely self-contained, closed aisle cooling system. Cold air from the center of the aisle is force-fed into the cabinets reducing the loss or wasted cool air significantly and helping to focus cold air where it's needed most. This takes the industry standard "hot aisle/cold aisle" designs a step further without having to do anything crazy like running servers under nitrogen pools (although that is pretty cool).




Step two: Networking to 9000

Most of the traffic on the network in TQ is happening between the servers on the internal network. While the routers we use are quite powerful (Cisco 7600's with the RSP720 route processors), our internal switching needed a kick in the pants. With the move we are going to be adding about 800% capacity to our side to side network along with some really nice Cisco Distributed Forwarding Cards (DFC3) to the network blades themselves to help reduce the latency and reduce burden on the supervisor cards that run the switches.



Step three: Pics or it didn't happen

We are going to continue the information sharing about the infrastructure that makes EVE work on the next installment. Although not everyone gets excited about cabinets and a datacenter, there are a few that do. I personally keep them posted on my wall at home. This is meant to be the first of many installments as we continue to improve the infastructure that EVE runs on.

Step four: But, how does this help me get my ship back?

The increase in Layer 2 switching capacity, reducing in latency through Distributed Forwarding, and the extra cold hamsters will have an impact on the ability to reduce overall latency in EVE. It is not a single solution, but a good foundation where core infrastructure can be eliminated as a possible concern.

The next tech installment will have more details on Remapping EVE, the next level of Fleet Fighting and better prediction of hot spots for dedicated nodes.

TQ Tech Details: (Not the whole system, just what runs TQ)

Servers
64 x IBM HS21
2x Dual Core 3.33GHz CPU's
32GB of RAM Each
1x72GB HDD Each

2 x IBM X3850 M2's
2x Six Core 2.66GHz
128GB of RAM
4 x 146GB HDD

Cores
- 280 total Cores
- ~1 THz

RAM
- 2.3TB of Total RAM

Storage
- 4.8TB of Local Storage
- 2TB of SSD SAN
- 256GB of RAM SAN

Network
- Gigabit Ethernet
- 4Gb/s Fiber Channel

So what will June 23 look like?

Here's our current downtime schedule for when TQ will be offline:


0900: All EVE Services go offline. (Web, Forums, Test Servers, EVE Gate, TQ, basically everything hosted in London)

1200: EVE Online web, secure and Test Servers come back online. (all network services reestablished in London. Only TQ should still be down at this time)

1500: TQ back online

[ajax34i]

I don't think they mind talking about technical details (or issues), so we'll probably get a devblog after explaining what went wrong.

[lac]

Updated time for Tranquility return from our round-the-clock-server crew is now around 16:00 GMT, following testing

Servers are looking good so far in testing. Our pale, pod-goo-wrinkled fingers are crossed. Time still stands.

[Phildo]

I completely missed this, and hence am not training anything on my main for the first time since the skill queue was introduced.

[Yoru]

I don't think they mind talking about technical details (or issues), so we'll probably get a devblog after explaining what went wrong.

Looks like it, see bold. Posted on EVE-O:

welcome back to tranquility
reported by CCP Wrangler | 2010.06.24 16:06:43 | NEW

Pilots,

After an unexpectedly long downtime following the physical relocation and updating of the Tranquility server cluster, we welcome you back to the EVE universe. For those of you not following along at www.eveonline.com, Twitter and Facebook, we discovered an issue during the final testing of the database configuration which, once fixed, required a few more layers of testing before we could begin the process of bringing the server back online. This downtime has been unprecedented in many ways and CCP understands its responsibility to address it as such. Once our database team catches some sleep, they will write a dev blog detailing the delay, our decision process for extending the downtime and the steps we took to bring back New Eden as stable as possible. That will be posted in the dev blog section on www.eveonline.com.

In gratitude for your patience, we will give an extra pool of skillpoints to all accounts (paying and trial) that were active at the beginning of this downtime, on one character per account. This skillpoint pool will be appropriately sized for the downtime time frame, universal across all accounts regardless of character attributes/implants and may be applied as each player wants.

This will be done through a new system in the development pipeline, currently scheduled for deployment next Tuesday’s patching opportunity* during regularly scheduled downtime. Since it has been “hot dropped” into the development plans, we will be providing step-by-step instructions for how to use it as soon as possible.

Thank you again for your understanding and patience.

Fly Safe,

The EVE Online Development Team

[Kitsune]

I giggled inside when I saw their 9-hour estimate for reconfiguring and moving their servers.  Whenever a major server job is being done, you need to make your best time estimate, then double it to get the more realistic time.  If you're at all iffy about any step in the process, triple or quadruple it.  It never ever fails that server work takes vastly longer than originally expected.

[ajax34i]

In case you're not reading the devblog posts and the zillion threads talking about it, CCP will give every player a bunch of skillpoints to distribute as you see fit, and a PI-related ship made by Ore.  Both are through the microtransaction system they have, which, in the case of the skillpoints, seems to suggest the possibility that we may be able to buy SP's like we can buy plexes.  I guess we'll see if they allow / implement it (right now there's a lot of flaming on the eve-o forums debating whether this is a good idea or not).

[Reg]

Being able to buy skill points would be a great way for them to continue to get new players into the game. No matter how many times we tell the newbies that the only real difference between them and a veteran player after 6 months is the number of different roles they can do well they still get discouraged when they play with people who have millions of skill points.

If CCP was smart they'd start every new account with 10 million points and let players distribute them however they like.

[ajax34i]

CCP has posted that they are not planning to let people buy SP's (EVE is not switching to microtransactions).  They do have a system in place; they implemented it when they gave everyone that solar-sail shuttle, and it's apparently only going to be used to distribute various gifts similar to that, if any, and not as a full-fledged microtransaction system, apparently.

[Phildo]

If CCP was smart they'd start every new account with 10 million points and let players distribute them however they like.

Welcome to the era where everyone has a custom-made dreadnought alts.

[Stabs]

Well there's already a system in place where you can buy skill points. You can buy plexes with money then use that isk to buy characters on the Eve O forums. Not quite so tailored of course.

[Numtini]

I think the time is probably right for another adjustment on skills and perhaps a bit longer boost than is currently available. It's awful rough if you're a bare newbie. Other alliances aren't as kind as goonswarm about low sp characters.

I'm glad they're not going the microtransaction route. There are games where I think that makes sense, but this isn't one of them. I also don't think it's a good idea to try to change midcourse. A good microtrans system is designed from the ground up, not grafted onto a subscription model.

[lac]

For new players the eve learning path can be a great experience. They grow with what they do, learn the skills that improve whatever path they choose. It's a great way to evolve in an mmo but it's crippling in attracting players who come from other games and want a great pvp experience.

I say lose the spaceship command and learning tree and give every new character the ability to use every t1 mod and fly every ship at skill 1. Let players use the accumulated subscription points to improve in specific areas but not to gain access to ships, caps and t3 perhaps not included.

It would allow new character to be useful in every fleet and small gang possible (they'd still be horrible) but they'd be so much more useful to every alliance and probably have a lot more fun too.

And while you're at it CCP, get some of your people in the newbie corps and have them hand out free t1 ships and mods while providing a good clone for all and a decent CCP FC to fight the other newbie corps in factional warfare on a voluntary basis. No personal losses possible.

Oh and while you're at it fuck cloning. Seriously, make it automagically. Nobody wants a death penalty, everybody wants to go out there and have fun.

[Phildo]

I demand the death penalty for my enemies.

[lac]

That would be included in the microtransaction bit then.

[Phildo]

What, I'd be able to pay for my enemies to lose skill points?

[IainC]

The clone fee is a tax on high SP pilots and an incentive to stop training on one character and start an alt. It costs me 13m a time plus implants to get podded on my main and (mostly) for that reason I intend to stop training that character before she hits the next clone level. 13m isn't a crippling amount for a high SP player but adds up to a non-trivial cost to anyone who runs in high-risk environments and can expect to get podded reasonably often.

Learning skills need to go away, and I wouldn't be horribly opposed to most meta 1 items being requirement free as well. I do think that the steep initial advancement curve is valuable to new players however (once you take learning skills out of the equation). You start off in a shitty frigate and pretty much every skill you can learn increases your character's abilities by a significant margin, you can see real growth and before long you are in a cruiser which is a huge improvement on your shitty frigate. Giving everyone 10m SPs would negate that as you could start with a lot of those stepping stone skills and you'd see only very gradual improvement from then on. Also it's easy to make a tightly tailored alt for pretty much any role with 10m SPs without needing any additional training, even a capital alt is possible in that framework.

[lac]

What, I'd be able to pay for my enemies to lose skill points?

Let me just put that in green for you then.

[lac]

Learning skills need to go away, and I wouldn't be horribly opposed to most meta 1 items being requirement free as well. I do think that the steep initial advancement curve is valuable to new players however (once you take learning skills out of the equation). You start off in a shitty frigate and pretty much every skill you can learn increases your character's abilities by a significant margin, you can see real growth and before long you are in a cruiser which is a huge improvement on your shitty frigate. Giving everyone 10m SPs would negate that as you could start with a lot of those stepping stone skills and you'd see only very gradual improvement from then on. Also it's easy to make a tightly tailored alt for pretty much any role with 10m SPs without needing any additional training, even a capital alt is possible in that framework.

Well I enjoyed all that. The growing into ships thing, building your character to accommodate whatever it is you are doing. I think it's a great, I'm having a good time. The Eve model, as it is, is something that suits me and I don't think gazillion skillpoints to make whatever character you want would be a solution to anything.

But I wasn't talking about me, what got me going was another 'what pvp mmo is good' thread mentioning Eve. Eve is horrible for an accomplished mmo pvp player trying to pvp on a new character. There is no way they can plunge right into the action and have fun unless they buy an old character. (yes, newbie tacklers and all that, much love).

Lowering the barrier for all commonly used ships so new players can field t1 fleet battleships or an interceptor in their first week might be an improvement. Of course they'd be very weak in their role without additional skilling but all roles commonly used in fleets or small gangs should be available to them.

I think Eve would be better off with newbies able to fly everything while they train their subsystems and factional warfare being free (as in beer) pvp organised by CCP or volunteers. Allowing new players, who came for or are interested in pvp, to experience organised fleet fights and gang warfare in empire without consequence until they feel confident enough to go into 0.0.

[Thrawn]

EVE should also add battlegrounds with no death penalty where you can fleet fight, and maybe you can get twice as fast of learning speed if you fly with a friend.  Also if you already have a high SP char you should be able to make a character of a special new race that starts out with 40mil SP.  If you get blown up you can just fly in a covert ops back to your ship with perma-stealth and then you recover your losses with no penalties too!

[Stabs]

This has turned into the terrible design ideas thread. Shouldn't you be posting this rubbish on the CSM boards?

[Phildo]

No it's cool, let's reward people that have been playing the game since 05 with additional All Vs dreadnought pilots and as many SP as my two year old character!

[Setanta]

EVE should also add battlegrounds with no death penalty where you can fleet fight, and maybe you can get twice as fast of learning speed if you fly with a friend.  Also if you already have a high SP char you should be able to make a character of a special new race that starts out with 40mil SP.  If you get blown up you can just fly in a covert ops back to your ship with perma-stealth and then you recover your losses with no penalties too!

You forgot the bit about the ability to flag or unflag your PvP status :D

[Reg]

Heh, giving them 10 million points is probably going overboard. I didn't realize you could do that much with one. My main guy has 80 million points so a 10 million point character didn't seem terribly threatening.

It's probably silly concerning myself about bringing new people into the game anyway. EVE is well past that stage at this point.

[Phildo]

There are a couple of really nasty alts you could make with 10m SP, including incredibly well-skilled stealth bomber pilots.

[IainC]

This plan weighs in at about 9.5m SPs

[ajax34i]

That plan also weighs in at about 10b ISK, or about $200 (if it were a character purchase).  I wouldn't pay the $200, and I don't even have the 10b.

[Numtini]

That's the difference between what you do with X million points if they're handed to you and what you would actually do with them if you were trying to play at the same time. Particularly if you didn't know what you were doing to min max it all.

[lac]

If you want to give new players more swing and avoid these specialised alts, don't make more skillpoints available for character creation but lower the skill requirements of plain t1 modules and ships to zero. Make unnamed t1 the new civilian modules.
That wouldn't alter the current philosophy on character progression but would give new people more options to explore while avoiding specialised alts.

[slog]

No it's cool, let's reward people that have been playing the game since 05 with additional All Vs dreadnought pilots and as many SP as my two year old character!

You know, now that I think about it, one of the main frustrations I had with Eve is that there was nothing I could ever do to compete with the long time players simply because I started later.  Eve isn't for everyone I guess...

[Phildo]

The free SP thing worked out pretty well, actually.  I got Large Projectile III and Minmatar BS II out of it on one character, and shaved a day and a half off another character's long skill.

[Gets]

After yesterday's patch the "Prefer Safer" autopilot option does not work, and will default to taking the shortest route instead.