f13.net

I try to fire steam up for the hell of it to play some Sven Coop or Specialists, maybe even play some HL2.

Steam is dead thanks to the power outage in Seattle.

Way to  have backup servers in more than one site Valve. Jesus.

But wait it gets better. You can't play single player mode for any of the games either cause you have to be able to connect to Steam to set the game into offline mode in the first place.

Is it wrong of me to laugh?  Because I am.

Remember, kids, digital distro with online verification is 'teh future'.  It'll only get worse from here.

This is mostly a result of braindead server decisions. How do you only have one site when you're making THAT much money?

Yeah this is just a result of someone being incredibly stupid. Considering Steam is a global system they should have at the very least a data center in PST, EST and GMT for DRM and availability, hell leasing it wouldn't be that expensive. Boneheaded.

You'd be surprised.

But it would cost something.  Weighed against the actual benifits it provides the COMPANY (not the users) someone made that call.   Considering the only alternative for the end user is boycotting the system in protest, and then never again playing  games they paid for, seems like a good business call to me.

Fucking Morons.

I don't care how you slice it, this is real stupidity.  It has nowt to do with business.

Living on the west coast myself, I know the reason it went down was due to severe 65+ mph wind storms that knocked out power to nearly everywhere.

To an extent, I can say that outages can happen with any online service, but Steam is a interesting exception.  Steam has a unique problem in that its an authentication service of sorts - you can't play an offline game you purchased unless Steam is up.

Last I heard, the technocracy is planning on having all of our software run from online services as the final word in piracy.  Kinda makes one worried for the future of computing when something like this happens.  Are redundant arrays really going to be enough?

I don't think decision making will change much, after all good decisions cost more to make and implement. Its all about milking consumers while spending absolute minimum.

Hiring a warehouse and some hardware in another geographic location should be a no-brainer with a minimal cost.

Outsource it to India for christ sake.

Data Centres != Warehouses.

But yes, it's not that expensive.

And probably one of those things noone got around to.

They could have at least gone with a better datacentre that has its own power generators and multiple connections to the internet.  Back when that massive blackout took out a chunk of the east coast our hardware didn't suffer so much as one second of downtime.

The point I'm trying to make is that there are many many small and medium sized businesses with a lot less money to toss around and much less important data, which also keep redundant servers in multiple sites.

And now I have to reset my password for some reason and I have garabage in the place where my secret question is...and I have no idea how to answer it. Looks like I'm emailing support to get my passsword reset manually then...

Your secret question is: What is candy?

Oh, that's right. I remember. Delicious. Candy is delicious.

Frankly, I hope their power goes out again and again until they let contacting servers be optional. I hate having to be online to play an offline game.  I bought it you cockmunchers,  I have a serial, let me god damned play the game.

There will be a second-tier of service providers whose only mission in business life is to make sure absolute, infallible redundancy is assured. Not using such services will end up being suicidal.

Or people could, y know NOT SUPPORT STEAM AND THEIR DRM CRAP.

The problem is that you can't really keep servers in a warehouse.  Not if you want them to work properly, at least.  You need raised-floor space.  That costs a bit.

You also need network connectivity.  The interesting thing about India is that it has an incredibly pathetic infrastructure.  Third-world, you might say.  Things are getting better, but if you want a decent response from servers in India, you are going to have to lease a line from here to there.  Cha-ching.  Nevermind poweroutages and whatnot at the remote site.  Our office in Punai isn't even able to be connected by cable to the net, we jump to satellite in Singapore and back down to Punai.  That's a tad slow.  It also costs a bit.

I don't know what the recovery time would be for Steam, but the faster you want the remote site to take over, the more it costs.  If you want the remote site to have all of the data that the primary site does, you can go for either speed or economy.  I seem to recall that a good connection from Atlanta to Cincinnati would cost $100k per month.  No, I don't remember what the bandwidth is, I wasn't taking notes.  Not my project.  It costs a huge amount more if you go past a certain distance; I want to say 1000 miles.

You will also need some semicompetent wage-slaves on the other end, or an outsource corp you trust.  $$$

No, really, if a company that rakes in over a billion a year doesn't want to spring for a remote-recovery site, a game company sure as hell won't.  They don't stop making money while Steam is down, not really.  When our site is offline, we literally stop producing product.  Worldwide.

But we're not talking about anything special here; just a user-authenticating server. I think we could all live with a day of downtime or so if it were only about us being unable to download or patch a game we have purchased. Another thing is being locked out of a single player game you already have installed. Since all it takes is user authentication, I don't think its asking much to have a second server set to do just that.

I wasn't commenting on DRM.  That whole thing is stupid.

I am sure they could do it, but they don't want to spend on it.  That's all I'm saying.  It's also probably not as cheap as everyone thinks it should be, especially if you want any sort of quick takeover.  Not that I think their DB is enormous, I just think they started looking at a plan and quickly just said "Meh.  What are the chances the power is down for longer than the generator can handle?"

I'd just like to add - I'm going on day four with no power at my house in Seattle...

No need to speculate how they would put together a giant supercomputing data server that's supposed to act as a replacement to your hard drive, as those are already being built.  The wheels are in motion to make all our software run like Steam before too much longer, except with less of our software actually being stored locally.

This is why I said, "Are redundant arrays really enough?"  A major catastrophe strikes and, whether or not you have power, you might not be able to reach any server that has all the apps you purchased.  Suddenly your computer's just a giant paperweight.

While people keep buying games through Steam, they won't bother with setting up another data centre, even if it would only cost em $1.  Why would they spend money when the status quo works fine for them?  I'm pretty sure they didn't guarantee 99.99% uptime when I bought HL2 through Steam, so there is no direct quantifiable financial incentive for them to provide redundant servers etc.  It sucks, but thats the simple reality of it - and I agree it will happen more rather than less.

Now if they got actual proof that this downtime hurts sales of future Steam games, then I'd bet they'd be all over a new datacentre in a shot.  But until someone can write up a business case to change things (and remember, any change is dangerous and expensive), its a case of "Shut up and send me your $50, my Porsche needs an upgrade."

Pants and Yeg give me warm fuzzies.

Stardock gives me the warm fuzzies.


Whether or not you know what you are talking about, you have a point in there somewhere.  When you are coming up with your datacenter plans, you have to decide what level of business continuity you are willing to pay for.  Leasing raised floor is fine for some companies, and you get to leave the headaches of managing the facility to the company you are leasing from.  They are going to have tiered plans, too, and they will be more than happy to bleed your budget if you want.  If Steam was something other than a entertainment delivery system, this would have a bigger impact on Things To Come.

It's one thing to plan for a building power outage for a few hours, and quite another to plan for a regional disaster.  One of those simply requires an appropriately-sized generator.  The other requires fully-functioning remote datacenters with similar hardware.  Guess which one a game company is going to pick?

The generator thing also requires supplier relationships and staff capable of maintaining the thing. Plus it doesn't protect you from events much smaller than a regional power outage, like a fire, or a bomb threat, or a disgruntled idiot.

For most people starting from scratch and only worried about a few servers, a short-term UPS and leasing remote hosting space is usually more economical. Especially in the case of something like Steam where you would probably just move half your authentication cluster offsite, link the two by VPN (they only need to keep authentication and licence info in sync), and reduce your primary site costs in the process.

Maybe dodgy architecture decisions at the start make remote operation impractical. /shrug. One would hope they had people smart enough to think around these problems at the start - but who knows.


Oh, and on an equally geeky note...


pffft. Raised floors are yesterday's thing.

These days all the cool kids building new centres run overhead cable trays instead of raised floors - the old raised floors are a fire risk and a complete bitch to keep clean and organised.

How do you cool your shit off without a raised floor?

Not to mention a lot of raised floors caused goddamned zinc whisker problems.

Our datacenter is using self-enclosed temperature-controlled sealed racks after that catastrophe (and overhead cable trays).

What's a zinc whisker?  Clipped cable?

Also, what sort of temperature control are you talking about?  Genuinely curious.  We started to use the cooling system that came with the Bladecenter rack, but we opted for something that didn't have water in it... meaning just a floor vent.

It's a rack unit that seals shut with a glass paned door and an air conditioner unit built into it. You remotely dial in the temps and each rack is its own environment (we have two of them), so inflow and outflow are controlled and unique to each unit. They are rolling chassis, too. Pretty sweet...not cheap.

Zinc whiskers (http://en.wikipedia.org/wiki/Zinc_whisker) are bits of metal that accrete under  some raised floors. After having water damage from a roof leak, our datacenter was 'cleaned' by a company not trained in datacenters (another story entirely, these are the folks on the next level up heirarchically from me, so it's all out of my control). When they 'cleaned' the underfloor, they spread the whiskers throughout the datacenter and anything that had active cooling sucked them in, shorting the electronics. Fried just about every server and network device they had in there and set us back about a year. As a bonus, the boneheaded system level folks reported it as water damage and got dick for insurance compensation.

Ok, add that to the 2006 grievances, I guess :P For me locally it wasn't a huge deal, though. Maybe a few days of downtime because I have redundant networking here (they don't!). I could do a better job over there, but I like things small, even if it costs me 'the big bucks'. I can sleep at night without my cell phone ringing (don't even have one!).

OK, I see.  The rack just vents the heat into the room then.  I think that was what the Bladecenter rack was going to do, but not much point to that with our setup.  We certainly can't do that with everything; we'd still have to refrigerate the room.  I don't care too much where the cables are, as long as I can see where I am walking.

I think your big mistake with those whiskers is that you had your datacenter cleaned.  After that, your other mistake was contracting numbskulls to do it.  Seems like they must have had some faulty equipment to spread that stuff all over the place.  Who knows.

You should see what it looks like under some of our floors, it's like a xenomorph queen has moved down there.  But our servers stay up as long as we don't fuck with it.

 :-o

Power cabling + several years of dust is a good recipe for fires, and in some cases for underfloor smoke detection shutting your power down.

Plus not keeping it clean and organised means assclowns feel compelled to leave decommissioned wiring and junk under there, before long you can't trace wiring faults or decide whether random junk that's been plugged in for 20 years is doing anything legitimate.

Sounds like the false ceiling in my basement. The previous owner was apparently batshit insane, but had a good supply of shitty coax. It is festooned with useless wires, most eventually connecting with an antenna amplifier. Too cheap to pay for cable, but let's buy $400 worth of cable and extension cords and build a web!

Clownshoes.

I had to wire cat5 across 5 floors in a building in another job. Someone else, some contractor, had done the same with the old cat3 wireing, but only used some of it. We had a live network, and we were told to do one floor at a time, to minimize downtime. You do not want to know the hell of clambering about on ladders into the roofing tiles to mess with foot thick bundles of cables, playing 'guess which one is currently connected to the bank down stairs!'

Fuck me, could people NOT take my 'warehouse' throwaway comment quite so literally ?  I do work in this field and know so many of do as well that actually specifying a data vault with full specs was unneccesary.

Though at this point a shack in Uganda would be better than NOTHING.

I think that's what the halon is for.


We had the SCSI cables pulled from under the floor maybe 1.5 years ago.  Not because they were unused (they were certainly not being used) but because we couldn't put any more cables down and have the tiles lay flat.  We had some outages, despite the fact that the SCSI cables looked nothing like the live cables.  The "if it ain't broke" saying is a good one.

Ironwood, I have talked to people at companies that kept their servers in some weird places.  In that vein... a network connection to Uganda would be error-prone.

No, it vents via a flex tube through the roof. If you've got whiskers in your subfloor and active cooling (fans), it's only a matter of time. Have good insurance.

On wiring: the guy who did data wiring before me was from a telco background and apparently hadn't heard of switches for ethernet (granted, in 2000 a lot of people were still using hubs, but switches were just cheaply available then..and he didn't use hubs, either). He had a dedicated wire for every machine. Have a cluster of eight pcs? Eight wires on a 100' run. It's nice to have a ton of wires to clip if I need an extra somewhere, because it was a good wiring job and he did use cat5, but holy crap there's a lot of wire running into our wiring closet.

Service contracts for servers, insurance for building.  Our machinery doesn't have that sort of lifespan anyway.  We have a few that are possibly eight years old, but most are just a few years old.  Another funny thing, our main datacenter has carpeted tiles.  Never saw that before.

Anyway, thanks for the link on whiskers, I never knew that.  Not that it matters.  If something on a server dies, we just replace it.  They don't seem to be accumulating under our floor last I looked, either.


Now that's funny.  Reminds me of how the LAN switched to ethernet from token ring in 2000, only waiting so long because some management type didn't trust the "new" technology.

Well, he also used to take the entire network down frequently when 'load-balancing' and then mysteriously disappeared without explanation a few years ago. Too bad, he was a cool guy and played guitar.

Speaking of Steam, it has been 3 days since I sent in an email asking if I could have my password manually reset and sent to me since their dumb secret question thing is garbled. No response. Great.

Environmental vandal! (http://en.wikipedia.org/wiki/Montreal_Protocol)

Hey, no fire, no problem.  It's funny that halon is banned (in 2010) because of ozone depletion rather than being deadly to aerobic life forms.

As long as DuPont also owns the patent on the replacement product, it's all good.  Like Freon.