Sony's PSN down "for a day or two"

[KallDrexx]

Though how a hacker could get in through a successful DDoS is beyond me... 

They aren't saying they got through directly because of the DDoS, they are saying they got through unnoticed because the network staff was busy dealing with the DDoS and not performing routine monitoring of the other aspects that could have detected an intrusion faster.

[CharlieMopps]

Though how a hacker could get in through a successful DDoS is beyond me... 

They aren't saying they got through directly because of the DDoS, they are saying they got through unnoticed because the network staff was busy dealing with the DDoS and not performing routine monitoring of the other aspects that could have detected an intrusion faster.

Total and complete horseshit. There is no defense against a DDOS attack, you block the IP they're trying to hit at the core router level and thats all you can do. An intrusion isn't something you detect and stop... like a burglar... A network intrusion is like a leak in a damn. If the route exists the water/intrusion is GOING to happen. You have to make your damn/network leak/intrusion proof. If the damns already got a crack in it, finding the leak doesn't do you any good. When it collapses it's your fault for building a shitty damn, you can't blame the water. They can make it illegal for water to flow downhill they want, but it'll never do them any good.

[KallDrexx]

Total and complete horseshit. There is no defense against a DDOS attack, you block the IP they're trying to hit at the core router level and thats all you can do. An intrusion isn't something you detect and stop... like a burglar... A network intrusion is like a leak in a damn. If the route exists the water/intrusion is GOING to happen. You have to make your damn/network leak/intrusion proof. If the damns already got a crack in it, finding the leak doesn't do you any good. When it collapses it's your fault for building a shitty damn, you can't blame the water. They can make it illegal for water to flow downhill they want, but it'll never do them any good.

1) You have to monitor what traffic is DDoS traffic and what isn't, and actively block those IP addresses.  There's no magical block_all_ddos_ips executable.  Thus my point, it takes resources away from general NOC operations to block and stave off the DDOS attack.

2) If there is no way to detect an intrusion then how do we know there was an intrusion?  Oh right, because they saw signs that an intrusion occurred and investigated it.  This requires resources, which may not have been available due to issue #1

There is no such thing as a network being intrusion proof, except for systems that have no network access at all.  People have hacked into DOD and pentagon systems before, and the most you can do is be vigilant in keeping up with security practices and monitoring to catch things before or as they are happening. 

I'm not excusing Sony, but what they are saying is believable.  It doesn't make it any less their fault or any less their problem, but it's still believable in what they are saying.

[CharlieMopps]

Total and complete horseshit. There is no defense against a DDOS attack, you block the IP they're trying to hit at the core router level and thats all you can do. An intrusion isn't something you detect and stop... like a burglar... A network intrusion is like a leak in a damn. If the route exists the water/intrusion is GOING to happen. You have to make your damn/network leak/intrusion proof. If the damns already got a crack in it, finding the leak doesn't do you any good. When it collapses it's your fault for building a shitty damn, you can't blame the water. They can make it illegal for water to flow downhill they want, but it'll never do them any good.

1) You have to monitor what traffic is DDoS traffic and what isn't, and actively block those IP addresses.  There's no magical block_all_ddos_ips executable.  Thus my point, it takes resources away from general NOC operations to block and stave off the DDOS attack.

2) If there is no way to detect an intrusion then how do we know there was an intrusion?  Oh right, because they saw signs that an intrusion occurred and investigated it.  This requires resources, which may not have been available due to issue #1

There is no such thing as a network being intrusion proof, except for systems that have no network access at all.  People have hacked into DOD and pentagon systems before, and the most you can do is be vigilant in keeping up with security practices and monitoring to catch things before or as they are happening. 

I'm not excusing Sony, but what they are saying is believable.  It doesn't make it any less their fault or any less their problem, but it's still believable in what they are saying.

1) you're wrong. I worked in a NOC for 3 years. Our customers would get hit by DDOS attacks all the time. You don't block the incoming IP addresses, that would be stupid... there are usually thousands, if not tens of thousands of IPs hitting you at once. You block the destination at a core router. Then the attacker has to start switching target IPs. Every new IP they add to their attack vector degrades their effectiveness.

2) I shouldn't have said their is no way to detect an intrusion... there is... but by the time you've detected it, it's pointless. In my example, the damn is already doomed. They're already in, you've already lost your info. All the info they need was probably smaller than a few hundred megabytes... maybe a couple of gigabytes that the most. Given the probable size of Sony's outbout trunks and that the attacker was probably also operating from inside some other hacked system with equivalent bandwidth the movement of data off Sonys network likely took minutes, maybe even seconds.

What they're saying is not believable. It's silly. The data that was stolen should never have been accessible from outside Sonys firewall... period. Unless you're sitting in the Corporate IS department of Sony Inc after going through 2 or 3 security doors, then logging into your work PC, then logging into their billing system should you have been able to see that kind of info. They only people at sony that would need to see that level of data are developers.

[Arthur_Parker]

Received the email


Figured was odd as last game I touched of SOE was SWG back in August 2003, then searched the old email account and the only reference to sony was the Vanguard Beta    I knew that 30 minutes was a mistake.

[Amaron]

There is no defense against a DDOS attack, you block the IP they're trying to hit at the core router level and thats all you can do.

You can also upgrade so that's not totally true.

[sinij]

At this point, Sony is officially Circling The Drain.

[Tale]

Figured was odd as last game I touched of SOE was SWG back in August 2003, then searched the old email account and the only reference to sony was the Vanguard Beta    I knew that 30 minutes was a mistake.

I specifically created an email redirect on my domain to use in applying for the Vanguard beta. It has received the SOE email, but I don't think my EverQuest email address has.

[Hawkbit]

At this point, Sony is officially Circling The Drain.

Soon all we'll have left is Steam. 

[Soln]

indeed, this DDoS excuse seems convenient.  Blame a loose confederation of well known (via Wikileaks) hackers.  In short, blame the only people politicians may have heard of.

[kildorn]

"network staff was busy dealing with the DDoS" => code for "we kept bugging the network folks and having hourly huddles to discuss our current status so often that they couldn't get any actual work done"

At least, if it's anything like any major outage I've worked on. My favorite was bugging the Ops staff every 30 minutes for a status update when the problem was electrical. So, you know, ask the fucking electrician, none of us are actually working on the issue and could be doing something useful instead of running between our office and meeting rooms all day just to say "I don't fucking know?"

[01101010]

"network staff was busy dealing with the DDoS" => code for "we kept bugging the network folks and having hourly huddles to discuss our current status so often that they couldn't get any actual work done"

At least, if it's anything like any major outage I've worked on. My favorite was bugging the Ops staff every 30 minutes for a status update when the problem was electrical. So, you know, ask the fucking electrician, none of us are actually working on the issue and could be doing something useful instead of running between our office and meeting rooms all day just to say "I don't fucking know?"

Working as intended. 

[bhodi]

Kildorn has it exactly right.

A conference bridge has been opened on this P1, please dial in for 6 hours of finger pointing and pacing!

[tgr]

We discovered that the intruders had planted a file on one of our Sony Online Entertainment servers named “Anonymous” with the words “We are Legion.”

Credible evidence.

[CharlieMopps]

And the real cause is? Incompetence!

"Sony was using outdated versions of the Apache Web server software, which "was unpatched and had no firewall installed." The issue was "reported in an open forum monitored by Sony employees" two to three months prior to the recent security breaches, said Spafford."

http://consumerist.com/2011/05/security-expert-sony-knew-its-software-was-obsolete-months-before-psn-breach.html

[Yegolev]

And the real cause is? Incompetence!

This is always the real answer in any major debacle.  That and "budget-minded" executives.

[rattran]

And I picked a bad time to switch to LastPass I guess.
http://blog.lastpass.com/2011/05/lastpass-security-notification.html

Hooray for the cloud, always available except when it isn't.

[KallDrexx]

You always have access to your passwords though without access to their servers.  Just not syncing.

[Minvaren]

More fun Sony/PSN news

According to Spafford, security experts monitoring open Internet forums learned months ago that Sony was using outdated versions of the Apache Web server software, which "was unpatched and had no firewall installed." The issue was "reported in an open forum monitored by Sony employees" two to three months prior to the recent security breaches, said Spafford.

I'm looking for an adjective past "clownshoes" here and failing...    

[Yegolev]

My new favorite word/phrase is "best shore".

[fuser]

More fun Sony/PSN news

According to Spafford, security experts monitoring open Internet forums learned months ago that Sony was using outdated versions of the Apache Web server software, which "was unpatched and had no firewall installed." The issue was "reported in an open forum monitored by Sony employees" two to three months prior to the recent security breaches, said Spafford.

I'm looking for an adjective past "clownshoes" here and failing...    

The IRC log of the "hack" is the one from #ps3dev on efnet.

[13:41:06]   <trixter>   I also know that the server that does the x-i-5 tickets is a bit more tight about the ciphers than any other system in sonyland
[13:41:56]   <trixter>   if sony is watching this channel they should know that running an older version of apache on a redhat server with known vulnerabilities is not wise, especially when that server freely reports its version and its the auth server
[13:42:33]   <SKFU>   its not old version, they just didnt update the banner
[13:43:03]   <trixter>   I consider apache 2.2.15 old
[13:43:08]   <SKFU>   which server
[13:43:11]   <trixter>   it also has known vulnerabilities

Curious that RHEL/CentOS doesn't ship with 2.2.15 or updated to it. From the time/kernel it sounds like an OS of 5.2/5.3 setup but with a custom/source compiled 2.2.15 Apache for some reason. I'm wondering if they pulled in the 2.2.15 for some feature that the vendor package didn't have let the whole security lapsed because the staff never kept on top of the source compiled updates. Even if they were running Spacewalk to monitor and deploy updates the httpd(apache) one would slip by any monitoring unless the staff monitoring updates were aware of the lack of a Apache package.

[Soln]

this RHEL3 or RHEL5?   

[Tale]

The IRC log of the "hack" is the one from #ps3dev on efnet.

[13:41:06]   <trixter>   I also know that the server that does the x-i-5 tickets is a bit more tight about the ciphers than any other system in sonyland
[13:41:56]   <trixter>   if sony is watching this channel they should know that running an older version of apache on a redhat server with known vulnerabilities is not wise, especially when that server freely reports its version and its the auth server
[13:42:33]   <SKFU>   its not old version, they just didnt update the banner
[13:43:03]   <trixter>   I consider apache 2.2.15 old
[13:43:08]   <SKFU>   which server
[13:43:11]   <trixter>   it also has known vulnerabilities

That's the log I posted on April 28: http://forums.f13.net/index.php?topic=20733.msg924037#msg924037

[Rendakor]

And this was posted 4 posts up:

More fun Sony/PSN news

According to Spafford, security experts monitoring open Internet forums learned months ago that Sony was using outdated versions of the Apache Web server software, which "was unpatched and had no firewall installed." The issue was "reported in an open forum monitored by Sony employees" two to three months prior to the recent security breaches, said Spafford.

I'm looking for an adjective past "clownshoes" here and failing...    

And the real cause is? Incompetence!

"Sony was using outdated versions of the Apache Web server software, which "was unpatched and had no firewall installed." The issue was "reported in an open forum monitored by Sony employees" two to three months prior to the recent security breaches, said Spafford."

http://consumerist.com/2011/05/security-expert-sony-knew-its-software-was-obsolete-months-before-psn-breach.html

[Simond]

More fun Sony/PSN news

According to Spafford, security experts monitoring open Internet forums learned months ago that Sony was using outdated versions of the Apache Web server software, which "was unpatched and had no firewall installed." The issue was "reported in an open forum monitored by Sony employees" two to three months prior to the recent security breaches, said Spafford.

I'm looking for an adjective past "clownshoes" here and failing...    

"Sony"

[kildorn]

this RHEL3 or RHEL5?   

It's Redhat, their repos are old enough I'm shocked they have x64 rpms. I have no idea why people insist on paying redhat for slow shitty service.

(RHEL 3->5 stock 2.0.57, RHEL6 stocks 2.2.15. Both are crazy out of date.)

[fuser]

To everyone who said "already posted", wanted to put some reference around the version numbers and OS posted  

It's Redhat, their repos are old enough I'm shocked they have x64 rpms. I have no idea why people insist on paying redhat for slow shitty service.

(RHEL 3->5 stock 2.0.57, RHEL6 stocks 2.2.15. Both are crazy out of date.)

RHEL5.x uses Apache 2.2.3 and RHEL6 moved up to 2.2.15 if I remember correctly (using CentOS everywhere so haven't used 6 in production). Ah here's a RHEL distro package overview. Both are technically behind the curve in latest and greatest but Redhat patches any venerabilities against the packages. Redhat is aimed to give a stable life cycle product not going to a bleeding edge version every release.

The problem here is the either they left a RHEL6 box totally unpatched (which doesn't work because there is only one errata for mod_auth_mysql) or a manually compiled package

[Minvaren]

And this was posted 4 posts up:

I r kin reed gud.   

[brellium]

Large fraud hit?  You call your card company, say the words "I didn't make this charge", and they go poof.  Shred the card and wait for your new one to arrive.  It's not like you ever have to pay it.

I use my card for absolutely everything.  I don't carry cash most of the time.  Checking account is for bills.  We've had one fraudulent charge in 10 years, and we live in the identity theft capitol of America.  Hell, the credit card calls us about charges they think might be fraud (mostly me buying shit late at night).

Maybe I'm just lucky, but I find the fraud risk completely manageable and easy to nullify in the case that it occurs.  

This, most thieves hit in the same manner, and the bank quickly stops charges.

[waffel]

Who can forget this classic?

http://www.youtube.com/watch?v=8AyVh1_vWYQ

[fuser]

It cannot get any worse from the latest reports

TOKYO, May 7 (Reuters) - Sony said on Saturday it had removed from the Internet the names and partial addresses of 2,500 sweepstakes contestants that had been stolen by hackers and posted on a website, and said it did not know when it could restart its PlayStation video games network.

[Tale]

Soon we will be marking three weeks of PSN downtime, when they initially said "a day or two", and there's no ETA. They don't deserve to get any of us back.

[Hawkbit]

Take a game like Brink, they're fucked on PS3 launch if Sony doesn't get it going.  At least Portal 2 got a day or two of sales before the crash, and also has a single player.  From what I've seen of Brink, it's about 98% online... if multiplayer isn't live at launch, there's potential for significant losses. 

[CaptainNapkin]

Yeah I typically pick up multiplatform games for the Xbox, but because of the Steam deal I went PS3 for Portal 2. Since I just set up a projector I want to play the co-op on the PS3 (I haven't built my HTPC yet). I suppose you could say it's set of circumstances/timing thing, but it's left a bad taste in my mouth for the PS3. On the other hand, my only PS3 purchases to date were Demon's Souls, Uncharted, Fat Princess, and Flower... so at the end of the day I guess I wasn't a Sony cash cow anywho.

For me it's the simple fact of how long it's been down. If it was back up in a week or so I likely wouldn't have given resubscribing to their services a second thought, now not so much.

*edit - bourbon = typos

[Raguel]

The only thing I care about really is getting my unlockables/dlc for Dragon Age 2.