Useless Conversation

[Lantyssa]

In the stratum where I work, we usually deal with relatively-intelligent people but IT is at odds with business decisions.  Business decisions often relating to how much an IT infrastructure costs versus the perceived benefit.  When you start telling someone that they need to replicate their entire infrastructure in New Jersey, people frown.

Oh gods, yes.

[Pennilenko]

In the stratum where I work, we usually deal with relatively-intelligent people but IT is at odds with business decisions.  Business decisions often relating to how much an IT infrastructure costs versus the perceived benefit.  When you start telling someone that they need to replicate their entire infrastructure in New Jersey, people frown.

Oh gods, yes.

I don't have the experience some of you guys have because I am still in school, but in my IT business continuity class they have us reading and assessing case studies of companies that had poor contingency and disaster recovery plans. I can't believe some of the case studies. I can't even begin to imagine why some companies refuse to even spend money on proper backups. Unfortunately, most of the companies in our studies are not named. One case discussed a cloud hosting service that refused to spend money on offsite backups. They didn't have any official contingency plan, no cold site, warm site, or hot site, nothing. They got wiped out by hurricane sandy in new york and never recovered.

My whole book is filled with examples of companies that were absolutely reliant on their IT infrastructure but were unwilling to support that structure financially beyond the shoestring budget that was used to get it up and running.

[HaemishM]

For Haemish and the other writers here: https://twitter.com/worstmuse

I'm so writing were-corgi erotica.

You know that shit would sell. You wouldn't want to go to a book signing but yeah... it would sell.

That Twitter feed made me titter. 

[Hammond]

Many companies act in a reactive manner rather than proactive when it comes to IT spending. Its hard to show payback for a investment that essentially going to just sit there doing nothing and depreciating in value just in case something happens. Such is the joy of IT under staffed and under budget departments (not all companies but a frighting amount that are that way).

[HaemishM]

I can't even begin to imagine why some companies refuse to even spend money on proper backups.

I have a horror story of a company that used Joomla for their web site at my company's recommendation. Unfortunately, in the two years since the site went live, no one updated the version of Joomla to the latest version, which made it very susceptible to hacking. We recommended they do an upgrade/migration to the latest version, which would have cost them probably less than $20k. They didn't want to spend the money. So their site got hacked. We spent time and money getting it back up, but they still didn't want to spend the money on a migration. Fast forward a few months. Their site, which they hosted internally for some dumbass reason, had a critical hardware failure. Their ENTIRE internal network went down and was restored from backups except... their web site, which had not been backed up at all. For over 3 years worth of almost daily content updates. Seems they were told by the IT firm that set this all up that the site wasn't being backed up, that no backups existed, but instead of paying to have the site backed up, they decided against it. $20k more to pay us to recreate their site with the latest version of the CMS. And they finally listened to us when we recommended a real host (rather than hosting in a box in their closet) with real backups and they still bitch about the money that IT firm lost them when their site went down.

Business people are fuckstupid when it comes to IT decisions.

[Chimpy]

Ugh, disaster recovery. I get to jump feet first into designing/implementing a new system for that when I get back to work on Monday. We need to get off Backup Exec on our virtual infrastructure because it sucks.

[WayAbvPar]

In my case my upper management actually responded to our warnings that a vital part of our process to create deliverables for customers was at huge risk (used an ancient, no longer supported piece of software that required a hardware dongle for licensing, so it had multiple points of failure). In the last phases of testing the new in house solution that was commissioned and built which is faster, FAR more user friendly, and more accurate. Sometimes they do get it.

[Yegolev]

My whole book is filled with examples of companies that were absolutely reliant on their IT infrastructure but were unwilling to support that structure financially beyond the shoestring budget that was used to get it up and running.

Your book must have been written by Humans from planet Earth.

Hammond has it.  You might note that I wrote "perceived benefit".  Doing IT correctly costs a lot of money, and it very often doesn't produce anything that you can sell to a customer.  Old fashioned executives can be shown an IT consultant but the mental image they get is one of those old medicine-show wagons from the Wild West.  It can be hard to convince a company that has never experienced a severe outage that redundancy isn't just me trying to sell you twice the hardware that you need to run your business.  It can be done, however, if they have an open mind and you draw enough venn diagrams.

[Yegolev]

Ugh, disaster recovery. I get to jump feet first into designing/implementing a new system for that when I get back to work on Monday. We need to get off Backup Exec on our virtual infrastructure because it sucks.

Just buy three of everything.

[Rasix]

I can put you in contact with people that will sell you three of that. 

I've designed a few DR solutions.  I feel though that I'm missing something not working with the end user software.  I don't know how many masochists actually use our provided GUI/CLI.  Our free GUI is quite nice, however.

[RhyssaFireheart]

If nothing else, my county is quick to pay their jurors.  The laughable part of my jury duty was actually getting that check though.  Two days of jury duty = $10, 64 miles commuting = $6.40, total check = $16.40.

They paid me almost as much in mileage as they did for me to just show up.  And the thank you letter was just hilariously filled with "we know you hate this, but thanks for doing it anyway" lines.

[RhyssaFireheart]

DR solutions are a pain to plan when you're working across continents, I found.  Had to do that back when I was at HP for a client and needed to make sure that the datacenter in France would use the datacenter in Marietta as it's failover location and planning what needed to be done in what order... ugh.  Odly enough, the reverse wasn't true - France wasn't the backup for the US site.  Couldn't figure that one out.

[WayAbvPar]

If nothing else, my county is quick to pay their jurors.  The laughable part of my jury duty was actually getting that check though.  Two days of jury duty = $10, 64 miles commuting = $6.40, total check = $16.40.

They paid me almost as much in mileage as they did for me to just show up.  And the thank you letter was just hilariously filled with "we know you hate this, but thanks for doing it anyway" lines.

Righteous bucks!

[Lantyssa]

Hammond has it.  You might note that I wrote "perceived benefit".  Doing IT correctly costs a lot of money, and it very often doesn't produce anything that you can sell to a customer.  Old fashioned executives can be shown an IT consultant but the mental image they get is one of those old medicine-show wagons from the Wild West.  It can be hard to convince a company that has never experienced a severe outage that redundancy isn't just me trying to sell you twice the hardware that you need to run your business.  It can be done, however, if they have an open mind and you draw enough venn diagrams.

Guess what happened when my boss went with a $20k "savings" and PASSION for a single guy with a couple of contractors over the company with a team that builds out systems as their day-to-day job?

Guess what happens when they sell clients on our DRP but won't actually fund it to work like stated?  (At least we have an off-site disk backup, just not an actual fail-over site with machines that can run more than one of our many VMs.)

Clients are getting more and more stringent about this stuff, but just getting us upgraded from 2000-era machines and OSes was teeth-pulling.  WTF push for a PCI cert and all the costs that entails in order to attract clients if you're not going to provide the resources to keep it?

[Ingmar]

Ugh, disaster recovery. I get to jump feet first into designing/implementing a new system for that when I get back to work on Monday. We need to get off Backup Exec on our virtual infrastructure because it sucks.

How big is your environment?

[Merusk]

Many companies act in a reactive manner rather than proactive when it comes to IT spending. Its hard to show payback for a investment that essentially going to just sit there doing nothing and depreciating in value just in case something happens. Such is the joy of IT under staffed and under budget departments (not all companies but a frighting amount that are that way).

Seen this in a lot of places over the years and I'm happy to be at a place that has a CEO that is savvy, if not wholly versed.  We've got enough people that we're almost over-staffed on some days. 

My gripes are almost all user-related on this side. Our biggest headache is getting accounting to approve the $80k we need for new switches to make the move to VOIP.  The infrastructure guy prayed really hard that our c.a. 1997 phone system would boot when we had an 12 hour power outage last year. (It runs in Windows 2000 and can't be upgraded anymore) 

[Chimpy]

Ugh, disaster recovery. I get to jump feet first into designing/implementing a new system for that when I get back to work on Monday. We need to get off Backup Exec on our virtual infrastructure because it sucks.

How big is your environment?

Not super big, though it is really hard to put an exact number on it as this solution is going to be scaling up pretty quickly to encompass our new system of record and document management systems. Right now we are probably at around 75 VMs using less than 10TBs of storage across the various blade enclosures and SANs. The new system will probably up the number of VMs to around 100 but the storage requirements will jump quite a bit. The plan right now is to move to using Veeam (which I have experience with), the big headache for me is wrapping my head around how to get it to work in an environment built around IBM blades versus the Dell rackmount servers I am used to. So a lot of what I need to do is figure out what IBM offers in terms of hardware that will work for building the disk repositories.

[Ingmar]

Veeam is a good product, but you still need to consider how you're going to get the data offsite. If you have shitloads of bandwidth available I'd go with a cloud sync option so you don't have to fuck around with tape, especially if your change rate is low. At <10-20 TB you're sized pretty well for one of the mid-range Barracuda backup appliances, for example, but there are other options. Bandwidth for pushing the data offsite is the biggest problem.

[Pennilenko]

There should be an IT sub-forum for the professionals to talk shop in where newbies might learn a thing or two.

[Chimpy]

Yeah we are going to look at a number of options overall. Since our primary production site is going to be in Chicago, and our long-term backups are probably going to be hosted in our office datacenter (which is in a different building from our secondary/failover site in Champaign) we are probably going to start with making sure everything works that way first with tape as an archival medium.

[Yegolev]

it is really hard to put an exact number on it as this solution is going to be scaling up pretty quickly to encompass our new system of record and document management systems.

Capacity Management: It's not just for breakfast anymore.  /oldpersonjoke

If you can get some quality deduplication on the source side, it will ease the bandwidth requirement.  You still want to test out some numbers to avoid nasty surprises.

I might guess you have several separate systems, and you probably already have given them a DR classification.  If not, do that, then assign times based around those categories.  Then you can let the business owners argue about who gets to be in what bucket.

Class 1 might be something like four hour recovery time, while Class 4 might be fifteen business days.

Physical tapes are not something I have dealt with for Many Moons.  Good luck with that nonsense.

You probably already know all this.  Too bad you have to write it all down.

[Ingmar]

Yeah we are going to look at a number of options overall. Since our primary production site is going to be in Chicago, and our long-term backups are probably going to be hosted in our office datacenter (which is in a different building from our secondary/failover site in Champaign) we are probably going to start with making sure everything works that way first with tape as an archival medium.

If you've got a good sized pipe between the two sites then you can just do replication yeah. Wasn't an option for us, our pipe to our Fargo site is tiny and we couldn't really afford to bump it up.

[Chimpy]

Oh, tape is going to be solely for "we need to look at the data as it was >1+month ago," if I can get the capacity right, tape will be the home to quarterly/annual archival stuff only and I will have 90 days worth of weekly fulls and probably 30ish days of nightly incrementals on disk.

If you've got a good sized pipe between the two sites then you can just do replication yeah. Wasn't an option for us, our pipe to our Fargo site is tiny and we couldn't really afford to bump it up.

We will have 10Gb between the two main sites, and our office where the backup copies will be stored has 1Gb link back into the campus network (we will look at getting a 10Gb pipe into there) so bandwidth is not really an issue for us.

[Hammond]

Oh the joys of tape backups. Thankfully I only have to do one tape a month for long term archival of our Accounting info. Although now that CA sold off their Arcserve line of products we are probably going to have to change vendors again...

[Trippy]

In the stratum where I work, we usually deal with relatively-intelligent people but IT is at odds with business decisions.  Business decisions often relating to how much an IT infrastructure costs versus the perceived benefit.  When you start telling someone that they need to replicate their entire infrastructure in New Jersey, people frown.

Oh gods, yes.

I don't have the experience some of you guys have because I am still in school, but in my IT business continuity class they have us reading and assessing case studies of companies that had poor contingency and disaster recovery plans. I can't believe some of the case studies. I can't even begin to imagine why some companies refuse to even spend money on proper backups. Unfortunately, most of the companies in our studies are not named. One case discussed a cloud hosting service that refused to spend money on offsite backups. They didn't have any official contingency plan, no cold site, warm site, or hot site, nothing. They got wiped out by hurricane sandy in new york and never recovered.

My whole book is filled with examples of companies that were absolutely reliant on their IT infrastructure but were unwilling to support that structure financially beyond the shoestring budget that was used to get it up and running.

Code Spaces, a source code hosting service running on Amazon AWS, was recently put out of business cause they didn't have proper backups (and cloud security).

http://www.networkcomputing.com/cloud-infrastructure/code-spaces-a-lesson-in-cloud-backup/a/d-id/1279116

[Pennilenko]

Thanks for that. I have actually been hunting for recent stuff like that for a paper.

[Yegolev]

I used to think that people who ran businesses were above-average, but I've learned that most of them are just regular old dumbasses.

[Ironwood]

Yes.  When you get in a room with the Captains of Industry, you realise you're just in a cage with a bunch of cunts.

[Ironwood]

Code Spaces, a source code hosting service running on Amazon AWS, was recently put out of business cause they didn't have proper backups (and cloud security).

http://www.networkcomputing.com/cloud-infrastructure/code-spaces-a-lesson-in-cloud-backup/a/d-id/1279116

That's a horror story.  Also, what bothers me about people who want to get into Cloud and ask what's 'better' AWS or Azure.

To which the answer is 'Both'.

[Merusk]

Geez, makes my "force the users to unlearn Defragging habits when giving them a SSD" problems insignificant.

[Cheddar]

IT is still viewed as expense and not investment.  Seeing this changing, but its a challenge.

[Selby]

I can't even begin to imagine why some companies refuse to even spend money on proper backups.

My first company out of college justified it because "legal told us we are not required to keep anything beyond 6 months and actually we want you to actively delete all emails and files older than 6 months from the network and your personal computers kthx!"  This was a company that produces the hardware that makes almost all modern technology circuits possible.  I may give DoE grief but at least they actively mandate backups of EVERYTHING and follow up with it all.

[bhodi]

Heh, IT. I hate it so very much, but as long as the checks keep clearing...

[Morat20]

I used to think that people who ran businesses were above-average, but I've learned that most of them are just regular old dumbasses.

Jesus fuck yes. I've been lucky lately -- I've been mostly working with competent people, but good lord.

I've got a friend that does environmental and waste disposal stuff for some sizable companies -- t his job is all about shipping, storing, finding uses for, selling, and generally handling hazardous stuff. The stories he tells -- companies manufacturing some really toxic crap deciding they'll just pile the shit up in drums and only deal with the hassle of disposing it every six months (the EPA frowns on that, because "we left it in some drums in our yard" is generally not sufficient for keeping toxic shit from getting into people). They found two entire railway cars someone had filled with chemicals and then lost for ten years.

It was a nightmare trying to figure out what the hell was inside it without opening it, so they'd know how dangerous it was -- all while dealing with the original company (who'd lost the records) and having a big slap fight over who had to pay to dispose of it, the owners of the chemicals inside it, the owners of the railway car, the insurance companies that paid out when the cars and chemicals were lost, or the company that found it in an audit.

I swear, half the people in the chemical and refining business seem like they're dumb enough to piss in their own well.

[Lantyssa]

Even the smart ones can be dumb.  My dad had so many horror stories about engineers not understanding what happens with chemicals in the real world.