Cloud computing services

[K9]

Thanks Trippy, this is really helpful. I'll have a good dig through this tomorrow and see what I can make work. I really appreciate it.

Just let us know when you get to the End-Boss.  He's hard.

 

At the moment I feel like I'm wiping on Lord Marrowgar all over again, only this time I can't blame it on inept, semi-afk people 

[Samwise]

I'm disappointed that this thread has yet to produce any gems via the cloud-to-butt translator plugin.

[Ironwood]

I would like you to know that it was considered.

[Ironwood]

Thanks Trippy, this is really helpful. I'll have a good dig through this tomorrow and see what I can make work. I really appreciate it.

Just let us know when you get to the End-Boss.  He's hard.

 

At the moment I feel like I'm wiping on Lord Marrowgar all over again, only this time I can't blame it on inept, semi-afk people 

Well, the good news is you can make some IAM accounts for Trippy and I and he can do the clever bits and I can do the charging you for it.

[Quinton]

If I get one of the free micro instances running, I should be able to load BEAST and my runfile into it fairly simply and set it running? I think this would be the next step, while considering the power:cost voodoo. Is it a matter of effectivley opening a remote desktop and then dragging and dropping the files and software in, or do I need some (semi-)permanent amazon skydrive folder for the instance to read and write from?

The micro instances are designed for bursty use and will throttle your CPU usage if you try to run continuously.  They are not going to be good for trying to figure out how the larger instances will perform.  They should be fine for figuring out how to setup and configure your environment though.

[Ingmar]

At the moment I feel like I'm wiping on Lord Marrowgar all over again, only this time I can't blame it on inept, semi-afk people 

CLOOOUDSTOOOORM

[Viin]

... They also have a built-in backup feature where you can take a "snapshot" of an EBS volume which gets stored on S3. They are, however, not very performant unless you pay more and even then you are still not getting the performance of a "local" drive. ...

Do you have to use an EBS volume to move data to an S3 bucket? Or is there another way to move data to an S3 bucket? (in the case of BEAST, maybe during a log roll?)

[Trippy]

No you don't have to use an EBS volume to move something to S3. S3 is independent of EC2 and will work with anything that can "speak" HTTP.

[Quinton]

Yeah, there's a ton of tools out there to work with S3, including things like: http://code.google.com/p/s3fs/ that use fuse to let you mount s3 buckets as remote filesystems, etc.

[K9]

Thanks Trippy, this is really helpful. I'll have a good dig through this tomorrow and see what I can make work. I really appreciate it.

Just let us know when you get to the End-Boss.  He's hard.

 

At the moment I feel like I'm wiping on Lord Marrowgar all over again, only this time I can't blame it on inept, semi-afk people 

Well, the good news is you can make some IAM accounts for Trippy and I and he can do the clever bits and I can do the charging you for it.

Do you accept DKP?

[Yegolev]

Huh? Netflix and a large number of other commercial companies run stuff on EC2 and AWS. This is why when AWS goes down a large swath of the Web goes down with it.

Hey, I'm just going by your post which didn't paint the most rosy picture.  Load issues and service interruptions were prominent, at least in my mind.

I do imagine, though, that if you're Netflix then you're paying extra for better performance and uptime.  The EBS description is closer to what I expect from a SAN, though.  Small-time instances would get low-tier storage, high-dollar instances would get tier 1, and everybody getting snapshots is great.  I had to configure clone devices for anything I wanted to take a snapshot of, meaning I was grabbing double the storage and getting frowned upon by the SAN people, aka the Scrooge McDucks of Disks.

I reckon in my "spare time" I will look to see what sort of playtime I can have with this stuff.

[Salamok]

... They also have a built-in backup feature where you can take a "snapshot" of an EBS volume which gets stored on S3. They are, however, not very performant unless you pay more and even then you are still not getting the performance of a "local" drive. ...

Do you have to use an EBS volume to move data to an S3 bucket? Or is there another way to move data to an S3 bucket? (in the case of BEAST, maybe during a log roll?)

Based on my limited knowledge (talking to people who use it, reading AWS documentation and going to a few AWS workshops):
S3 is not a normal file system, it very quickly writes data to 2 locations before firing a callback giving the thumbs up that your data is safe (then proceeds to copy your data to 1 or more additional locations for performance) the process for getting stuff to and from it does not use normal file system commands so programs that are written to write files that way (aka everything) will not just work without help.  EBS is exactly like a normal file system, when you create an EC2 instance it is running on an EBS volume.  

If it were me in the case of BEAST I wouldn't even bother with S3 (unless it has a version that directly supports it), I would just have it write it's results to the local (ie root) file system and periodically pull down the results to a local machine and clear out the logs.  Use the EBS snapshot tool to backup the setup w/o data.

Actually if it were me I would just build a machine that was designed to crunch this type of data.  The only 3 upsides to using AWS for this that I can see are:

• Scalability - IF this proves to be even moderately cost effective using AWS then you can scale it very very easily - this would allow you to turn around and sell unlimited amounts of BEAST time to other people.
• Grant Money - It is probably much much easier to write a successful grant proposal filled with cloudy buzzwords than it is to write up a proposal for what looks to be a series of really kick ass gaming rigs.
• Awesome Learning Experience - You get to learn a few things about AWS and if you wrote your grant properly you will be getting paid to learn AWS (and hopefully attend cool conferences and workshops)

Huh? Netflix and a large number of other commercial companies run stuff on EC2 and AWS. This is why when AWS goes down a large swath of the Web goes down with it.

Hey, I'm just going by your post which didn't paint the most rosy picture.  Load issues and service interruptions were prominent, at least in my mind.

I do imagine, though, that if you're Netflix then you're paying extra for better performance and uptime.  The EBS description is closer to what I expect from a SAN, though.  Small-time instances would get low-tier storage, high-dollar instances would get tier 1, and everybody getting snapshots is great.  I had to configure clone devices for anything I wanted to take a snapshot of, meaning I was grabbing double the storage and getting frowned upon by the SAN people, aka the Scrooge McDucks of Disks.

I reckon in my "spare time" I will look to see what sort of playtime I can have with this stuff.

Banks and credit card companies are also heavy cloud users, I was talking to a guy that said he helped some Australian commonwealth bank set up an entire system in the cloud to handle servicing a south east asian country that they didn't not feel was stable/safe enough to want a large physical presence/heavy infrastructure investment in.  Cloud scaleability works well for robustness as well, shit crashing regularly isn't as inconvenient in a large scale self aware cluster that is designed to spin up more nodes to deal with demand.  I suppose it comes down to what you define as a commercial application, the AWS system certainly is not designed to run off the shelf (shrink wrap) software but it is definitely aimed at enterprise software.   I consider Amazon (the website) to be one of the most amazing examples of a website on the internet (it is complex, fast and very robust), AWS was written first and foremost to create that "commercial application".

[Yegolev]

I agree with all of that stuff, it's mostly the term "cloud" that confuses things.  There are many, many ways to define and deploy virtual computers; my domain is not web services but rather what is traditionally called "enterprise computing", which has different architectural requirements than something like AWS and web stores and so on.  So it's really different from what I'm used to, while probably using tech that I'm somewhat-to-very familiar with.

So a bank "in the cloud" can mean lots of things.  What probably is meant here is that the main computing systems are not in Unstable Asian Location but there are leveraged endpoints for performance reasons.  Again that might mean "web interface for end users" or it might be old-style branch-office shit.  Or maybe something else, not sure.

My experiences in this bank story would be back in HQ, as it were, dealing with the largest systems and their concerns for performance, reliability, DR, etc.  I think that's not "cloud" because it's not abstracted enough (via some slick management front-end, which is how Cloud is sold to IT directors: reduce the number of FTE required to manage the systems) or because it doesn't provide a slick interface to end users.

I suppose this also connects to my advice to learn virtualization or get out of IT.  The fact is that major efforts are underway to automate away old-fashioned admin tasks, under the banner of Cloud Computing.  Amazon's thingy (I read a bit this morning) that lets you enter basic requirements and then builds out a partition for you?  That's a huge item in corp IT that will replace lots of frontline people sooner or later.  I you want to keep employed, it's advisable to be one of the people that know how to work on the cloud management systems when they break or need upgrading (and they will).

[Salamok]

Amazon's thingy (I read a bit this morning) that lets you enter basic requirements and then builds out a partition for you?  That's a huge item in corp IT that will replace lots of frontline people sooner or later.  I you want to keep employed, it's advisable to be one of the people that know how to work on the cloud management systems when they break or need upgrading (and they will).

The thing that amazed me (even more amazing since I don't have a heavy bash/powershell background) was the amount of stuff that could be done from the cli. We look at "cloud services" and think that's pretty cool then look a little bit deeper and see how people are manageing cloud services (AWS CLI tool, puppet, chef, vagrant, etc...) and think holy fuck.  I don't have a any recent exposure to large scaleable environments (like since 1998) so the thought that people are out there spinning up complex infrastructures almost as quickly as I start my little toy virtual box VM is pretty amazing to me, on top of that they are writing programs that monitor these infrastructures and spin up/down resources based on demand is just sick...

It actually becomes the cheapest option for many large scenarios that produce peak hour needs that would be very expensive to build as your baseline.

[Yegolev]

Since you've managed to make a left turn into my IT arena, here's a link to an IBM product for this:
http://www-03.ibm.com/systems/software/director/

Systems Director, or ISD as I heard it from Nigel Griffiths yesterday, is the Borg of system management tools.  I mean suite, not tool.  It works with many other lesser tools, such as all that Tivoli shit, NIM, etc., and it at least wants to give a GUI interface to doing just that: spin up on-demand partitions, rearrange workload on demand and automatically, and mostly take the neckbeard out of provisioning as well as general management of an IT infrastructure.  Of course, we're not there yet but it's coming.  Eventually, executives will be able to press the EASY button and get a fully-formed IT infrastructure as if it sprang from the forehead of Zeus (probably not really) and get a bill in the mail on the following week (probably really).  I expect that Amazon's press-button-get-computer system will be how everyone does things in the future, to some degree.

I believe HP's parallel product is Operations Orchestration (HPOO, nyuk nyuk) but I'll know soon enough once I take this training class.

The adoption is a bit slow, like virtualization, but the advantages of the required subcomponents are very real.  Now that virtualization is mostly in the wild, we can do newer things like IBM's Live Partition Mobility (if you have the latest hardware frames) which lets us move LPARs from one physical host to another with a subsecond network interruption.  This is even better than older shit like partitions within partitions and OS-based workload management.  That of course paves the way for enhanced load balancers that can move shit around based on need.  Automating this stuff is already possible.

So what will I be doing?  It's a fact that people will always need custom solutions and so I'm going to be the guy that designs those solutions, or architects them if I get moving on the career fast enough before I retire or die.  That will require knowledge of all these systems and the directors on top of them.  The admins that currently do ground-level things either manually or via larger toolset will need to know how to use and abuse the overarching suite that will be doing what they are doing now.

[Viin]

I think this is all very cool stuff. I peruse the AWS job openings every once in awhile and day dream, but the wife would never move to Seattle.

[Yegolev]

Heh, why does Amazon require you to be on site?  I know you're in management but still.  Employees are going to the Cloud as well.

[Ironwood]

It is rather hard to get in the door though. 

[Hammond]

Interesting stuff Yegolev, I have worked on a few smaller virtual clusters and never had anything big enough to justify the neato wizzbang stuff that IBM director / HPOO brings to the table. I am finally getting to the point that I am building out the bones of a new cluster here with a eye to retire / migrate most of the physical boxes.

Oh on the subject of Amazon one of my old bosses who current works there is going to be moving on. He is in the network side of the house and I guess that the hours are just brutal and he is tired of putting in 60 to 80 hour weeks.

[Ard]

Yeah, that jives with what more than a few people have told me about Amazon since I moved here, "Do not work there, unless you're only doing it to get experience and move on".

[Yegolev]

It is rather hard to get in the door though. 

This is very true for quite a while, and moreso now with all this push to have the machines fix themselves, plus every company embracing LEAN and downsizing teams.  There are portholes in India and Manilla, though I'm not suggesting that's the best way to get into large-corp IT.

There's still an opening to backfill my old spot that will go external on August 6, but I believe an old coworker with experience in the environment will be getting it.  Besides knowing someone, of course, I suppose what first-world job-seekers need to do is go contract and get converted.  Places to look include those which don't allow offshore resources, such as gubment and financial.

[Ironwood]

It's not my goal anymore.  Ironically, it's much easier to be a God amongst insects than kill yourself working in an environment where everyone's a clever dick and you don't stand out.

 

[Yegolev]

Not sure that's ironic.

I've done a great job keeping my hours at 40/week, and from a recent meeting about timekeeping, it seems others in my group are also doing fine with that.  I imagine it depends greatly on how awful your organization is.  The smaller orgs I have been part of have all been shittier than the two massive global corps that I have worked for.

Also interesting is that since I switched over to another account, one that really seems to be at least as ornery and probably-unquestionably more important to my employers, my stress and frustration is lower because I'm no longer personally vested in any shenanigans or personalities.