A tech support incident at work today

[bhodi]

Security auditors came in to check our network out; They send two trained monkeys in to hit a big green 'go' button on some software that trolls around and doing port scans. They brought in a fluke and two desktops running something similar to nessus to scan our datacenter. They go to plug the desktops in and their power cables won't fit.

You see, our datacenter racks run 220v power, as opposed to your household 110v. The 220 power cables use the same connector that plugs into your computer, but the other end uses a different kind of connector, not your standard 3 prong wall outlet. This confused them briefly, but they were smart! They just found some 220 cables lying around, plugged them into our racks and then into their computers.

Then they flipped the power switch. BANG! Magic blue smoke. I stood watching, agape -- these guys had already thrown their weight around, hassled us into dropping all of our ACLs, so I'd be damned if I was going to lift a finger to help them. They look at the ruined power supply, shrug, and plug in the SECOND desktop. Then they flipped the power switch. BANG! Magic blue smoke again.

Then they look at me. I walk over, and as casually as I can, point to the 110<->220 selector switch on the back of the desktop power supplies.

[Trippy]

It's a good thing they didn't set off the fire supression system (assuming you have one). In their defense for the first mishap most laptop power supplies and a lot of desktop ones are auto-switching so I can understand them not realizing what was about to happen. Of course the second time they did it...

[Yegolev]

See, in that situation I would not have even pointed out the problem, particularly if they had an attitude.  It would have been even more awesome if they set off the halon system.  Everyone hates that, and I know where the exits are.

Port scans are fun.  They can drop an HACMP cluster in short order if they hit the right port, which I know because we experienced some port scans that did exactly that, dropping our biggest production servers.  Every time after that, we were informed beforehand that the auditors were going to be doing port scans and which ports.

A few years ago, we installed a plastic dome over the Big Red Button in our main datacenter.  Someone mistook it for the button that opens an automatic door.

[Roac]

A few years ago, we installed a plastic dome over the Big Red Button in our main datacenter.  Someone mistook it for the button that opens an automatic door.

At least that's something.  The last place I worked at had a cardbord box cut just to the right side that was over it - held together and attached to the wall with gobs of tape, and a sharpie note that said "DO NOT LEAN AGAINST THIS".  Because apparently a few years before I arrived, someone did.

[Polysorbate80]

Reminds me of the sign posted on one of the glass windows in the tech booth in our university's football stadium: "DO NOT PLACE BUTTOCkS AGAINST GLASS."

I'll leave it to your imaginations to visualize what prompted that one.

[Morat20]

A few years ago, we installed a plastic dome over the Big Red Button in our main datacenter.  Someone mistook it for the button that opens an automatic door.

At least that's something.  The last place I worked at had a cardbord box cut just to the right side that was over it - held together and attached to the wall with gobs of tape, and a sharpie note that said "DO NOT LEAN AGAINST THIS".  Because apparently a few years before I arrived, someone did.

*snort*. I've had to ask about a few of the more interesting ad-hoc signs around. (My personal favorite, written in sharpie on the bottom of a paper tray: "DO NOT PUT PAPER IN THIS TRAY! IT JAMS PRINTER!"). I found that while trying to clear a paper jam on that printer. Apparently, someone can't read.

One of the more fun parts of our new-hire orientation is the "DO NOT MOUNT TO THE /tmp DIRECTORY!" lecture. Someone caused a 20+ hour downtime because he mounted the entire user partition to the /tmp directory, forgot to unmount it, and hopped a plan to Russia. Sometime around 3:00 AM, a cleanup job got kicked off -- including a recursive delete of the /tmp directory.

[Yegolev]

Got to be more stingy with that root password.  Why would someone do that anyway?

[Morat20]

Got to be more stingy with that root password.  Why would someone do that anyway?

He wanted to copy some data, and he was stupid. It got worse -- since he was a contractor, root access got restricted to insane levels so that ANYONE (and mind you, contractors did ALL administrative tasks for the entire network) needing root had to go hunt down the two vaguely Unix-literate folks who now had root, and had to have them type in all commands.

That lasted about two weeks.

After that, they just disallowed mounts to the /tmp directory and made the cleanup-script a little more intelligent.

There was also the guy who had an app lock up on his workstation -- during a flight -- and almost had to be physically restrained from hard rebooting the app's server. Of course, he was a user. He didn't have root access, so he went for the plug.

[Yegolev]

I am really glad I work for a regimented bureaucracy when I read stuff like that.  I won't act like our environment is tight like a military operation, but we do have plenty of regulations for preventing ding-dongs from doing bad things, or at least providing us some CYA in that we are not responsible for certain things... such as security, which includes all user administration.  Hard to explain to people that I am not supposed to go around resetting passwords or handing out server access like candy even though I have root.  Besides that, such things are beneath me; we have robots to reset passwords.

[Morat20]

I am really glad I work for a regimented bureaucracy when I read stuff like that.  I won't act like our environment is tight like a military operation, but we do have plenty of regulations for preventing ding-dongs from doing bad things, or at least providing us some CYA in that we are not responsible for certain things... such as security, which includes all user administration.  Hard to explain to people that I am not supposed to go around resetting passwords or handing out server access like candy even though I have root.  Besides that, such things are beneath me; we have robots to reset passwords.

Oh, we have that. We just have people who don't understand it. They don't understand hardware lockdowns, software lockdowns, and most don't even grasp the concept of "administrative privaleges". I spend most of my support time (about a week every three months) either explaining how Word works, or explaining that I'm there to support our software and cannot reset their laptop/network/workstation passwords. Oh, and saying things like "I don't care how badly you need to get to work. That laptop hasn't been virus-scanned in two years. Come back with an up-to-date definition file and a completed full scan, then we'll let you plug in".

Major-grade screwups like I mentioned above are fairly rare. Mostly it's users plugging in unvetted laptops, pestering the wrong person for software support, or being totally unable to remember to lock their workstation if they're getting up.

[Yegolev]

Ah, yes, your problem is that you are interfacing with the end user.  Truly a special hell.  I tip my hat.

[Morat20]

Ah, yes, your problem is that you are interfacing with the end user.  Truly a special hell.  I tip my hat.

I do try to avoid that. I keep getting asked whether I have an urges towards advancement. I keep saying "No, because then I'd have to go to meetings and write shit down and talk to customers. NO NO NO. I'll take any technical lead positions, yes. But if I have to write performance reviews, cajole customers, or attend more than two customer meetings a month I don't want it."