Useless Conversation

[Hammond]

Like Chimpy said it is probably network congestion.  Give us some more details about your setup and we might be able to steer you in the right direction.  Is the PBX onsite or remote.  Is it internal calls or external.  Do you use SIP provider or do you use old school PRI's.   Oh and just a fyi VOIP is a BLAST to troubleshoot :)

[Yegolev]

A question for the other IT peeps around here:

Say you are blindsided by management about an issue that is "pervasive and is costing the company money."  You ask for details, and management replies that it's random and they cannot give you more detail other than it appears to happen less than 5% of the time.  Noone is able to reliably reproduce the issue, so you ask to be contacted when the issue is happening to debug it on the spot.  Management refuses to do so, and tells you to either get it fixed or bring in an outside consultant to fix the problem.

What is the correct response to this situation?

Those other guys were waaaaay off.  You organize a task force, unless the amount of money being lost is greater than $100,000, in which case you organize a swat team.

While waiting on your team, force, or squad to come together, plan the kick-off meeting and create some PowerPoint slides with your situation and mission goal.

Once this is in place, schedule bi-weekly meetings (unless you pass the $100,000 mark, then weekly).  Schedule a monthly executive meeting to present either milestones or tollgates, at your discretion.

When you have this set up, hire a consultant to fix your system.

[Minvaren]

VOIP PBX onsite.
External SIP provider used (avg 7ms ping to them).
Appears to be external calls only.
We have real-time traffic monitoring.

VOIP call quality is all about the latency, and I suspect that it's either congestion at our border router or on the SIP provider's end to specific destinations.  Without any details of specific calls or times that it happens, though, I'm just playing a guessing game.  And my frustration is that they flat-out refuse to provide any details.

[Yegolev]

You're not going to move up in the organization that way.

[Paelos]

Tell them you contacted an auditor, and any errors that are less than 5% are not material.

[Hammond]

Since the issue is so vague I would do what Yegolev suggests :).  Most likely the consultant would point to the SIP provider after charging you a arm and leg though. 

[Lantyssa]

You're not going to move up in the organization that way.

Heh.

[Arthur_Parker]

Without any details of specific calls or times that it happens, though, I'm just playing a guessing game.  And my frustration is that they flat-out refuse to provide any details.

Can't you just ring them from an external line ~20 times till it happens to you?

Edit to be clear.  Not saying your approach is wrong, just that if someone wants to play silly buggers then I'd get them worrying that people are going to realise their 5% figure was pulled from their arse by appearing even more helpful than usual.

[MuffinMan]

Crazy freaking storm in Austin last night. Opened all the shades and could have had a lightning rave.

[schild]

Crazy freaking storm in Austin last night. Opened all the shades and could have had a lightning rave.

I didn't even wake up, but it has been described to me as ravey also.

Pretty sure I could sleep through anything though.

[Lantyssa]

The wind from it reached us at 3am.  For the next two hours we had gale-force winds.  Power was up and down (more down).

The rain is supposed to hit us between mid-morning and mid-afternoon, which everyone is predicting to be abysmal.  Considering half of Texas looked like a red blob on the radar map, I expect it will be.

[ghost]

Crazy freaking storm in Austin last night. Opened all the shades and could have had a lightning rave.

My older child, very oddly, slept through all of it. 

The younger one was freaked out though.

[Salamok]

Luckily no serving of insane hail in that one, still a pretty loud storm though.  Seemed like it went for 4 or 5 hours too.

[Yegolev]

Not saying your approach is wrong, just that if someone wants to play silly buggers then I'd get them worrying that people are going to realise their 5% figure was pulled from their arse by appearing even more helpful than usual.

A more serious answer from me, still related to the previous, would be to do something like this.  Minvaren says they have logging, and if it's not granular enough then he can set up some test calls and one-off monitoring.  Generate the data, plop it into Excel (YES I SAID EXCEL) and make some graphs/charts from it.  Firstly, it might reveal the actual issue; secondly, it can demonstrate to the execs that they are imagining things.

[Johny Cee]

Not saying your approach is wrong, just that if someone wants to play silly buggers then I'd get them worrying that people are going to realise their 5% figure was pulled from their arse by appearing even more helpful than usual.

A more serious answer from me, still related to the previous, would be to do something like this.  Minvaren says they have logging, and if it's not granular enough then he can set up some test calls and one-off monitoring.  Generate the data, plop it into Excel (YES I SAID EXCEL) and make some graphs/charts from it.  Firstly, it might reveal the actual issue; secondly, it can demonstrate to the execs that they are imagining things.

Excel can fix anything!

[Yegolev]

It can fix "Where did you get this data?" with "Right here, Sir!"

[Ironwood]

If it's a call centre, they should have a box that says 'call cut off' or 'Bad connection'.

Generate the report from there for fucks sake.

[ghost]

All you Houston people are getting ready to get the smackdown from this storm.

[Minvaren]

We're running Asterisk here because management refuses to pony up the money for a call-center grade phone system.  Believe it or not, this was actually an upgrade for us, because the previous PBX hadn't been supported or parts made for it in over 5 years, and we were one critical hardware failure away from (essentially) closing the doors.

Likewise, we can do real-time traffic monitoring, but as there's no spending approval for software to actually log that data somewhere, all I can do is watch the graphs and charts live.  I've cranked the logging on the Asterisk box up so high that it's generating up to 1GB of logs daily and found 1-2 minor issues that have been corrected, and a couple that are pending to be corrected.  Lantyssa pointed out an error I made in QoS level translations when replacing the border firewall/routers, and that's about all that's been come up with in a week of trying to track this down.

Another complication regarding test calls is that we are a collection agency - ringing any number that's related to a collection attempt multiple times daily is begging for a lawsuit, so there are only limited avenues to test.  All inquiries along these lines (conference calls when the issue has happened, etc.) have been ignored when I have asked for specifics.

In the end, this was more about venting than anything.  The place I work for has a history of coming and saying, "we've had this problem for months that we've never told you about, we know you don't have any time or resources, and we're not going to give you any details, but you need to fix it immediately."  This is another example of this, and along with a litany of other issues (not allowed to take vacation, failure to deliver on promises, underpayment, verbal abuse, and so forth) is why I gave notice on Monday.

That said, with the storm coming through today, I'm curious to see if the wind breaks the last intact 2" of the power distribution pole to the building.  With any luck, it should fall over onto the water chillers and cause one hell of an explosion.

[Hammond]

Collection agency call center?  That sounds like close to the bowels of hell to me.  Do you have something else lined up or did you just say fuck it and give notice?

On a side note you do not need to pay for real-time traffic monitoring there are programs like mrtg, zenoss that all do logging for you and are open source.  For alarms you can use nagios. As long as you have hardware of some kind you are good to go.  Both nagios and mrtg run with no overhead and can be setup on a low end desktop.  If you add a loghost on the box (simple with linux) you have a all in one network monitoring, alarm, logging server.  While not the best in the world to maintain mostly because it is all command line except for the pretty graphs it is hard to beat free. 

[Yegolev]

Don't be talkin shit bout command line.

[Hammond]

Pshh I do command line all day.  I was thinking more of once you set it up can you pawn it off on help desk types to maintain.

[Lantyssa]

We are the help desk types.  It's a tiny shop.

[rattran]

How many times do I have to tell the collection agency that calls my landline that there's no Carlos here, I don't know any Carlos, stop fucking calling me before I get big money? Because if it's less than 5 times a week for the last year, I should have big money. At least when it was ComEd calling for Carlos, it only took a couple calls to get them to apologize and never call again.

[Der Helm]

How many times do I have to tell the collection agency that calls my landline that there's no Carlos here, I don't know any Carlos, stop fucking calling me before I get big money? Because if it's less than 5 times a week for the last year, I should have big money. At least when it was ComEd calling for Carlos, it only took a couple calls to get them to apologize and never call again.

Do you answer the phone with a Spanish accent ?

[Johny Cee]

How many times do I have to tell the collection agency that calls my landline that there's no Carlos here, I don't know any Carlos, stop fucking calling me before I get big money? Because if it's less than 5 times a week for the last year, I should have big money. At least when it was ComEd calling for Carlos, it only took a couple calls to get them to apologize and never call again.

"Son of a bitch, did you really think you could elude us forever, Carlos?"

[Minvaren]

How many times do I have to tell the collection agency that calls my landline that there's no Carlos here...

Call them back and ask for a supervisor from the phone # in question (to show them the caller ID).  That will generally get it handled.  If not, log every time they call for a week or two, then give THAT information to a(nother) supervisor and state you'll sue for harassment if they don't stop.  If they don't stop then, find an FDCPA lawsuit-chaser and get some money out of the fools.

Collection agency call center?  That sounds like close to the bowels of hell to me.  Do you have something else lined up or did you just say fuck it and give notice?

You have no idea.  And nothing lined up at present, but I saved up a few years of expenses anticipating something like this.

[Salamok]

Oddly enough when collections people call me I always end up with poor call quality as well.  Which brings up the point of how can you not just pawn this off on the phone service of the people who you are calling?

[Sky]

We are the help desk types.  It's a tiny shop.

Had a webinar with a vendor a couple weeks ago for my current big project. He asked when the rest of the team would log in. "Yep, we're all here!" (me)

Sucks not having someone to at least bounce ideas off. Or blame.

[RhyssaFireheart]

Oh holy mother of god.

I'm going in circles here trying to get the proper access to this fucking SAP system.  How many times do I have to tell them I need to be able to create PO requisitions in the system.  "You have all the access your role was mapped for."  No, no I obviously fucking don't!!!!  I could create PO requisitions in the old system, and no one consulted with our group about the role mapping, so I wasn't mapped properly, how do we fix that?  "Talk to your Training Coordinator"  And who the hell is that?  No one can tell me who that is!!  Finally had someone leverage some contacts and was sent a name to contact, a role mapping catalog, and what training courses I need to take (of which I've taken 1/2 and will take the second, even though I really don't need that ability).  So I contact name I was given and he responds by forwarding his message along to two people I've already tried working with in the past and who've done nothing beyond "have you called the held desk?"

ARGH!! 

[Hammond]

Yea I feel you on the tiny shop.  My current job is me and 1 other admin.  Job is really good though and I do not have to work any overtime so I can put up with the help desk questions.  That and we buy any kind of hardware we want....  They hired me because the company is doing more and more computer related projects. (we are a manufacturer and have a in house engineering / design department)  

[Merusk]

You're playing it all wrong

"You have all the access your role was mapped for."

"Oh, great!  Take a look in the system and let me know who's in charge of POs now so I can drop all this on their desk, please."

[RhyssaFireheart]

You're playing it all wrong

"You have all the access your role was mapped for."

"Oh, great!  Take a look in the system and let me know who's in charge of POs now so I can drop all this on their desk, please."

Semi-tried that already.  Gave one PO that we've been trying to get entered since the start of the month to the group that used to do POs before I came (they're another IT group) and not only did it take the guy 3 days before he looked at what I sent him, but now he's got to enter it into the new system (apparently, he had no trouble getting access!) and there is info I need to find out about the project to put into the new system.  Okie-dokie!  Let me ask Project Accounting for this info.  No response yet.  /cry.  And now I have one other req ready to go in and at least 2-3 others on hold waiting to get signatures, so I need to get this access.  /cries more.

[Soln]

Crazy freaking storm in Austin last night. Opened all the shades and could have had a lightning rave.

I arrived here last night from Seattle.  Go figure it would rain.

"Hail in Austin!   Hide yo cars."

[WayAbvPar]

http://www.reuters.com/article/2012/01/25/us-weather-idUSTRE80O1GB20120125

Apparently there was actually a tornado tucked into all that fun. I can't believe it didn't hit schild's car.