Woe's of an ISP user

[fuser]

http://www.getdropbox.com/gallery/109162/1/public?h=5fd7c4  (can someone show me how to embed a dropbox image)

This is whats happening to me pretty much on a prime time nightly basis now (When I run WoW + ventrillo and nothing else).

 ISP so far has blamed my router, torrents, virus, my pc, my network card, ventrillo, and probably "sun spots". I have been dealing with this latency for a month (pulled my router out, power cycled modem, etc), and I have tried to escalated the issue to higher tiers with no response. I have only ran into one tech who said "this is a fscking weird issue man, but sorry I cannot even check modems in the same area to see if they are having issues".

 Oh well after complaining to tech support with no resolution, I have finally given up and ordered service from the only other provider in town. My cable co has tried to retain me with free service but wtf am I suppose to do with a non functioning service that's "free". I had a long talk with the sales rep and basically got them to note on my account "Your communication channels between your tiers of support has failed, and as a long time customer I am so disgusted I have disconnected my cable + phone + internet from your service." Told her to take it to her supervisor as a lost client that could of been retained if your tech dept wasn't so crippled.

 What other actions can I take, I'm debating offering my cable ISP to bring in a box or something if they really want to track the issue down.

[SnakeCharmer]

If you can get it, write a long, detailed email to the president of the local branch of your cable/internet company.  When I bought my first house about 8 years ago, I ran into problem after problem after problem with just getting the cable turned on, much less getting a good signal in a neighborhood that was only 10 years old at the time.  Anyway, I wrote this 8 page monstrosity and sent it, not expecting a response.  It was more of a therapeutic exercise to be honest. 

Long story short, I got a call 30 minutes later and it was the local general manager who was extremely polite, well mannered.  Asked when I would be home (not that they would be there between 12 and 5 or whatever) and that it would be taken care of.  I tell him 530 pm, and actually get home about 445.  When I pull up, there's two of the big Comcast trucks, one smaller one, a Comcast van, and a Mercedes parked in front of my house.  I had 3 cable techs, one internet tech, and the general manager to oversee everything.  They did a complete rewire/install in about one hour.  Also got 50 percent off my cable/internet bill for 9 months.

Moral of the long story:  Get your issue into the hands of a person that is really too high up to be dealing with it and you'll get results. 

Edit to add:  If you use Comcast, as part of your service agreement, they are required to completely replace the cable/coaxial cable in your house once a year (it may be once every two years).  Most people don't know that and think they have to pay for it themselves.  But it's part of the monthly maintenance agreement you pay for.

[Salamok]

what a coincidence I just got my cable internet speed issues resolved after 6 months of putting up with 1-1.5mbps and sporatic performance.  The connection was so bad they couldn't even remotely check to see if my modem was up. 

Anyway they sent a guy out and here is what he found:

1 - Cable modem was going bad, replaced it.
2 - Splitter I was using (5-way) should have been a 3-way. Apparently even unconnected splits on a 5 way splitter cause extra signal loss.
3 - The connections on the original splitter they used on the exterior of the home were oxidized (apparently some watter got in).
4 - Cable from the demarc to my house was faulty (chewed on by squirrels or something), replaced it.
5 - 1 of the ends on the cable between the exterior connection and interior connection was hanging by a thread.
6 - My linksys router (wrv54g) was hosed.  bought a new router.

Basically anything that could fail between myself and the demarc did fail.   Cable guy was here like 3 hours, he was pretty nice but you could tell he was getting a little annoyed when each time he thought he absolutely fixed the problem there was another problem waiting behind it.

[Soln]

you should ask your provider to put in a repeater into your home.  It in effect acts as an amp to the signal.  If you are far from the drop it should help improve performance (it did for my last place).

[fuser]

you should ask your provider to put in a repeater into your home.  It in effect acts as an amp to the signal.  If you are far from the drop it should help improve performance (it did for my last place).

Problem is, the issue is upstream in their routers.

24.89.228.1 is a node past my cable modem on the other side of their distribution node. The issue is past this node and to their core routers for the city :(

[Trippy]

Back in the good old days (get off my lawn!) when Internet routing was more or less symmetric, a "one-way" traceroute could be used for figuring out where the bottlenecks were. These days with asymmetric routing that's not the case. I.e. where you see the red in your traceroute is not necessarily the router where the problem is.

E.g. to give a real simple example let's say you are "A" and your path toC from your side looks like:

A -> B -> C

and C looks red. However from C the path to A is:

C -> E -> A

So the problem could actually be at E and not C because the traceroute path from A to C is actually going A -> B -> C -> E -> A.

In your picture it could be that it's actually telus.com's fault and the red you see on the eastlink hops is actually packets through telus.com taking a different return route. I.e. again because of asymmetric routing the red you see isn't necessarily the router where that's happening, and there could be, for example, a problem at some eastlink.ca and telus.com interchange you don't even see or even some totally different NSP you don't see on your side of the traceroute. Unfortunately it looks like eastlink.ca is blocking traceroute from going too far into their network which makes it difficult to look at their network from the other direction.

Here's an actual example. This is the traceroute in both directions from my server to f13.net (with my server taken out):

From f13.net

Code:

 1  ip-72-167-43-251.ip.secureserver.net (72.167.43.251)  0.441 ms  0.518 ms  0.604 ms
 2  ip-208-109-113-198.ip.secureserver.net (208.109.113.198)  1.500 ms  1.563 ms  1.617 ms
 3  ip-208-109-113-173.ip.secureserver.net (208.109.113.173)  1.425 ms  1.478 ms  1.532 ms
 4  ip-208-109-113-165.ip.secureserver.net (208.109.113.165)  1.404 ms  1.473 ms  1.497 ms
 5  ip-208-109-113-158.ip.secureserver.net (208.109.113.158)  1.420 ms  1.488 ms  1.515 ms
 6  ip-208-109-112-162.ip.secureserver.net (208.109.112.162)  1.347 ms  1.336 ms  1.403 ms
 7  ip-208-109-112-138.ip.secureserver.net (208.109.112.138)  1.484 ms  1.551 ms  1.635 ms
 8  te-3-1.car1.Phoenix1.Level3.net (4.53.104.1)  1.521 ms  1.573 ms  1.644 ms
 9  ae-11-11.car2.Phoenix1.Level3.net (4.69.133.34)  1.434 ms  1.501 ms  1.564 ms
10  ae-4-4.ebr2.LosAngeles1.Level3.net (4.69.133.38)  20.357 ms  20.435 ms  20.426 ms
11  ae-72-72.csw2.LosAngeles1.Level3.net (4.69.137.22)  10.696 ms ae-82-82.csw3.LosAngeles1.Level3.net (4.69.137.26)  18.195 ms ae-92-92.csw4.LosAngeles1.Level3.net (4.69.137.30)  10.622 ms
12  ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33)  23.355 ms ae-73-73.ebr3.LosAngeles1.Level3.net (4.69.137.37)  20.059 ms ae-83-83.ebr3.LosAngeles1.Level3.net (4.69.137.41)  11.014 ms
13  ae-2.ebr3.SanJose1.Level3.net (4.69.132.9)  21.394 ms  21.368 ms  21.372 ms
14  ae-93-93.csw4.SanJose1.Level3.net (4.69.134.238)  23.622 ms ae-73-73.csw2.SanJose1.Level3.net (4.69.134.230)  26.512 ms ae-83-83.csw3.SanJose1.Level3.net (4.69.134.234)  18.861 ms
15  ae-14-69.car4.SanJose1.Level3.net (4.68.18.6)  19.227 ms ae-24-79.car4.SanJose1.Level3.net (4.68.18.70)  19.328 ms ae-34-89.car4.SanJose1.Level3.net (4.68.18.134)  19.434 ms
16  PEER-1-NETW.car4.SanJose1.Level3.net (4.71.114.50)  19.436 ms  19.610 ms  19.441 ms
17  * * *
18  65.39.248.100 (65.39.248.100)  45.730 ms  45.696 ms  45.665 ms

From my server

Code:

 2  sj-mkp2-dis-1.peer1.net (65.39.248.99)  1.187 ms  1.139 ms  1.293 ms
 3  10ge.ten1-1.sj-mkp16-dis-1.peer1.net (216.187.88.133)  1.126 ms  1.164 ms  0.617 ms
 4  ge4-0.mpr2.pao1.us.mfnx.net (198.32.176.11)  2.049 ms  19.325 ms  1.956 ms
 5  ge-4-0-0.mpr2.sjc2.us.above.net (64.125.31.70)  2.744 ms  3.001 ms  2.069 ms
 6  so-1-0-0.mpr4.lax9.us.above.net (64.125.26.29)  11.254 ms  11.395 ms  10.993 ms
 7  so-0-0-0.mpr3.lax9.us.above.net (64.125.26.145)  11.260 ms  11.305 ms  11.368 ms
 8  xe-1-1-0.mpr3.phx2.us.above.net (64.125.28.69)  19.258 ms  18.835 ms  19.622 ms
 9  64.124.196.38.allocated.above.net (64.124.196.38)  19.240 ms  20.068 ms  19.727 ms
10  ip-208-109-112-137.ip.secureserver.net (208.109.112.137)  21.493 ms  20.347 ms  20.788 ms
11  ip-208-109-112-161.ip.secureserver.net (208.109.112.161)  20.496 ms  20.587 ms  19.883 ms
12  ip-208-109-113-157.ip.secureserver.net (208.109.113.157)  20.516 ms  21.178 ms  20.826 ms
13  ip-208-109-113-166.ip.secureserver.net (208.109.113.166)  21.387 ms  21.030 ms  20.987 ms
14  ip-208-109-113-174.ip.secureserver.net (208.109.113.174)  20.119 ms  20.899 ms  25.591 ms
15  ip-72-167-43-94.ip.secureserver.net (72.167.43.94)  21.128 ms  20.881 ms  20.517 ms

Notice how the packets from my server to f13 use a completely different set of interchanges (peer.net -> above.net -> GoDaddy) than the f13.net to my server route (GoDaddy -> level3 -> peer.net). And even the router IP at peer.net right "in front" of my server is not the same on both routes.

So...

You first need isolate out the rest of the Internet beyond eastlink.ca to really see if it's a problem with your physical connection, an eastlink problem or a problem somewhere else.

Test traceroute and ping to as many internal eastlink.ca routers (like 24.222.79.205) as you can and also do the same from a neighbors house (you are friends with some of your neighbors right?). If you are dropping packets to just an eastlink router but your neighbor isn't you could have a physical wire problem or your router/modem may be hosed. If neither of you are dropping packets as far down the eastlink chain as you can go then the problem may be at one of their interchanges or somewhere else that's "invisible" to you entirely.

Because of the traceroute blockage going into the eastlink you might have to get their tech support involved again and have somebody who can get on a machine closer to the site(s) you are having trouble with (e.g. a machine that's directly off the 24.222.79.205 router). From there they can run traceroute and ping to see how it looks which will remove your side of the connection from the equation.

[fuser]

So the problem could actually be at E and not C because the traceroute path from A to C is actually going A -> B -> C -> E -> A.

I'm thinking along these lines because 24.89.228.1 -> br1 is suppose to be routing to a different ip asr1 (which is *'d out in the pic) I just noticed now that it captured the route change which really shouldn't be happening in the network. I can get it to route out "known" peering points (local university, datacenter connected to it) but the lag still is present on the internal ISP routes.

In your picture it could be that it's actually telus.com's fault and the red you see on the eastlink hops is actually packets through telus.com taking a different return route. I.e. again because of asymmetric routing the red you see isn't necessarily the router where that's happening, and there could be, for example, a problem at some eastlink.ca and telus.com interchange you don't even see or even some totally different NSP you don't see on your side of the traceroute. Unfortunately it looks like eastlink.ca is blocking traceroute from going too far into their network which makes it difficult to look at their network from the other direction.

What your seeing there is the software changing the old route out to a * I can verify its not an ICMP drop but the old hostname changing on the route moving to the new BR1 for that one echo request.

Well I have setup MTR to run nightly to asr-1 (before telus) inside eastlink's network. MTR keeps reporting the dropped packets/latency past 24.89.228.1 when it tries to reach ASR1/BR1.

So...

You first need isolate out the rest of the Internet beyond eastlink.ca to really see if it's a problem with your physical connection, an eastlink problem or a problem somewhere else.

I had a nice looong post written out but I figured its too blogish and a wall of text. But basically I have done mtr traces to google, asr1, and past my router. it all points at the internal linkage between the isp's router.

Because of the traceroute blockage going into the eastlink you might have to get their tech support involved again and have somebody who can get on a machine closer to the site(s) you are having trouble with (e.g. a machine that's directly off the 24.222.79.205 router). From there they can run traceroute and ping to see how it looks which will remove your side of the connection from the equation.

From their NOC the tier1 techs have seen the latency but their only recommendation is pulling the plug on the modem for a few hours. :(

Thanks for your post trippy, gave me some idea's to prove I'm not 100% insane(I'll run primetime logging on some other hosts for fun tonight and post em later) but really after a month into this issue speaking with multiple techs/supervisors and offering to send mtr/d3tr (nice little program) logs which were rejected because "we don't support that program" or support Linux, I'm at wits end.

[bhodi]

Yeah, you really need to get a hold with someone in their network architecture team. There's really nothing you can do but prove that it's in their network, and prove it to the right person.

[Sky]

Woes.