High latency between AT&T and Level3 in Chicago - AT&T U-verse (2024)

kwayzcat

Member

2018-Jun-4 9:13 am

I've noticed my pings have gone up in everything. Overwatch and other gaming too

It appears to be an issue between AT&T and level3...

Can anyone escalate to the ATT NOC?

TRACEROUTE:
traceroute to xx (xx), 15 hops max, 60 byte packets
1 10.56.70.56 (10.56.70.56) 0.377 ms 0.334 ms 0.320 ms
2 37.244.22.3 (37.244.22.3) 0.961 ms 0.971 ms 0.972 ms
3 24.105.30.162 (24.105.30.162) 0.990 ms 1.025 ms 1.084 ms
4 137.221.66.6 (137.221.66.6) 2.364 ms 2.397 ms 2.398 ms
5 137.221.68.68 (137.221.68.68) 2.357 ms 2.379 ms 2.381 ms
6 137.221.68.32 (137.221.68.32) 2.324 ms 1.526 ms 1.490 ms
7 xe-9-3-0.edge2.LosAngeles9.Level3.net (4.53.230.237) 1.452 ms 1.497 ms 1.495 ms
8 * * *
9 192.205.37.145 (192.205.37.145) 5.555 ms 5.584 ms 5.080 ms
10 cr1.la2ca.ip.att.net (12.122.128.102) 51.501 ms 51.512 ms 51.513 ms
11 slkut21crs.ip.att.net (12.122.1.186) 49.926 ms 48.727 ms 48.707 ms
12 cr2.dvmco.ip.att.net (12.122.28.45) 52.154 ms 52.419 ms 52.407 ms
13 cr1.kc9mo.ip.att.net (12.122.28.78) 53.096 ms 49.855 ms 49.838 ms
14 12.123.159.101 (12.123.159.101) 48.515 ms 51.544 ms 51.528 ms
15 * * *

04/06/2018 13:08:20 UTC
--------------------

TRACEROUTE:
traceroute to xx (xx), 15 hops max, 60 byte packets
1 10.57.68.13 (10.57.68.13) 0.311 ms 0.307 ms 0.306 ms
2 37.244.23.2 (37.244.23.2) 1.016 ms 1.032 ms 1.033 ms
3 24.105.62.150 (24.105.62.150) 1.229 ms 2.823 ms 2.840 ms
4 137.221.66.14 (137.221.66.14) 2.859 ms 2.859 ms 2.860 ms
5 137.221.69.68 (137.221.69.68) 2.837 ms 2.838 ms 2.838 ms
6 137.221.69.32 (137.221.69.32) 2.823 ms 1.847 ms 1.849 ms
7 xe-11-3-0.edge4.Chicago3.Level3.net (4.53.98.45) 1.498 ms 1.149 ms 1.127 ms
8 * * *
9 192.205.32.209 (192.205.32.209) 56.865 ms 56.877 ms 56.875 ms
10 cr1.sffca.ip.att.net (12.122.114.6) 49.980 ms 49.996 ms 53.610 ms
11 cgcil21crs.ip.att.net (12.122.1.173) 55.522 ms 54.356 ms 54.332 ms
12 12.123.159.101 (12.123.159.101) 50.501 ms 50.509 ms 50.504 ms
13 * * *
14 71.145.64.161 (71.145.64.161) 49.758 ms 50.056 ms 50.393 ms
15 71.151.17.35 (71.151.17.35) 51.176 ms 50.753 ms 50.927 ms

04/06/2018 13:08:20 UTC
--------------------

MTR:
Start: Mon Jun 4 13:08:20 2018
HOST: Blizzard Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.56.70.56 0.0% 10 0.3 0.3 0.2 1.0 0.0
2.|-- 37.244.22.3 0.0% 10 0.5 0.5 0.4 1.0 0.0
3.|-- 24.105.30.162 0.0% 10 0.8 1.1 0.8 2.2 0.3
4.|-- 137.221.66.6 0.0% 10 1.3 1.4 1.2 1.6 0.0
5.|-- 137.221.68.68 0.0% 10 1.4 1.5 1.3 2.3 0.0
6.|-- 137.221.68.32 0.0% 10 1.2 2.2 1.1 10.7 3.0
7.|-- xe-9-3-0.edge2.LosAngeles9.Level3.net 0.0% 10 1.5 1.1 1.0 1.5 0.0
8.|-- ae-1-3501.ear3.Tustin1.Level3.net 80.0% 10 527.4 575.3 527.4 623.2 67.7
9.|-- 192.205.37.145 0.0% 10 5.7 6.0 3.9 11.3 2.1
10.|-- cr1.la2ca.ip.att.net 0.0% 10 49.9 51.3 48.7 56.0 2.4
11.|-- slkut21crs.ip.att.net 0.0% 10 48.2 49.5 48.2 51.8 1.2
12.|-- cr2.dvmco.ip.att.net 0.0% 10 52.7 53.3 48.2 62.9 3.8
13.|-- cr1.kc9mo.ip.att.net 0.0% 10 54.8 51.6 48.6 60.7 4.0
14.|-- 12.123.159.101 0.0% 10 49.7 49.9 47.8 58.2 3.0
15.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
16.|-- 71.145.64.161 90.0% 10 48.2 48.2 48.2 48.2 0.0
17.|-- 71.151.17.35 0.0% 10 48.9 49.0 48.9 50.1 0.3
18.|-- xx.lightspeed.cicril.sbcglobal.net 0.0% 10 70.7 70.5 68.8 74.5 1.6

04/06/2018 13:08:20 UTC
--------------------

MTR:
Start: Mon Jun 4 13:08:20 2018
HOST: Blizzard Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.57.100.13 0.0% 10 0.3 0.3 0.2 0.6 0.0
2.|-- 37.244.23.130 0.0% 10 0.6 0.6 0.5 1.3 0.0
3.|-- 24.105.62.150 0.0% 10 1.7 1.1 0.8 1.8 0.0
4.|-- 137.221.66.14 0.0% 10 2.1 2.2 1.9 2.8 0.0
5.|-- 137.221.69.68 0.0% 10 1.9 1.9 1.8 2.0 0.0
6.|-- 137.221.69.32 0.0% 10 1.2 2.5 1.1 12.4 3.4
7.|-- xe-11-3-0.edge4.Chicago3.Level3.net 0.0% 10 1.2 1.2 1.1 1.6 0.0
8.|-- ae-24-24.ebr3.Atlanta2.Level3.net 70.0% 10 50.4 50.5 50.4 50.7 0.0
9.|-- 192.205.32.209 0.0% 10 55.3 54.3 53.1 55.4 0.7
10.|-- cr1.sffca.ip.att.net 0.0% 10 51.0 51.1 49.1 52.7 1.2
11.|-- cgcil21crs.ip.att.net 0.0% 10 56.8 54.1 51.0 56.8 1.9
12.|-- 12.123.159.101 0.0% 10 50.4 49.9 48.5 52.5 1.1
13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
14.|-- 71.145.64.161 90.0% 10 49.5 49.5 49.5 49.5 0.0
15.|-- 71.151.17.35 0.0% 10 52.1 50.6 50.2 52.1 0.5
16.|-- xx.lightspeed.cicril.sbcglobal.net 0.0% 10 70.5 71.8 70.5 73.9 0.9

04/06/2018 13:08:20 UTC
--------------------

here is my traceroute to 4.2.2.2 which used to be 20ms, not its 70+

traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets
1 xxxxx
2 108-89-16-1.lightspeed.cicril.sbcglobal.net (108.89.16.1) 28.811 ms 29.336 ms 29.428 ms
3 71.151.17.34 (71.151.17.34) 23.556 ms 23.795 ms 25.868 ms
4 * * *
5 12.83.43.9 (12.83.43.9) 26.606 ms 26.715 ms 26.348 ms
6 cgcil402igs.ip.att.net (12.122.133.161) 25.950 ms 25.376 ms 25.267 ms
7 * * *
8 b.resolvers.Level3.net (4.2.2.2) 70.966 ms 70.782 ms 70.941 ms

· actions · 2018-Jun-4 9:13 am ·

maartena

Premium Member

2018-Jun-4 10:15 am

Don't use traceroute to determine latency. Level3 puts ICMP traffic on a low priority, so responses from hops along the way may look slow, but that is by design to prevent DDoD attack from crippling their border routers.

Use PING to the end host to determine your latency to that end host, anything in between could just be ICMP QoS by Level3.

· actions · 2018-Jun-4 10:15 am ·

kwayzcat

Member

2018-Jun-4 10:31 am

· actions · 2018-Jun-4 10:31 am ·

Shadow01

Premium Member

2018-Jun-4 11:54 am

· actions · 2018-Jun-4 11:54 am ·

sean8102 to kwayzcat

Member

2018-Jun-5 7:54 pm

hmm maybe its a regional thing, I'am in Arkansas and did some tests. They aren't great but a little bit lower than what you are showing. I tried pinging 4.2.2.2 for a while and got this.

Ping statistics for 4.2.2.2:
Packets: Sent = 63, Received = 63, Lost = 0 (0% loss), round trip times in milli-seconds: Minimum = 30ms, Maximum = 33ms, Average = 31ms

I tried to ping 10.56.70.56 but it wouldn't respond to ping requests. 37.244.22.3 dose though and when I traceroute to that it is inside blizzards network. Here are my results for that. These tests were done at ~7PM so peak hours and I'm on 100/20 VDSL2 so the VDSL2 adds a tiny bit of latency.

Ping statistics for 37.244.22.3:
Packets: Sent = 138, Received = 138, Lost = 0 (0% loss),
round trip times in milli-seconds: Minimum = 62ms, Maximum = 68ms, Average = 62ms

· actions · 2018-Jun-5 7:54 pm ·

ssmithT
join:2011-06-28

ssmithT to kwayzcat

Member

2018-Jun-5 8:55 pm

· actions · 2018-Jun-5 8:55 pm ·

maartena to kwayzcat

Premium Member

2018-Jun-5 9:09 pm

West coast seems to be just fine...

Pinging 4.2.2.2 with 32 bytes of data:Reply from 4.2.2.2: bytes=32 time=4ms TTL=55Reply from 4.2.2.2: bytes=32 time=4ms TTL=55Reply from 4.2.2.2: bytes=32 time=4ms TTL=55Reply from 4.2.2.2: bytes=32 time=4ms TTL=55 Ping statistics for 4.2.2.2: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),Approximate round trip times in milli-seconds: Minimum = 4ms, Maximum = 4ms, Average = 4ms 

It may be helpful to know the IP addresses that are marked out as "xx", so other AT&T users can trace to the same host. I understand the need to mask your OWN IP address, but I never understood the need to mask target IP addresses that you have a hard time connecting to. As a community, we can't do much to help you nail down the issue if we don't have a chance to trace to the same hosts....

For the record: NO ONE here can "escalate to the NOC", unless an AT&T employee wishes to become public. And as soon as that person does so, they will likely be hounded by half the forum every time there is a grumpy ping. ;)

· actions · 2018-Jun-5 9:09 pm ·

Mr Fel

Premium Member

2018-Jun-5 9:26 pm

· actions · 2018-Jun-5 9:26 pm ·

rewtyh to kwayzcat

Member

2018-Jun-27 1:18 am

Experiencing the same issue in the chicago area. Latency doubled to colocated server in the same zip code.

»imgur.com/a/qxQK0GZ

 Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 217 0.4 0.3 0.3 0.5 0.0 2. X-X-X-X.lightspeed.cicril.sbcglobal.net 0.0% 217 21.0 21.4 17.6 48.6 4.2 3. 71.151.17.0 0.0% 217 17.9 19.3 17.6 40.4 2.7 4. 71.145.65.224 0.0% 217 21.2 20.2 18.2 47.5 3.2 5. 12.83.43.9 0.0% 217 19.5 21.6 17.9 83.5 4.8 6. cgcil403igs.ip.att.net 0.0% 217 22.0 21.9 18.7 38.7 2.8 7. ae16.cr7-chi1.ip4.gtt.net 0.0% 217 19.3 21.1 18.5 66.4 4.9 8. xe-0-0-2.cr2-chi1.ip4.gtt.net 0.0% 217 19.0 20.5 18.4 80.6 4.8 xe-5-3-2.cr2-chi1.ip4.gtt.net 9. as23352.chi12.ip4.gtt.net 0.0% 216 37.6 39.6 37.5 66.6 4.610. 0.ae4.cr1.ord6.scnet.net 0.0% 216 39.2 39.2 37.4 66.9 3.711. 71.ae1.ar1.ord6.us.scnet.net 0.0% 216 39.4 39.2 37.7 68.3 3.212. vlan-41.aggrfo181-1.ord6.scnet.net 0.0% 216 38.1 40.0 37.9 80.1 3.413. X.X.X.X 0.0% 216 37.6 38.7 37.4 67.6 2.5 Host Loss% Snt Last Avg Best Wrst StDev 1. 172.22.22.1 0.0% 84 0.2 0.1 0.1 0.2 0.0 2. 0.ge-0-0-1.aggrfo181-1.ord6.scnet.net 0.0% 84 3.6 1.7 0.5 13.6 2.0 3. 42.ge-5-1-5.ar2.ord6.scnet.net 0.0% 84 0.2 0.3 0.2 1.6 0.2 4. 71.ae2.cr1.ord6.scnet.net 0.0% 84 0.2 0.4 0.2 6.3 0.7 5. lag-150.ear4.Chicago2.Level3.net 14.3% 84 1.3 24.4 1.0 418.6 78.8 6. ??? 7. att-level3-te.washingtondc12.Level3.net 0.0% 84 24.1 23.4 21.6 25.5 1.1 8. cr2.wswdc.ip.att.net 0.0% 84 22.7 22.2 20.1 24.1 1.3 9. cgcil21crs.ip.att.net 0.0% 84 22.1 24.4 20.0 28.0 2.510. 12.123.159.101 0.0% 84 20.6 22.1 20.3 24.0 1.111. ???12. 71.145.65.225 91.6% 84 21.2 21.2 21.2 21.2 0.013. 71.151.17.1 0.0% 84 21.0 21.4 20.9 25.6 0.914. x-x-x-x.lightspeed.cicril.sbcglobal.net 0.0% 84 37.7 38.7 37.4 47.6 1.7 

· actions · 2018-Jun-27 1:18 am ·

Anon72ea8
@sbcglobal.net

Anon72ea8

Anon

2018-Jun-27 8:08 am

 Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 217 0.4 0.3 0.3 0.5 0.0 2. X-X-X-X.lightspeed.cicril.sbcglobal.net 0.0% 217 21.0 21.4 17.6 48.6 4.2 3. 71.151.17.0 0.0% 217 17.9 19.3 17.6 40.4 2.7 4. 71.145.65.224 0.0% 217 21.2 20.2 18.2 47.5 3.2 5. 12.83.43.9 0.0% 217 19.5 21.6 17.9 83.5 4.8 6. cgcil403igs.ip.att.net 0.0% 217 22.0 21.9 18.7 38.7 2.8 7. ae16.cr7-chi1.ip4.gtt.net 0.0% 217 19.3 21.1 18.5 66.4 4.9 8. xe-0-0-2.cr2-chi1.ip4.gtt.net 0.0% 217 19.0 20.5 18.4 80.6 4.8 xe-5-3-2.cr2-chi1.ip4.gtt.net 9. as23352.chi12.ip4.gtt.net 0.0% 216 37.6 39.6 37.5 66.6 4.610. 0.ae4.cr1.ord6.scnet.net 0.0% 216 39.2 39.2 37.4 66.9 3.711. 71.ae1.ar1.ord6.us.scnet.net 0.0% 216 39.4 39.2 37.7 68.3 3.212. vlan-41.aggrfo181-1.ord6.scnet.net 0.0% 216 38.1 40.0 37.9 80.1 3.413. X.X.X.X 0.0% 216 37.6 38.7 37.4 67.6 2.5 Host Loss% Snt Last Avg Best Wrst StDev 1. 172.22.22.1 0.0% 84 0.2 0.1 0.1 0.2 0.0 2. 0.ge-0-0-1.aggrfo181-1.ord6.scnet.net 0.0% 84 3.6 1.7 0.5 13.6 2.0 3. 42.ge-5-1-5.ar2.ord6.scnet.net 0.0% 84 0.2 0.3 0.2 1.6 0.2 4. 71.ae2.cr1.ord6.scnet.net 0.0% 84 0.2 0.4 0.2 6.3 0.7 5. lag-150.ear4.Chicago2.Level3.net 14.3% 84 1.3 24.4 1.0 418.6 78.8 6. ??? 7. att-level3-te.washingtondc12.Level3.net 0.0% 84 24.1 23.4 21.6 25.5 1.1 8. cr2.wswdc.ip.att.net 0.0% 84 22.7 22.2 20.1 24.1 1.3 9. cgcil21crs.ip.att.net 0.0% 84 22.1 24.4 20.0 28.0 2.510. 12.123.159.101 0.0% 84 20.6 22.1 20.3 24.0 1.111. ???12. 71.145.65.225 91.6% 84 21.2 21.2 21.2 21.2 0.013. 71.151.17.1 0.0% 84 21.0 21.4 20.9 25.6 0.914. x-x-x-x.lightspeed.cicril.sbcglobal.net 0.0% 84 37.7 38.7 37.4 47.6 1.7 

Can you please point out the issue and/or concern with this traceroute? Thank you

· actions · 2018-Jun-27 8:08 am ·

rewtyh to kwayzcat

Member

2018-Jun-27 3:20 pm

Latency increases by approximately 20ms, which is higher than expected, with hop 9 and 7 respectively. ICMP rate limiting does not appear to be causing this increase as all subsequent hops are showing increased latency. Below is a route to a server in the same zip code that does not use level3. This route has normal latency.

 Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 11 0.4 0.4 0.4 0.5 0.0 2. 108-67-X-X.lightspeed.cicril.sbcglobal.net 0.0% 11 20.5 19.9 17.9 22.7 1.6 3. 71.151.17.0 0.0% 11 18.2 18.7 17.7 23.3 1.7 4. 71.145.65.224 0.0% 11 19.0 20.5 18.3 28.3 3.7 5. 12.83.43.17 0.0% 11 19.7 19.6 18.1 21.9 1.2 6. cgcil403igs.ip.att.net 0.0% 11 19.5 21.9 18.5 27.2 2.6 7. chi-b21-link.telia.net 0.0% 11 21.9 23.0 19.4 45.8 7.8 8. serverhub-ic-324864-chi-b21.c.telia.net 0.0% 11 21.8 20.8 19.8 21.8 0.7 9. agg1.c13.r14.s101.chi03.singlehop.net 0.0% 10 33.2 40.6 33.2 71.7 11.310. aswg1.c26.r04.s101.chi03.singlehop.net 0.0% 10 21.7 20.8 19.4 27.0 2.311. X.60.40.X 0.0% 10 20.0 20.0 19.6 20.4 0.3 

· actions · 2018-Jun-27 3:20 pm ·

jhclce
join:2016-11-26

jhclce to rewtyh

Member

2018-Jun-27 4:14 pm

 Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 217 0.4 0.3 0.3 0.5 0.0 2. X-X-X-X.lightspeed.cicril.sbcglobal.net 0.0% 217 21.0 21.4 17.6 48.6 4.2 3. 71.151.17.0 0.0% 217 17.9 19.3 17.6 40.4 2.7 4. 71.145.65.224 0.0% 217 21.2 20.2 18.2 47.5 3.2 5. 12.83.43.9 0.0% 217 19.5 21.6 17.9 83.5 4.8 6. cgcil403igs.ip.att.net 0.0% 217 22.0 21.9 18.7 38.7 2.8 7. ae16.cr7-chi1.ip4.gtt.net 0.0% 217 19.3 21.1 18.5 66.4 4.9 8. xe-0-0-2.cr2-chi1.ip4.gtt.net 0.0% 217 19.0 20.5 18.4 80.6 4.8 xe-5-3-2.cr2-chi1.ip4.gtt.net 9. as23352.chi12.ip4.gtt.net 0.0% 216 37.6 39.6 37.5 66.6 4.610. 0.ae4.cr1.ord6.scnet.net 0.0% 216 39.2 39.2 37.4 66.9 3.711. 71.ae1.ar1.ord6.us.scnet.net 0.0% 216 39.4 39.2 37.7 68.3 3.212. vlan-41.aggrfo181-1.ord6.scnet.net 0.0% 216 38.1 40.0 37.9 80.1 3.413. X.X.X.X 0.0% 216 37.6 38.7 37.4 67.6 2.5 Host Loss% Snt Last Avg Best Wrst StDev 1. 172.22.22.1 0.0% 84 0.2 0.1 0.1 0.2 0.0 2. 0.ge-0-0-1.aggrfo181-1.ord6.scnet.net 0.0% 84 3.6 1.7 0.5 13.6 2.0 3. 42.ge-5-1-5.ar2.ord6.scnet.net 0.0% 84 0.2 0.3 0.2 1.6 0.2 4. 71.ae2.cr1.ord6.scnet.net 0.0% 84 0.2 0.4 0.2 6.3 0.7 5. lag-150.ear4.Chicago2.Level3.net 14.3% 84 1.3 24.4 1.0 418.6 78.8 6. ??? 7. att-level3-te.washingtondc12.Level3.net 0.0% 84 24.1 23.4 21.6 25.5 1.1 8. cr2.wswdc.ip.att.net 0.0% 84 22.7 22.2 20.1 24.1 1.3 9. cgcil21crs.ip.att.net 0.0% 84 22.1 24.4 20.0 28.0 2.510. 12.123.159.101 0.0% 84 20.6 22.1 20.3 24.0 1.111. ???12. 71.145.65.225 91.6% 84 21.2 21.2 21.2 21.2 0.013. 71.151.17.1 0.0% 84 21.0 21.4 20.9 25.6 0.914. x-x-x-x.lightspeed.cicril.sbcglobal.net 0.0% 84 37.7 38.7 37.4 47.6 1.7 

It looks to me like the problem is with scnet/server Central network/ AS23352. Where they interface with their upstreams, there is a 20ms latency jump. This appears with both GTT (in the forward traceroute) and level3 (in the reverse). I think the hop "9. as23352.chi12.ip4.gtt.net" where the latency increases on the forward traceroute is actually a "remote party demarc" (see this presentation https://www.nanog.org/sites/default/files/10_Roisman_Traceroute.pdf page 20/21) and is actually on the server central network. I don't see any real issues with AT&T or their network edges. Try complaining to server central.

Side note about the second (reverse) traceroute: there are some hops on AT&T and level3 that suggest a location of Washington DC based on their reverse DNS. However, these hops have minimal latency difference from neighboring Chicago hops, so it's extremely unlikely they're actually in DC. The reverse DNS is probably wrong on those hops.

· actions · 2018-Jun-27 4:14 pm ·

rewtyh to kwayzcat

Member

2018-Jun-27 4:59 pm

Here is a route that crosses as23352.chi12.ip4.gtt.net but does not show abnormal latency.
I suspect the latency at hop 9. as23352.chi12.ip4.gtt.net is caused by 7. att-level3-te.washingtondc12.Level3.net on the return path. I will reach out to servercentral.

HOST: wp100 Loss% Snt Last Avg Best Wrst StDev 1.|-- router1-nac.linode.com 0.0% 4 0.7 0.9 0.7 1.1 0.0 2.|-- 173.255.239.26 0.0% 4 0.7 1.4 0.5 3.6 1.3 3.|-- ae5.cr9-nyc3.ip4.gtt.net 0.0% 4 1.4 1.3 1.3 1.4 0.0 4.|-- xe-5-1-2.cr2-chi1.ip4.gtt.net 0.0% 4 19.7 18.1 17.5 19.7 1.0 5.|-- as23352.chi12.ip4.gtt.net 0.0% 4 20.6 20.6 20.6 20.6 0.0 6.|-- 0.ae11.cr1.ord6.scnet.net 0.0% 4 19.9 20.0 19.9 20.0 0.0 7.|-- 71.ae1.ar2.ord6.scnet.net 0.0% 4 20.0 20.5 20.0 21.9 0.6 8.|-- vlan-42.aggrfo181-1.ord6.scnet.net 0.0% 4 19.0 19.3 18.8 19.8 0.0 9.|-- 208.93.X.X 0.0% 4 18.1 18.1 18.1 18.2 0.0 

· actions · 2018-Jun-27 4:59 pm ·

rewtyh to kwayzcat

Member

2018-Jul-3 11:04 am

Servercentral stopped using level3 on this path. Latency is normal again to colocated server.

 Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 1398 0.3 0.3 0.3 3.0 0.1 2. X-X-X-X.lightspeed.cicril.sbcglobal.net 0.0% 1398 20.9 20.4 17.5 40.8 3.4 3. 71.151.17.0 0.0% 1398 17.8 18.2 17.5 26.0 1.1 4. 71.145.65.224 0.0% 1397 18.9 26.9 18.1 2496. 111.9 5. 12.83.43.9 0.0% 1397 22.1 20.4 18.1 44.1 1.6 6. cgcil403igs.ip.att.net 0.0% 1397 22.1 21.0 18.6 26.5 1.5 7. ae16.cr7-chi1.ip4.gtt.net 0.1% 1397 19.0 19.8 18.4 40.5 2.4 8. xe-4-2-2.cr2-chi1.ip4.gtt.net 0.0% 1397 19.9 19.7 18.7 41.8 2.1 9. as23352.chi12.ip4.gtt.net 0.0% 1397 19.4 20.2 18.7 61.6 4.810. 0.ae4.cr1.ord6.scnet.net 0.0% 1397 19.8 20.7 19.5 67.7 3.511. 71.ae1.ar1.ord6.us.scnet.net 0.0% 1397 19.5 20.0 19.2 39.6 1.812. vlan-41.aggrfo181-1.ord6.scnet.net 0.0% 1397 21.2 21.4 20.0 54.6 1.713. X.X.X.X 0.0% 1397 19.9 20.1 19.5 28.1 1.0 Host Loss% Snt Last Avg Best Wrst StDev 1. 172.22.22.1 0.0% 392 0.1 0.1 0.1 0.3 0.1 2. 0.ge-0-0-1.aggrfo181-1.ord6.scnet.net 0.0% 392 1.6 1.3 0.5 5.9 0.6 3. 42.ge-5-1-5.ar2.ord6.scnet.net 0.0% 392 0.4 0.3 0.2 8.0 0.6 4. 72.ae3.cr2.ord6.scnet.net 0.0% 392 0.3 0.8 0.2 44.8 3.9 5. ae-12.a02.chcgil09.us.bb.gin.ntt.net 0.0% 392 1.7 3.0 1.2 38.8 3.6 6. ae-3.r07.chcgil09.us.bb.gin.ntt.net 0.0% 392 1.4 1.4 1.2 2.0 0.1 7. ae-30.a00.chcgil09.us.bb.gin.ntt.net 0.0% 392 2.4 2.6 1.1 26.6 3.3 8. 192.205.32.193 0.0% 392 3.1 3.5 1.5 9.1 1.1 9. cr2.cgcil.ip.att.net 0.0% 392 3.7 6.0 2.1 10.5 2.310. 12.123.159.105 0.0% 391 5.3 4.1 2.2 6.5 1.111. ???12. 71.145.65.227 82.9% 391 3.4 3.7 3.2 10.8 1.213. 71.151.17.3 0.0% 391 3.2 70.1 2.9 5514. 518.714. X-X-X-X.lightspeed.cicril.sbcglobal.net 0.0% 391 19.9 20.5 19.5 35.9 1.3 

Latencies to other destinations that route between att and level3 still show abnormally high latency. They used to be approximately 20ms, now it's 40ms.

 Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 102 0.4 0.4 0.3 3.2 0.3 2. 108-67-X-1.lightspeed.cicril.sbcglobal.net 0.0% 102 18.2 21.0 17.8 37.8 4.0 3. 71.151.17.0 0.0% 102 21.4 18.7 17.6 28.0 1.4 4. 71.145.65.224 0.0% 102 18.5 19.4 18.2 26.1 1.4 5. 12.83.43.17 0.0% 102 23.3 20.3 18.1 23.3 1.2 6. cgcil402igs.ip.att.net 0.0% 101 21.7 21.2 18.7 24.3 1.4 7. ??? 8. b.resolvers.Level3.net 0.0% 101 37.0 37.3 36.6 40.5 0.8 Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 103 0.4 0.4 0.3 0.5 0.0 2. 108-67-X-1.lightspeed.cicril.sbcglobal.net 0.0% 103 20.8 21.2 17.7 35.7 4.2 3. 71.151.17.0 0.0% 102 18.7 19.8 17.6 60.6 4.5 4. 71.145.65.224 0.0% 102 18.9 19.7 18.2 26.0 1.6 5. 12.83.43.9 0.0% 102 19.9 20.6 18.1 24.3 1.4 6. cgcil402igs.ip.att.net 0.0% 102 23.0 20.9 18.5 30.3 1.6 7. ??? 8. ae-2-3607.ear4.Chicago2.Level3.net 74.5% 102 37.0 37.4 37.0 38.7 0.4 9. RCN-CORPORA.ear4.Chicago2.Level3.net 0.0% 102 37.4 37.6 36.8 41.1 0.610. bdle2.aggr2.chgo.il.rcn.net 0.0% 102 38.1 38.2 37.5 41.8 0.811. il.speedtest.rcn.net 0.0% 102 37.3 37.7 37.0 41.7 0.8 

If I tunnel traffic to my colocated server at servercentral, that no longer routes over level3, latencies are normal 20ms.

 Host Loss% Snt Last Avg Best Wrst StDev 1. 172.22.42.1 0.0% 44 20.3 20.0 19.6 20.9 0.2 2. 0.ge-0-0-1.aggrfo181-1.ord6.scnet.net 0.0% 44 21.1 21.4 20.1 24.4 0.9 3. 41.ge-5-1-5.ar1.ord6.us.scnet.net 0.0% 44 20.3 21.1 19.7 24.6 1.4 4. 72.ae1.cr2.ord6.scnet.net 0.0% 44 20.1 20.7 19.7 24.0 1.2 5. lag-155.ear4.Chicago2.Level3.net 0.0% 43 21.4 21.1 20.6 22.4 0.3 6. ae-16-3506.ear1.Chicago2.Level3.net 88.1% 43 21.0 21.4 21.0 21.9 0.4 7. b.resolvers.Level3.net 0.0% 43 21.5 21.6 20.6 25.1 1.1 1. 172.22.42.1 0.0% 32 20.9 20.1 19.7 23.6 0.7 2. 0.ge-0-0-1.aggrfo181-1.ord6.scnet.net 0.0% 32 20.5 22.1 20.3 25.1 1.4 3. 41.ge-5-1-5.ar1.ord6.us.scnet.net 0.0% 32 20.8 20.4 19.5 24.9 1.1 4. 72.ae1.cr2.ord6.scnet.net 0.0% 32 20.0 21.4 19.7 33.6 3.0 5. lag-155.ear4.Chicago2.Level3.net 0.0% 32 20.9 21.2 20.7 22.5 0.4 6. RCN-CORPORA.ear4.Chicago2.Level3.net 0.0% 31 22.5 22.1 21.0 25.5 1.3 7. bdle2.aggr2.chgo.il.rcn.net 0.0% 31 21.2 21.4 21.1 21.7 0.2 8. il.speedtest.rcn.net 0.0% 31 21.2 22.0 20.8 25.1 1.2 

· actions · 2018-Jul-3 11:04 am ·