[SAGE] So you think you know traceroute...
Most network engineers and sysadmins would probably say that they're
intimately familiar with 'traceroute<http://en.wikipedia.org/wiki/Traceroute>',
and consider it one of their fundamental network troubleshooting tools... I
certainly do. But you might be amazed to learn, as I did, how much you *
don't* know about traceroute.
Richard Steenbergen of nLayer Communications, Inc., did an *excellent*
presentation
on traceroute<http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf>
at
this month's NANOG <http://www.nanog.org/> (North American Network Operators
Group) meeting:
- A Practical Guide to (Correctly) Troubleshooting with
Traceroute<http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf>,
by Richard A.
Steenbergen<http://www.linkedin.com/pub/richard-steenbergen/0/8b/398>
of nLayer Communications <http://www.nlayer.net/>.
Among other things, this presentation shows you:
- How traceroute works
- What you can learn from the DNS hostnames returned by traceroute
- Where the ISP/carrier boundaries are
- Where the equipment is located, geographically (do you know what a
CLLI code is?)
- What type of equipment the ISP/carrier is using
- What the round trip times reported by traceroute *really* mean
- How you can be led astray by ICMP prioritization, rate limiting,
asymmetric paths, and load balancing
One of the coolest tricks I learned from this presentation is, to find out
more about what's at the other end of some hop that appears to be a
point-to-point link, assume that the IP address you see is one of the two
addresses in a /30 subnet (as is commonly assigned to point-to-point links),
and do a DNS reverse lookup of the other address in the /30.
This is useful, for example, in figuring out which egress port a packet went
out on, since traceroute normally only shows you the ingress ports for each
device along the way. For example, let's say I was looking at the following
traceroute output, and wanted to know the egress port on router #3, as the
packet moved to router #4:
*brent%* traceroute www.google.com
traceroute: Warning: www.google.com has multiple addresses; using 208.67.219.230
traceroute to google.navigation.opendns.com (208.67.219.230), 64 hops
max, 40 byte packets
1 192.168.0.1 (192.168.0.1) 3.145 ms 2.573 ms 2.382 ms
2 75-101-29-1.dsl.static.sonic.net (75.101.29.1) 9.555 ms 9.054 ms 9.089 ms
3 127.at-X-X-X.gw3.200p-sf.sonic.net (208.106.96.193) 9.510 ms
9.871 ms 9.194 ms
4 200.ge-0-1-0.gw.equinix-sj.sonic.net (64.142.0.210) 11.965 ms
11.870 ms 11.839 ms
5 0.as0.gw2.equinix-sj.sonic.net (64.142.0.150) 11.928 ms 12.519
ms 12.394 ms
6 GigabitEthernet3-1.GW2.SJC7.ALTER.NET (157.130.194.17) 11.360 ms
16.257 ms 11.268 ms
7 0.so-0-0-1.XL4.SJC7.ALTER.NET (152.63.51.50) 11.729 ms 11.679 ms
11.403 ms
8 0.so-7-0-0.XL2.PAO1.ALTER.NET (152.63.113.21) 14.775 ms 17.455
ms 0.so-5-0-0.XL2.PAO1.ALTER.NET (152.63.48.9) 15.548 ms
9 POS7-0.GW6.PAO1.ALTER.NET (152.63.55.14) 12.886 ms 13.143 ms 13.029 ms
10 65.203.37.46 (65.203.37.46) 13.517 ms 14.708 ms 16.566 ms
11 * * *
12 * * *
^C
To find out more about router #3's egress port, I look at the IP address for
router #4 (64.142.0.210), figure out what would be the other IP address in
the same /30 (64.142.0.209; hint: the lower address in a /30 pair always
ends in an odd number, and the higher address always ends in an even number,
so if the address you know ends in an odd number, the other address in the
same /30 is going to be the next *higher* number, and if the address you
know is even, the other is going to be the next *lower* number), and do a
DNS reverse lookup of that address:
*brent%* dig -x 64.142.0.209
; <<>> DiG 9.4.3-P3 <<>> -x 64.142.0.209
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49382
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;209.0.142.64.in-addr.arpa. IN PTR
;; ANSWER SECTION:
209.0.142.64.in-addr.arpa. 259200 IN PTR 200.ge-6-3-0.gw3.200p-sf.sonic.net.
;; Query time: 31 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Nov 13 09:42:05 2009
;; MSG SIZE rcvd: 91
Another handy tip from the presentation is that, since light travels through
fiber optic cable at about 200 km (or 125 miles, if you prefer) per
millisecond, each 1 ms of delay shown by traceroute (which, remember, is*round
trip* delay) should represent about 100 km (62.5 mi) of distance if the
delay were due entirely to the distance travelled (i.e., no queuing or
processing delays). Using that fact, you can see that 40ms for a packet to
go from San Francisco to New York (about 2500 miles, or 4000km) would be
"normal", but 40ms for a packet to go from San Francisco to San Jose (about
50 miles, or 80km) would indicate a problem; it should take the packet less
than 1ms to cover that distance and back, so something else (congestion or
processing delays, for example) must account for the other 39ms.
There's a *lot* more in this presentation, about more complex issues such as
- how the way in which routers handle traceroute packets can produce
biased results (most routers handle traceroute packets much more slowly than
they handle "real" data packets, which can make things look much worse than
they are)
- how asymmetric paths can lead you astray (traceroute only shows you the
path *to* a system, but if you're pulling lots of bytes from the system,
as would typically be the case with a remote server, you probably care more
about the path back *from* the system, which might be totally different
- how using MPLS, which is increasingly common in carrier networks, can
lead to very confusing round-trip times in traceroute
Anyway, if you ever use traceroute, I highly recommend that you review this
excellent presentation. I think you'll be pleasantly surprised at how much
you learn.
Thanks to Strata Chalup of
Virtual.net<http://www.linkedin.com/in/virtualnet> for
bringing this very informative presentation to my attention.
-Brent
--
Brent Chapman <brent@xxxxxxxxxxxx>
Netomata, Inc. -- www.netomata.com
Making networks more cost-effective, reliable, and flexible by automating
network configuration
This archive was generated by a fusion of
Pipermail (Mailman edition) and
MHonArc.