Re: [SAGE] So you think you know traceroute...

This is an incredibly useful presentation. All the little bits and
pieces you can't learn from text books.

I've never been so excited about traceroute.

On Fri, Nov 13, 2009 at 1:56 PM, Brent Chapman <brent@xxxxxxxxxxxx> wrote:
> Most network engineers and sysadmins would probably say that they're
> intimately familiar with 'traceroute<http://en.wikipedia.org/wiki/Traceroute>',
> and consider it one of their fundamental network troubleshooting tools... I
> certainly do. But you might be amazed to learn, as I did, how much you *
> don't* know about traceroute.
>
> Richard Steenbergen of nLayer Communications, Inc., did an *excellent*
> presentation
> on traceroute<http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf>
> at
> this month's NANOG <http://www.nanog.org/> (North American Network Operators
> Group) meeting:
>
>   - A Practical Guide to (Correctly) Troubleshooting with
> Traceroute<http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf>,
>   by Richard A.
> Steenbergen<http://www.linkedin.com/pub/richard-steenbergen/0/8b/398>
>    of nLayer Communications <http://www.nlayer.net/>.
>
> Among other things, this presentation shows you:
>
>   - How traceroute works
>   - What you can learn from the DNS hostnames returned by traceroute
>      - Where the ISP/carrier boundaries are
>      - Where the equipment is located, geographically (do you know what a
>      CLLI code is?)
>      - What type of equipment the ISP/carrier is using
>   - What the round trip times reported by traceroute *really* mean
>   - How you can be led astray by ICMP prioritization, rate limiting,
>   asymmetric paths, and load balancing
>
> One of the coolest tricks I learned from this presentation is, to find out
> more about what's at the other end of some hop that appears to be a
> point-to-point link, assume that the IP address you see is one of the two
> addresses in a /30 subnet (as is commonly assigned to point-to-point links),
> and do a DNS reverse lookup of the other address in the /30.
>
> This is useful, for example, in figuring out which egress port a packet went
> out on, since traceroute normally only shows you the ingress ports for each
> device along the way. For example, let's say I was looking at the following
> traceroute output, and wanted to know the egress port on router #3, as the
> packet moved to router #4:
>
> *brent%* traceroute www.google.com
> traceroute: Warning: www.google.com has multiple addresses; using 208.67.219.230
> traceroute to google.navigation.opendns.com (208.67.219.230), 64 hops
> max, 40 byte packets
>  1  192.168.0.1 (192.168.0.1)  3.145 ms  2.573 ms  2.382 ms
>  2  75-101-29-1.dsl.static.sonic.net (75.101.29.1)  9.555 ms  9.054 ms  9.089 ms
>  3  127.at-X-X-X.gw3.200p-sf.sonic.net (208.106.96.193)  9.510 ms
> 9.871 ms  9.194 ms
>  4  200.ge-0-1-0.gw.equinix-sj.sonic.net (64.142.0.210)  11.965 ms
> 11.870 ms  11.839 ms
>  5  0.as0.gw2.equinix-sj.sonic.net (64.142.0.150)  11.928 ms  12.519
> ms  12.394 ms
>  6  GigabitEthernet3-1.GW2.SJC7.ALTER.NET (157.130.194.17)  11.360 ms
> 16.257 ms  11.268 ms
>  7  0.so-0-0-1.XL4.SJC7.ALTER.NET (152.63.51.50)  11.729 ms  11.679 ms
>  11.403 ms
>  8  0.so-7-0-0.XL2.PAO1.ALTER.NET (152.63.113.21)  14.775 ms  17.455
> ms 0.so-5-0-0.XL2.PAO1.ALTER.NET (152.63.48.9)  15.548 ms
>  9  POS7-0.GW6.PAO1.ALTER.NET (152.63.55.14)  12.886 ms  13.143 ms  13.029 ms
> 10  65.203.37.46 (65.203.37.46)  13.517 ms  14.708 ms  16.566 ms
> 11  * * *
> 12  * * *
> ^C
>
> To find out more about router #3's egress port, I look at the IP address for
> router #4 (64.142.0.210), figure out what would be the other IP address in
> the same /30 (64.142.0.209; hint: the lower address in a /30 pair always
> ends in an odd number, and the higher address always ends in an even number,
> so if the address you know ends in an odd number, the other address in the
> same /30 is going to be the next *higher* number, and if the address you
> know is even, the other is going to be the next *lower* number), and do a
> DNS reverse lookup of that address:
>
> *brent%* dig -x 64.142.0.209
>
> ; <<>> DiG 9.4.3-P3 <<>> -x 64.142.0.209
> ;; global options:  printcmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49382
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;209.0.142.64.in-addr.arpa.     IN      PTR
>
> ;; ANSWER SECTION:
> 209.0.142.64.in-addr.arpa. 259200 IN    PTR     200.ge-6-3-0.gw3.200p-sf.sonic.net.
>
> ;; Query time: 31 msec
> ;; SERVER: 208.67.222.222#53(208.67.222.222)
> ;; WHEN: Fri Nov 13 09:42:05 2009
> ;; MSG SIZE  rcvd: 91
>
>
> Another handy tip from the presentation is that, since light travels through
> fiber optic cable at about 200 km (or 125 miles, if you prefer) per
> millisecond, each 1 ms of delay shown by traceroute (which, remember, is*round
> trip* delay) should represent about 100 km (62.5 mi) of distance if the
> delay were due entirely to the distance travelled (i.e., no queuing or
> processing delays). Using that fact, you can see that 40ms for a packet to
> go from San Francisco to New York (about 2500 miles, or 4000km) would be
> "normal", but 40ms for a packet to go from San Francisco to San Jose (about
> 50 miles, or 80km) would indicate a problem; it should take the packet less
> than 1ms to cover that distance and back, so something else (congestion or
> processing delays, for example) must account for the other 39ms.
>
> There's a *lot* more in this presentation, about more complex issues such as
>
>   - how the way in which routers handle traceroute packets can produce
>   biased results (most routers handle traceroute packets much more slowly than
>   they handle "real" data packets, which can make things look much worse than
>   they are)
>   - how asymmetric paths can lead you astray (traceroute only shows you the
>   path *to* a system, but if you're pulling lots of bytes from the system,
>   as would typically be the case with a remote server, you probably care more
>   about the path back *from* the system, which might be totally different
>   - how using MPLS, which is increasingly common in carrier networks, can
>   lead to very confusing round-trip times in traceroute
>
> Anyway, if you ever use traceroute, I highly recommend that you review this
> excellent presentation. I think you'll be pleasantly surprised at how much
> you learn.
>
> Thanks to Strata Chalup of
> Virtual.net<http://www.linkedin.com/in/virtualnet> for
> bringing this very informative presentation to my attention.
> -Brent
> --
> Brent Chapman <brent@xxxxxxxxxxxx>
> Netomata, Inc. -- www.netomata.com
> Making networks more cost-effective, reliable, and flexible by automating
> network configuration
> _______________________________________________
> sage-members mailing list
> sage-members@xxxxxxxxxxxxxxxx
> http://mailman.sage.org/mailman/listinfo/sage-members
>
This archive was generated by a fusion of Pipermail (Mailman edition) and MHonArc.