Category: IP routing

The Optimal L3 Forwarding with VARP/VRRP post generated numerous comments, ranging from technical questions about VARP (more about that in a few days) to remarks along the lines of “you can do that with X” or “vendor Y supports Z, which does the same thing.” It seems I’ve opened yet another can of worms, let’s try to tame and sort them.

I’ve blogged about the need for optimal L3 forwarding across the whole data center in 2012 when I introduced it as one of the interesting requirements in Data Center Fabrics webinar. Years later, the concept became one of the cornerstones of modern EVPN fabrics, but there are still only a few companies that can deliver this functionality in a more traditional environment.

I stumbled upon a fantastic RFC - Evolution of IP Model (RFC 6250) - that should be made mandatory reading for everyone remotely involved with networking. It describes numerous "truths" (politely called misconceptions) that everyone from programmers to network designers still rely upon. Some of my favorites: reachability is symmetric and transitive, loss is rare, addresses are stable, each host has a single interface and a single IP address ... Enjoy!

Adam Sweeney, VP of EOS Engineering @ Arista Networks posed me a challenging question after my I-so-hate-PBR-CLI rant: “Is there something in particular that makes the IOS PBR CLI so painful? Is there a PBR CLI provided by any of the other systems out there that you like a lot better?”

My Twitter friends helped me find the answer to the second question: PBR in Junos is even more convoluted than it is in Cisco IOS... but what would be a better CLI?

During one of my ExpertExpress consulting engagements I encountered an interesting challenge:

We have a network with two data centers (connected with a DCI link). How could we ensure the applications in a data center stay reachable even if all local Internet links fail?

On the face of it, the problem seems trivial; after all, you already have the DCI link in place, so what’s the big deal ... but we quickly figured out the problem is trickier than it seems.

Thomas wanted to check whether the IP traffic is actually delivered to a remote site and sent me the following question:

I would like to know whether the packets I sent from site A to site B have been received. I don't want to create test traffic using ip sla, I would like to know that the production traffic has been delivered. I could use ACL counters but I'm running a full mesh of tens of sites. Ipanema does this very well, but I'm surprised that this doesn’t exist on Cisco IOS.

Short answer: that’s not how Internet works.

One of my readers sent me a great question:

I agree with you that L2 DCI is like driving without a seat belt. But is L3 DCI safer in case of DCI link failure? Let's say you have your own AS and PI addresses in use. Your AS spans multiple sites and there are external BGP peers on each site. What happens if the L3 DCI breaks? How will that impact your services?

Simple answer: while L3 DCI is orders of magnitude safer than L2 DCI, it will eventually fail, and you have to plan for that.

A while ago someone sent me an interesting problem: the moment he enabled simple MPLS in his enterprise network with ip mpls interface configuration commands, numerous web applications stopped working. My first thought was “MTU problems” (the usual culprit), but path MTU discovery should have taken care of that.

Almost everyone agrees the current way of implementing virtual networks with dumb hypervisor switches and top-of-rack kludges (including Edge Virtual Bridging – EVB or 802.1Qbg – and 802.1BR) doesn’t scale. Most people working in the field (with the notable exception of some hardware vendors busy protecting their turfs in the NVO3 IETF working group) also agree virtual networks running as applications on top of IP fabric are the only reasonable way to go ... but that’s all they currently agree upon.

Very short answer: no.

You might think that layer-3 switches perform bridging and routing, while routers do only routing. That hasn’t been the case at least since Cisco introduced Integrated Routing and Bridging in IOS release 11.2 more than 15 years ago. However, Simon Gordon raised an interesting point in a tweet: “I thought IP L3 switching includes switching within subnet based on IP address, routing is between subnets only.”

Layer-3 switches and routers definitely have to perform some intra-subnet layer-3 functions, but they’re usually not performing any intra-subnet L3 forwarding.

Keith sent me a set of Mobile ARP questions, starting with “What’s your view on using Mobile ARP in a large enterprise?”

Short summary: Mobile ARP is an ancient technology that was designed to solve a problem that disappeared with the deployment of DHCP. Now, let’s look at the bigger picture.

Most current SDNish tools are too cumbersome for everyday use: OpenFlow is too granular (the controller interacts directly with the FIB or TCAM), and NETCONF is too coarse (it works on the device configuration level and thus cannot be used to implement anything the networking device can’t already do). In many cases, we’d like an external application to interact with the device’s routing table or routing protocols (similar to tracked static routes available in Cisco IOS, but without the configuration hassle).

Every data center network has a mixture of bridging (layer-2 or MAC-based forwarding, aka switching) and routing (layer-3 or IP-based forwarding); the exact mix, the size of L2 domains, and the position of L2/L3 boundary depend heavily on the workload ... and I would really like to understand what works for you in your data center, so please leave as much feedback as you can in the comments.

Matthew sent me the following remarkable fact (and he just might have saved some of you a few interesting troubleshooting moments):

I was bringing up an OSPF adjacency between a Catalyst 6500 and an ASR 9006 and kept getting an MTU mismatch error. The MTU was set exactly the same on both sides. So I reset them both back to default (1500 on the 6500 and 1514 on the ASR 9006) and the adjacency came back up, even though now the MTU is off by 14 bytes. So I attempted to bump the MTU up again, this time setting the MTU on 6500 to 1540 and the MTU on the ASR 9006 to 1554. Adjacency came right up. Is there something I am missing?

The 14 byte difference is the crucial point – that’s exactly the L2 header size (12 bytes for two 6-byte MAC addresses and 2 bytes for ethertype). When you specify MTU size on the IOS classic (either with the ip mtu command or with the mtu command), you specify the maximum size of the layer-3 payload without the layer-2 header. Obviously IOS XR works differently – there you have to specify the maximum size of a layer-2 frame, not of its layer-3 payload (comments describing how other platforms behave are most welcome!).

Every time I mention that Cisco IOS had Local Area Mobility (LAM) (the feature that would come quite handy in today’s virtualized data centers) more than a decade ago, someone inevitably asks “why don’t we use it?” LAM looks like a forgotten step-child, abandoned almost as soon as it was created (supposedly it never got VRF support). The reason is simple (and has nothing to do with the size of L3 forwarding tables): LAM was always meant to be a short-term kludge and L3 gurus never appreciated its potentials.