Category: IP routing

I just read a great article by Kurt (the Network Janitor) Bales eloquently describing how a series of stupid decisions led to the current situation where everyone (but the people who actually work with the networking infrastructure) think stretched layer-2 domains are the mandatory stepping stone toward the cloudy nirvana.

It’s easy to shift the blame to everyone else, including storage vendors (for their love of FC and FCoE) and VMware (for the broken vSwitch design), but let’s face the reality: the rigid mindset of layer-3 gurus probably has as much to do with the whole mess as anything else.

If you’re confused about the numerous meanings of a switch, you’re not the only one. If you wonder how the whole mess started, here’s the full story (from a biased perspective of a grumpy GONER):

35 years ago, there were no bridges or routers. Hosts communicated directly with each other or used intermediate nodes (usually hosts, sometimes dedicated devices called gateways) to pass traffic ... and then a few overly-bright engineers at DEC decided their application (LAT) will run directly on layer 2 to make it faster.

Their company has been dead (actually, sold in pieces) for over a decade, but their eagerness to cut corners still haunts every one of us.

Before someone accuses me of being totally FCoE/DCB-focused, here’s an interesting EEM trick. Damian wanted to have time-dependent static routes to ensure expensive backup path is only established during the working hours. I told him to use cron with EEM to modify router configuration (and obviously lost him in the acronym forest)... but there’s an even better solution: use reliable static routing and modify just the track object’s state with EEM.

A while ago I had a chat with a fellow CCIE (working in a large enterprise network with reasonably-sized Data Center) and briefly described vMotion to him. His response: “Interesting, I didn’t know that.” ... and “Ouch” a few seconds later as he realized what vMotion means from bandwidth consumption and routing perspectives. Before going into the painful details, let’s cover the basics.

During the Big Hot and Heavy Switches podcast, Dan Hughes complained that the Nexus 7000 switch cannot take the full BGP table. The reason is simple: it’s TCAM (FIB) has only 56.000 entries and the BGP table has almost 350.000 routes.

Nexus 7000 is a Data Center switch, so the TCAM size is not really a limitation (it would usually have a default route toward the WAN core), but the same problem is experienced by Service Providers all over the world – the TCAM/FIB size of their high-speed routers is limited.

Every now and then, I’m asked about the difference between Routing Information Base (RIB), also known as IP Routing Table and Forwarding Information Base (FIB), also known as CEF table (on Cisco’s devices) or IP forwarding table.

Let’s start with an overview picture (which does tell you more than the next thousand words I’ll write):

Almost 30 years ago, I was lucky enough to work on one of the best systems of those days, VAX/VMS (BTW, it was able to run 30 interactive users in 2 MB of main memory), which had everything we’d wished for – it was truly interactive with hierarchical file system and file versioning (not to mention remote file access and distributed clusters). I couldn’t possible understand the woes of IBM mainframe programmers who had to deal with virtualized 132-column printers and 80-column card readers (ironically running in virtual machines that the rest of the world got some 20 years later). When I wanted to compile my program, I started the compiler; when they wanted to do the same, they had to edit a batch job, submit the batch job (assuming the disk libraries were already created), poll the queues to see when it completed and then open the editor to view the 132-column printout of compiler errors.

After a long discussion, I started to understand the problem: the whole system was burdened with so many legacy decisions that still had to be supported that there was nothing one could do to radically change it (yeah, it’s hard to explain that to a 20-year old kid full of himself).

Based on the readers’ comments on my “Bridging and Routing: is there a difference?” post (thanks you!), here are a few more differences between bridging and routing:

Cost. Layer-2 switches are almost always cheaper than layer-3 (usually combined layer-2/3) switches. There are numerous reasons for the cost difference, including:

In his comment to one of my TRILL posts, Petr Lapukhov has asked the fundamental question: “how is bridging different from routing?”. It’s impossible to give a concise answer (let alone something as succinct as 42) as the various kludges and workarounds (including bridges and their IBM variants) have totally muddied the waters. However, let’s be pragmatic and compare Ethernet bridging with IP (or CLNS) routing. Throughout this article, bridging refers to transparent bridging as defined by the IEEE 802.1 series of standards.

Design scope. IP was designed to support global packet switching network infrastructure. Ethernet bridging was designed to emulate a single shared cable. Various design decisions made in IP or Ethernet bridging were always skewed by these perspectives: scalability versus transparency.

One of the scenarios I’m discussing in the DMVPN webinar is redundant DMVPN network with two ISPs. It’s not a particularly complex setup, unless the ISPs decide to deploy anti-spoofing filters (more precisely: unicast RPF checks) in which case it becomes crucially important which outbound interface you use for your DMVPN tunnel.

Anyhow, I was trying to make the whole thing work in a lab and it was repeatedly failing, so I decided to log uRPF violations. According to the documentation, it’s a piece of cake:

Bidirectional Forwarding Detection (BFD) protocol has finally been published as a series of RFCs. BFD gives you quick failure detection between L3 hops (routers) regardless of the underlying technology and equipment (modems, media converters, bridges). It’s been gradually introduced in Cisco IOS during the last few years; release 15.0M and 12.2SRE contain almost everything you’ll ever need (missing: multihop BGP support and MPLS LSP support).

I wrote about BFD in Improve the Convergence of Mission-Critical Networks with Bidirectional Forwarding Detection (BFD) article (you’ll find it somewhere in this list). To learn more, read the RFCs in this order:

A few days ago I’ve received a cryptic e-mail with exactly this content: “I am having a issue "static routes not flushed when next hop is unreachable" please advice.” I suspected that the sender actually wanted to ask me what to do if a static route pointing to an IP next-hop does not disappear when the next hop becomes unreachable and told him to adjust the ip route static adjust-time parameter while monitoring the CPU usage.

Recently-published RFC 5837 describes additions to ICMP messages that would allow you to gather more information (including interface ifIndex, IP address and name). Two obvious applications are enhanced traceroute and path MTU discovery where the new ICMP extensions could indicate which interface is the MTU bottleneck.

The RFC authors come from BT, Juniper and Cisco, so there’s a non-zero chance it will actually get implemented where it’s most needed.

There are three tools that can (according to a CCIE friend of mine) solve any networking-related problems: GRE tunnels, PBR and VRFs. The solutions to the L2TP default routing challenge nicely prove this hypothesis; most of them use at least one of those tools.

Policy-based routing on virtual template interface. Use the default route toward the Internet and configure PBR with set default next-hop on the virtual template interface. The PBR is inherited by all virtual access interfaces, ensuring that the traffic from remote sites always passes the network core (and the firewall, if needed).

Imagine you have a large retail network: your remote offices use ISDN to dial into the central site and upload/download whatever periodic reports they have. Having a core router connected to an ISDN PRI interface is the perfect solution:

A few years later, ISDN is becoming too slow for your increased traffic needs and you want to replace it with DSL or VPN-over-Internet solution. Your Service Provider offers you PPPoE forwarding with L2TP. This is a perfect solution as it allows you to minimize the changes: