Building Network Automation Solutions
6 week online course starting in September 2017

Cisco, congratulations on yesterday’s megalaunch!

Cisco’s Data Center megalaunch is probably old news by now. They announced numerous new Nexus products, linecards for Catalyst 6500 (will that box ever retire?), UCS servers, tons of FCoE-related features and few extra goodies I’m most interested in (LISP and MPLS/VPN on Nexus 7000). Scott Lowe and Chad Sakac already wrote extensive reviews of the new products and features, so I’d just like to say: Congratulations! Well done! Keep surprising me with good news ;)

Open Networking Foundation – fabric craziness reaches new heights

Some of the biggest buyers of the networking gear have decided to squeeze some extra discount out of the networking vendors and threatened them with open-source alternative, hoping to repeat the Linux/Apache/MySQL/PHP saga that made it possible to build server farms out of low-cost commodity gear with almost zero licensing costs. They formed the Open Networking Foundation, found a convenient technology (OpenFlow) and launched another major entrant in the Buzzword Bingo – Software-Defined Networking (SDN).

Networking vendors, either trying to protect their margins by stalling the progress of this initiative, or stampeding into another Wild West Gold Rush (hoping to unseat their bigger competitors with low-cost standard-based alternatives) have joined the foundation in hordes; the list of initial members reads like Who’s Who in Networking.

NAT-PT is dead! Long live NAT-64!

I’m getting questions like this one all the time: “Where are we with NAT-PT? It was implemented in IOS quite a few years ago but it has never made it into ASA code.

Bad news first: NAT-PT is dead. Repeat after me: NAT-PT is dead. Got it? OK.

More bad news: NAT-PT in Cisco IOS was seriously broken after they pulled fast switching code out of IOS. Whatever is left in Cisco IOS might be good enough for a proof-of-concept or early deployment trials, but not for a production-grade solution.

MPLS/VPN-over-GRE-over-IPSec: Does it really work?

Short answer: yes, it does.

During the geeky chat we had just after we’d finished recording the Data Center Fabric Packet Pushers podcast, Kurt (@networkjanitor) Bales asked me whether the MPLS/VPN-over-DMVPN scenarios I’m describing in Enterprise MPLS/VPN Deployment webinar (register here) really work (they do seem a bit complex).

I always test the router configurations I use in my webinars and I usually share them with the attendees. Immediately after registering for the Enterprise MPLS/VPN Deployment webinar you get access to complete sets of router configurations covering 10 scenarios, including five different MPLS/VPN-over-DMVPN designs, so you can easily test them in your lab and verify that they do work. But what about a live deployment?

Interesting links (2010-03-27)

Tip-of-the-week

The Tip-of-the-Week award goes to Jeremy (@packetlife) Stretch for his excellent Tips for starting an IT blog advice. He’s also experimenting with a three-year-delayed blog feed. Marvelous idea, I have to do something similar.

IPv6

Shortly after my Where's my AAAA record post, RIPE Labs published an excellent explanation of the broken-IPv6 phenomena. Most of the problems are caused by 6to4 tunnels; obviously it’s time to get rid of them (but unfortunately they’re built into operating systems like Windows Vista).

Geoff Huston published a two-part series on transitioning to IPv6. Part 1 describes dual stack and tunnels and part 2 focuses on Service Provider technologies like CGN, DS-Lite, 6rd and 6PE.

Jason Braddy posted an interesting link in one of the comments to my IPv6-to-IPv4 load balancing post: check out Zeus Traffic Manager; supposedly it can rewrite web pages on-the-fly.

VRF-aware services in Cisco IOS

"Which Cisco IOS services work in a VRF?" is the question I get in almost any VRF-related discussion, so I made sure it’s covered very early in my Enterprise MPLS/VPN deployment webinar (register here). This is the explanation I usually give in the webinar:

TRILL/Fabric Path – STP integration

Every Data Center fabric technology has to integrate seamlessly with legacy equipment running the venerable Spanning Tree Protocol (STP) or one of its facelifted incarnations (for example, RSTP or MST). The alternative, called rip-and-replace when talking about other vendors’ boxes or synchronized upgrade when promoting your wares (no, I haven’t heard it yet, but I’m sure it’s coming), is simply indigestible to most data center architects.

TRILL and Cisco’s proprietary Fabric Path take a very definitive stance: the new fabric is the backbone of the network routing TRILL-encapsulated layer-2 frames across bridged segments (TRILL) or contiguous backbone (Fabric Path). Both architectures segment the original STP domain into small chunks at the edges of the network as shown in the following figure:

IPv6-enabling your legacy applications with F5 BIG-IP LTM

In every enterprise-focused IPv6 presentation, including my Enterprise IPv6 – the first steps webinar (buy the recording or register for an online session), I’m telling the attendees that they can easily make their legacy applications reachable over IPv6 with a little help from F5 load balancers. After all, Facebook is doing exactly that, so it should work (in theory) ... but as we all know, in practice, the theory and practice are wildly different.

Cisco launched two more IPv4-only Data Center products

Cisco recently launched two very interesting products: layer-3 routing for the Nexus 5000 switch and the Virtual Security Gateway (which is a fantastic solution that you’ll hear more about in my future posts). Sadly, both products support only IPv4.

I’m in this industry long enough to understand the need for “baby steps” and “focusing on what customers want” (and I know there are hundreds of great engineers within Cisco who know what needs to be done, but still have to read blog posts like this one), but launching critical products without IPv6 support after the IPv4 global address pool has already been depleted definitely doesn’t look futuristic (just for fun, you might want to watch John Chambers talking about Cisco’s IPv6 thought leadership).

Don’t try to fake Multi-chassis Link Aggregation (MLAG)

Martin sent me an interesting challenge: he needs to connect an HP switch in a blade enclosure to a pair of Catalyst 3500G switches. His Catalysts are not stackable and he needs the full bandwidth between the switches, so he decided to fake the multi-chassis link aggregation functionality by configuring static LAG on the HP switch and disabling STP on it (the Catalysts have no idea they're talking to the same switch):

Interesting links (2010-03-20)

Advice of the Week

Straight from Scott Adams: true to his (and Dilbert’s) nature, he writes about Happiness Engineering.

Virtualization

Brad Hedlund is back from the corporate overload writing about inverse virtualization (isn’t that a great name for a server farm – Brad has some serious future in corporate marketing if he ever decides to change jobs again ;)

Where’s my AAAA record?

I’ll conclude this week’s IPv6 saga with a fair question I’ve received several times during the last few days: “Where’s your AAAA record?” The snappy answer would be “if you can’t see it, your ISP is not ready for production-grade IPv6”; the reality is a bit more complex.

IPv6 security issues: Fixing implementation problems

Let’s assume we’re all past the IPv6 myths phase and know that IPv6 does not offer more (or less) inherent security than IPv4. Will the IPv6 networks be as secure as IPv4 ones? Not necessarily, because we’re lacking feature parity and implementation experience. As I explained in the “IPv6 security issues: Fixing implementation problems” I wrote for SearchTelecom:

Until equipment vendors fill in the gaps and offer true feature parity between IPv4 and IPv6 security features, we can expect the IPv6 networks to be less secure that today’s IPv4 networks -- not because IPv6 is insecure, but because today’s IPv6 implementations still lag behind their IPv4 counterparts.

Read more @ SearchTelecom (or consider the excellent IPv6 Security book by Eric Vyncke).

You can't ignore IPv6 any longer (in seven steps)

We know the world will eventually run out of IPv4 addresses, but while at least some service providers got the message and already deployed IPv6, it seems like most enterprise IT departments still practice the denial strategy. It’s worrisome to read articles from Jeff Doyle describing the ignorance of his enterprise clients, so I’ll try (yet again) to explain why you should start IPv6 planning NOW.

Does Bridge Assurance make UDLD obsolete?

I got an interesting question from Andrew: “Would you say that bridge assurance makes UDLD unnecessary? It doesn't seem clear from any resource I've found so far (either on Cisco's docs or on Google).

It’s important to remember that UDLD works on physical links whereas bridge assurance works on top of STP (which also implies it works above link aggregation/port channel mechanisms). UDLD can detect individual link failure (even when the link is part a LAG); bridge assurance can detect unaggregated link failures, total LAG failure, misconfigured remote port or a malfunctioning switch.

Get the right troubleshooting tools for the job

A while ago Matthew Norwood wrote an excellent article describing the troubleshooting process they used to figure out why a particular web application worked way too slowly. Greg Ferro was quick to point out that it doesn’t make sense to assume the network is the problem and work through the whole chain slowly eliminating every potential networking device as the source of the problem when you might be facing an application design issue. However, there’s an even more important consideration: your network troubleshooting toolbox lacks the right troubleshooting tools for this job.

Interesting links (2010-03-13)

Collected in my Inbox and Twitter feed throughout last week:

To Tell the Truth: Multihop FCoE. An excellent and very precise explanation. It’s so nice to see a vendor’s blogger joining my efforts to take confusion out of multihop FCoE (it probably helps that Cisco finally has a full-blown multihop FCoE code).

Ensuring multi-tenant security in cloud services

One of the interesting problems I was facing in the recent weeks was multi-tenant security. Combine it with fuzzy all-encompassing vapor-based terminology and you have a perfect mix that can fit anything you want to sell. In the Ensuring multi-tenant security in cloud services I wrote for SearchTelecom.com I tried to structure the cloudy visions a bit: let’s figure out which type of service we’re talking about, then we can discuss what security mechanisms make sense.

As you might expect, I find IaaS the most challenging as you’re bound to hit a number of roadblocks, from VLAN limitations to architectural limitations of virtual security appliances.

Read more @ SearchTelecom ...

Framed-IPv6-Prefix used as delegated DHCPv6 prefix

Chris Pollock from io Networks was kind enough to share yet another method of implementing DHCPv6 prefix delegation on PPP interfaces in his comment to my DHCPv6-RADIUS integration: the Cisco way blog post: if you tell the router not to use the Framed-IPv6-Prefix passed from RADIUS in the list of prefixes advertised in RA messages with the no ipv6 nd prefix framed-ipv6-prefix interface configuration command, the router uses the prefix sent from the RADIUS server as delegated prefix.

This setup works reliably in IOS release 15.0M. 12.2SRE3 (running on a 7206) includes the framed-IPv6-prefix in RA advertisements and DHCPv6 IA_PD reply, totally confusing the CPE.

Building IPv6 SP Core webinar - last session in 1H2011

As expected, the demand for the Building IPv6 Core webinar is slowly diminishing – those ISPs that know what IPv6 is all about are already implementing it. The session on March 10th is thus the last live session in the first half of 2011; if you’d like to attend a live session, now is the time to register for it.

Please note that the webinar is not going away: you’ll still be able to buy the webinar recording or watch the recordings as part of the yearly subscription package.

Delegated IPv6 prefixes – RADIUS configuration

Last week I described how Cisco IOS uses two RADIUS requests to authenticate an IPv6 user (request#1) and get the delegated prefix (request#2). The second request is sent with a modified username (-dhcpv6 is appended to the original username) and an empty password (the fact that is conveniently glossed over in all Cisco documentation I found).

FreeRADIUS server is smart enough to bark at an empty password, to force the RADIUS server to accept a username with no password you have to use Auth-Type := Accept:

Site-A-dhcpv6   Auth-Type := Accept
        cisco-avpair = "ipv6:prefix#1=fec0:1:2400:1100::/56"

F5: the first vendor participating in @ioshints webinars

Regardless of the current creed preached by Gartner, multi-vendor networks are a fact of life. I’m therefore trying to make my webinars as multi-vendor as possible and as every vendor has its own Data Center Interconnect solutions, I’ve reached out to my various contacts trying to collect as much information as possible while developing the materials for my latest Data Center Interconnect webinar. I was already familiar with Cisco’s DCI solutions and Juniper’s BGP MPLS-based MAC VPN technology, but knew almost nothing about what F5 (BIG-IP with EtherIP) and others are doing. To make matters worse, I had no contacts with F5.

Interesting links (2010-03-06)

The best-discovery-of-the-week award undoubtedly goes to CCIE Routing and Switching Review Kit: Fantastic 63 pages of mind maps covering all CCIE R&S technologies, from Frame Relay and LAN switching to BGP and MPLS/VPN (link found on The Quest for CCIE blog by @mfp).

The rest of the collection is primarily focused on IPv6. I can’t possibly fathom why that would be the case ...

Internet morons are so amazing!

A few days ago I used Google to search for an article I’d written. My article was among the top results, but there was also another web site with very similar text. I’m used to blockheads publishing content stolen from my RSS feed (which is one of the primary reasons you won’t see a full feed of my blog any time soon), but this guy seemed to be copying the whole articles ... only they sounded somewhat crazy. For example,

Yesterday I described how the IPv6 architects split the functionality of IPCP into three different protocols (IPCPv6, RA and DHCPv6). While the split undoubtedly makes sense from the academic perspective, the service providers offering PPP-based services (including DSL and retrograde uses of PPP-over-FTTH) went berserk.

... became ...

Yesterday we dеѕсrіbеd hοw thе IPv6 architects rip thе functionality οf IPCP іntο 3 odd protocols (IPCPv6, RA аnd DHCPv6). Whіlе thе rip positively mаkеѕ clarity frοm thе educational perspective, thе use providers charity PPP-based services (counting DSL аnd opposing uses οf PPP-over-FTTH) wеnt berserk.

Don’t lie about proprietary protocols

A few months ago Brocade launched its own version of Data Center Fabric (VCS) and the VDX series of switches claiming that:

The Ethernet Fabric is an advanced multi-path network utilizing an emerging standard called Transparent Interconnection of Lots of Links (TRILL).

Those familiar with TRILL were immediately suspicious as some of the Brocade’s materials mentioned TRILL in the same sentence as FSPF, but we could not go beyond speculations. The Brocade’s Network OS Administrator’s Guide (Supporting Network OS v2.0, December 2010) shows a clear picture.

The Data Center Fabric architectures

Have you noticed how quickly fabric got as meaningless as switching and cloud? Everyone is selling you data center fabric and no two vendors have something remotely similar in mind. You know it’s always more fun to look beyond white papers and marketectures and figure out what’s really going on behind the scenes (warning: you might be as disappointed as Dorothy was). I was able to identify three major architectures (at least two of them claiming to be omnipotent fabrics).

Business as usual

Each networking device (let’s confuse everyone and call them switches) works independently and remains a separate management and configuration entity. This approach has been used for decades in building the global Internet and thus has proven scalability. It also has well-known drawbacks (large number of managed devices) and usually requires thorough design to scale well.

DHCPv6-RADIUS integration: the Cisco way

Yesterday I described how the IPv6 architects split the functionality of IPCP into three different protocols (IPCPv6, RA and DHCPv6). While the split undoubtedly makes sense from the academic perspective, the service providers offering PPP-based services (including DSL and retrograde uses of PPP-over-FTTH) went berserk. They were already using RADIUS to authenticate PPP users ... and were not thrilled by the idea that they should deploy DHCPv6 servers just to make the protocol stack look nicer.

IPv6CP+DHCPv6+SLAAC+RA = IPCP

Last week I got an interesting tweet: “Hey @ioshints can you tell me what is the radius parameter to send ipv6 dns servers at pppoe negotiation?” It turned out that the writer wanted to propagate IPv6 DNS server address with IPv6CP, which doesn’t work. Contrary to IPCP, IPv6CP provides just the bare acknowledgement that the two nodes are willing to use IPv6. All other parameters have to be negotiated with DHCPv6 or ICMPv6 (RA/SLAAC).

The following table compares the capabilities of IPCP with those offered by a combination of DHCPv6, SLAAC and RA (IPv6CP is totally useless as a host parameter negotiation tool):