Building Network Automation Solutions
6 week online course starting in September 2017

That’s it for 2013

12 months, 260 blog posts, and a dozen of webinars … and it’s time for another end-of-year post. It’s amazing how quickly a year goes by when you have fun.

I’d like to thank you for your insightful comments, great questions you asked, and wonderful challenges you keep sending me … and special thanks to all of you who trusted me enough to buy my webinars or decided to rely on my professional judgment.

Don’t forget to shut down your pagers and smartphones (if at all possible), and enjoy the simpler (and less stressful) life with the loved ones. Have a great holiday season and all the best (including plenty of SDN fun) in the coming year!

Webinars in 2013

As you know, I’m promising my subscribers 4-6 new sessions a year. 2013 definitely wasn’t a bad year in that respect ;)

The year started with IPv6 Transition Mechanisms, and virtual firewalls.

There was a deep dive into scale-out architectures and load balancing in April, and the mandatory Data Center Fabrics Update session in May.

Does It Make Sense to Build New Clouds with Overlay Networks?

TL&DR Summary: It depends on your business model

With the explosion of overlay virtual networking solutions (with every single reasonably-serious vendor having at least one) one might get the feeling that it doesn't make sense to build greenfield IaaS cloud networks with VLANs. As usual, there's significant difference between theory and practice.

You should always consider the business requirements before launching on a technology crusade. IaaS networking solutions are no exception.

Overlay Virtual Networking Solutions Overview

2013 was definitely the year of overlay virtual networks, with every major networking and virtualization vendor launching a new product or adding significant functionality to an existing one. Here’s a brief overview of what they’re currently offering:

Focus on Your Business, Not Fancy Technologies

After my Clouds, Overlays and SDN: What really matters keynote presentation @ MENOG 12 a few attendees asked me for a recording; one of them said “I want everyone in my organization to watch it.” Alas, wishes don’t always come true: the video team was streaming the presentations, but not recording them.

Fortunately I had the same presentation @ PLNOG 11 and like always the PLNOG organizers did a marvelous job. The video has just been posted on YouTube. Enjoy!

IGMP and PIM in Multicast VXLAN Transport Networks

Got a really interesting question from A. Reader: “When and how does VXLAN use IGMP and PIM in transport (underlay) networks?

Obviously you need IGMP and PIM in multicast environments only (vCNS 5.x, Nexus 1000V in multicast mode).

SDN 101 Webinar is Free – Register Now

In late November I got a perfect excuse for visiting South Africa – I was invited to be a guest speaker at CCIE Club Africa meeting talking about the obvious topics: SDN, OpenFlow and Network Function Virtualization (NFV).

In the end, I delivered three SDN presentations in two days, all of them for engineers focusing primarily on Cisco, and got pleasantly surprised by their keen interest in the basics of these new technologies.

Control Plane in OpenFlow Networks

It’s easy to saySDN is the physical separation of the network control plane from the forwarding plane, and where a control plane controls several devices,” handwave over the details, and let someone else figure them out. Implementing that concept in a reliable manner is a totally different undertaking.

Internet Traffic Gets MPLS Labels When You Deploy MPLS/VPN

A good friend of mine sent me an interesting question:

When I configure mpls ip on an interface, will all packets on that interface be labeled, or just the MPLS/VPN packets received through VRFs? I always assumed that stuff in the global routing table just got forwarded as IP packets without any labels.

Well, that’s not how MPLS works (at least not in its default incarnation on Cisco IOS).

Packet Forwarding in Amazon VPC

Packet forwarding behavior of VMware NSX and Hyper-V Network Virtualization is well documented; no such documentation exists for Amazon VPC. However, even though Amazon uses a proprietary solution (heavily modified Xen hypervisor with homemade virtual switch), it’s pretty easy to figure out the basics from the observed network behavior and extensive user documentation.

Still Waiting for the Stupid Network

More than 15 years ago the cover story of ACM netWorker magazine discussed the dawn of the stupid network – an architecture with smart edge nodes and simple packet forwarding code. Obviously we learned nothing in all those years – we’re still having the same discussions.

Here are a few juicy quotes from that article (taken completely out of context solely for your enjoyment).

Layer-2 Gateways in VMware NSX

Gateways between overlay virtual world and (VLAN-based) physical reality are a crucial component in every design using overlay virtual networks. Ideally one could use virtual appliances, but sometimes the users keep asking for layer-2 gateways.

The VMware NSX Layer-2 Gateways video from the VMware NSX Architecture webinar describes the use cases for layer-2 gateways and the VMware NSX implementations.

Hyper-V Network Virtualization Packet Forwarding Improvements in Windows Server 2012 R2

Initial release of Hyper-V Network Virtualization (HNV) was an add-on to the Hyper-V Extensible Switch, resulting in an interesting mixture of bridging and routing. In Windows Server 2012 R2 the two components became tightly integrated, resulting in a pure layer-3 solution.

OMG, Who Will Manage All Those Virtual Firewalls?

Every time I talk about small (per-application) virtual appliances, someone inevitably criesAnd who will manage thousands of appliances?” Guess what – I’ve heard similar cries from the mainframe engineers when we started introducing Windows and Unix servers. In the meantime, some sysadmins manage more than 10.000 servers, and we’re still discussing the “benefits” of humongous monolithic firewalls.

BGP Routing in DMVPN Networks

Once you decide to use BGP as the routing protocol in your DMVPN network, you face a few more design choices:

  • Should you use IBGP or EBGP?
  • Should you use a unique AS number for every DMVPN site, or the same AS number on all spoke sites?

The BGP Routing in DMVPN Access Networks ExpertExpress case study describes these dilemmas in more details; if you face a similar problem and would like me to review your design, get in touch.

Virtual Packet Forwarding in Hyper-V Network Virtualization

Last week I explained how layer-2 and layer-3 packet forwarding works in VMware NSX – a solution that closely emulates traditional L2 and L3 networks. Hyper-V Network Virtualization (HNV) is different – it’s almost a layer-3-only solution with only a few ties to layer-2.

We Had SDN in 1993 … and Didn’t Know It

I had three SDN 101 presentations during last week’s visit to South Africa and had tried really hard to overcome my grumpy skeptic self and find the essence of SDN while preparing for them. As I’ve been thinking about controllers, central visibility and network device programmability, it struck me: we already had SDN in 1993.

Terastream Part 2: Lightweight 4over6 and Network Function Virtualization (NFV)

In the first Terastream blog post I mentioned Deutsche Telekom decided to use an IPv6-only access network. Does that mean they decided to go down the T-Mobile route and deployed NAT64 + 464XLAT? That combo wouldn’t work well for them, and they couldn’t use MAP-E due to lack of IP address space, so they deployed yet another translation mechanism – Lightweight 4over6.

Layer-3 Forwarding with VMware NSX Edge Services Router

The easiest way of connecting overlay virtual networks implemented with VMware NSX for vSphere to the outside world is NSX Edge Services Router. It’s a much improved version of vShield Edge and provides way more than just layer-3 forwarding services – it’s also a firewall, load balancer, DHCP server, DNS forwarder, NAT and VPN termination device.

Don’t Use ULA Addresses in Service Provider Core

Dan sent me the following question:

I had another read of the ‘Building IPv6 Service Provider Networks’ material and can see the PE routers use site local ipv6 addressing. I’m about to build another small service provider setup and wondered: would you actually use site local for PE loopbacks etc, or would you use ULA or global addressing? I’m thinking ULA would be better from a security point of view?

TR&DR summary: Don’t do that.

VMware NSX: the Need for Overlay Virtual Networks

In the second section of VMware NSX Architecture webinar I explained the need for overlay virtual networks and what their benefits are as compared to traditional VLANs.

Programming the Network – A Few Guidelines

Even though I questioned the wisdom of writing your own network programming applications, I know I would immediately jump into those waters if I were 20 years younger. If you’re like my younger self, you might want to keep a few guidelines in mind.

Typical Enterprise Application Deployment Process is Broken

Months ago VMware started promoting NSX with a catchy “fact” – you can deploy a new VM or virtual disk in minutes, but it usually takes days or more before you can get a new VLAN or a firewall or load balancer rule from the networking team.

Ignoring the complexity of network virtualization, they had a point, and the network services rigidity really bothered me … until I finally realized that we’re dealing with a broken process.

Layer-2 and Layer-3 Switching in VMware NSX

All overlay virtual networking solutions look similar from far away: many provide layer-2 segments, most of them have some sort of distributed layer-3 forwarding, gateways to physical world are ubiquitous, and you might find security features in some products.

The implementation details (usually hidden behind the scenes) vary widely, and I’ll try to document at least some of them in a series of blog posts, starting with VMware NSX.

Deutsche Telekom TeraStream: Designed for Simplicity

Almost a year ago rumors started circulating about a Deutsche Telekom pilot network utilizing some crazy new optic technology. In spring I’ve heard about them using NFV and Tail-f NCS for service provisioning … but it took a few more months till we got the first glimpses into their architecture.

TL&DR summary: Good design always beats bleeding-edge technologies

Technical Debt – and How We Can Fix It

In late October I had the closing presentation at our yearly customer event, and decided to talk about one of the most pressing (at least in my opinion) IT problems – the technical debt from the networking/sysadmin perspective.

You can view the presentation on my web site. It’s one of those presentations that look way better on video (which will be published … but it’s in Slovenian), but I’m positive the meme-lovers will enjoy it.

Make Every Application an Independent Tenant

Traditional data centers are usually built in a very non-scalable fashion: everything goes through a central pair of firewalls (and/or load balancers) with thousands of rules that no one really understands; servers in different security zones are hanging off VLANs connected to the central firewalls.

Some people love to migrate the whole concept intact to a newly built private cloud (which immediately becomes server virtualization on steroids) because it’s easier to retain existing security architecture and firewall rulesets.

Scalable OpenFlow Solutions: Between PowerPoint and Production

The Shipping SDN, NFV, OpenFlow, Network Virtualization Products and Services page on SDN Central lists dozens of SDN products, from switches to controllers and applications. I’m positive all of them work great in PowerPoint, most of them work well in lab environments, but do they also work in large-scale production deployments?

If you’re a regular reader of my blog, you probably know all about OpenFlow scalability challenges, for everyone else I included a section on OpenFlow scalability in free Real Life OpenFlow-based SDN Use Cases webinar (we have just a few places left - register here).

Cisco Nexus 9000 and ACI: Promising P+V Architecture

Last week’s Nexus 9000 and Application Centric Architecture launch triggered an avalanche of opinions. Some bloggers focused on the fact that there’s a Linux kernel underneath the NX-OS (So what? What else would make sense?), others tried to make sense of ACI from the marketing materials (good luck with that) … and almost nobody mentioned what might be a crucial piece of the architecture: the Application Virtual Switch (AVS).

Two and a Half Years after OpenFlow Debut, the Media Remains Clueless

If you repeat something often enough, it becomes a “fact” (or an urban myth). SDN is no exception, industry press loves to explain SDN like this:

[SDN] takes the high-end features built into routers and switches and puts them into software that can run on cheaper hardware. Corporations still need to buy routers and switches, but they can buy fewer of them and cheaper ones.

That nice soundbite contains at least one stupidity per sentence:

It's just a quick change, really ...


Couldn't resist ... (source: DevOps Reactions)

Real Life OpenFlow/SDN Use Cases

While meeting-goers debate viable SDN use cases, bleeding-edge engineers who weren't afraid to get their hands dirty started rolling out real-life OpenFlow- and SDN-based solutions. I’ll describe some of them (including smarter network monitoring, tap aggregation networks, scale-out appliances, and distributed DoS detection) in the OpenFlow-based SDN Use Cases webinar next Thursday. The webinar is free (NEC decided to sponsor it), but the number of places is limited, so register ASAP.

One-Tier Data Centers? Now Bigger Than Ever with Arista 7300 Switches

Early November seems to be the right time for data center product harvest: after last week’s Juniper launch, Arista launched its new switches on Monday. The launch was all we came to expect from Arista: better, faster, more efficient switches … and a dash of PureMarketing™ – the Splines.

Use ThousandEyes to Implement IP SLA on Steroids

You did read my blog post on ThousandEyes, didn’t you? What I forgot to mention was that they have this cool API that allows you to extract measurement data (including BGP topology) from their system. Can we do something cool with that?

Finally: Juniper Supports a Leaf-and-Spine Virtual Chassis

The recent Juniper product launch included numerous components, among them: a new series of data center switches (including a badly-needed spine switch), MetaFabric reference architecture (too meta for me at the moment – waiting to see the technical documentation beyond the whitepaper level), and (finally) a leaf-and-spine virtual chassis – Virtual Chassis Fabric.

Are Your Applications Cloud-Friendly?

A while ago I had a discussion with someone who wanted to be able to move whole application stacks between different private cloud solutions (VMware, Hyper-V, OpenStack, Cloud Stack) and a variety of public clouds.

Not surprisingly, there are plenty of startups working on the problem – if you’re interested in what they’re doing, I’d strongly recommend you add CloudCast.net to your list of favorite podcasts – but the only correct way to solve the problem is to design the applications in a cloud-friendly way.

Overlay Virtual Networking Video

PLNOG organizers published the video of my Overlay Virtual Networking Explained presentation. They did a fantastic job, nicely merging live video with slides and splendid background.

If you need more details or an in-depth evaluation of products from numerous vendors, check out the Overlay Virtual Networking webinar (the final videos have just been published).

Create Network Models with CML’s AutoNetKit

Last week I described how Cisco Modeling Lab (CML, the product formerly known as VIRL) works behind its fantastic UI, and promised more information about the UI once I get access to a preview version of CML, which I got a few days ago. Here are the results of the first brief stroll down the virtual lane.

IBGP Migrations Can Generate Forwarding Loops

A group of researches presented an “interesting” result @ IETF 87: migrating from IBGP full mesh to IBGP reflectors can introduce temporary forwarding loops. OMG, really?

Don’t panic, the world is not about to become a Vogon hyperspace bypass. Let’s put their results in perspective.

Traceroute of the year

Michael Haines (probably inspired by my data center design and recent tweet) sent me the best traceroute hint I've ever seen. Do traceroute obiwan.scrye.net and you'll see something like this:

VMware NSX: Defining the Problem

Every good data center presentation starts with redefining The Problem and my VMware NSX Architecture webinar was no exception – the first section describes Infrastructure-as-a-Service Networking Requirements.

I sprinted through this section during the live session, the video with longer (and more detailed) explanation comes from the Overlay Virtual Networking webinar.

Cisco Modeling Lab (VIRL) behind the scenes

The first hints of VIRL started appearing around Cisco Live US 2013 where the product development team demonstrated Cisco’s take on 21st century network modeling tool. A few days ago, Omar Sultan, Joel Obstfeld and Ed Kern gave us a brief peek behind the scenes of this totally awesome tool (note to Cisco haters: I haven’t been drinking the teal Kool-Aid for a long time – this is my honest impression).

Lego Data Center

Ashton Bothman from Juniper invited me to an interesting contest: build a Lego Data Center. I just happen to have in-house Lego Design Experts (read: kids), so I gladly delegated the task to that team. Here are the results (using the Force instead of unicorns).

Can You Find SQL Injection Vulnerabilities with Spirent Avalanche NEXT?

An odd idea stroke me when watching the Avalanche NEXT presentation during Networking Tech Field Day – they have a fuzzing module that you can use to test whether your servers and applications survive all sorts of crazy illegal requests. Could that be used to detect SQL injection vulnerabilities in your web apps?

Flow Table Explosion with OpenFlow 1.0 (And Why We Need OpenFlow 1.3)

The number of flows in hardware switches (dictated by the underlying TCAM size) is one of the major roadblocks in a large-scale OpenFlow deployment. Vendors are supposedly making progress, with Intel claiming up to 4000 12-tuple flow entries in their new Ethernet Switch FM6700 series. Is that good enough? As always, it depends.

TL&DR summary: Use switches that support OpenFlow 1.3.

Network Troubleshooting Checklist

Ronald Bartels created an interesting network troubleshooting checklist that covers numerous aspects of the troubleshooting process, from information gathered during problem reporting phase to timelines, investigation activities, device and port checks ... Feedback highly welcome!

IPv6-Only Data Centers: Deployment Guidelines

During the final part of the IPv6-only data centers webinar Tore Anderson described his deployment guidelines and answered a few more questions.

Estimating the Number of TCP Sessions per Host

Another day, another stateful debate, this time centered on the number of flows per hypervisor. Previously I guestimated 2.500 connections-per-second-per-(user-facing)gigabit and 37.500 concurrent sessions per user-facing gigabit, but wanted to align my numbers with reality before reaching any conclusions.

My web sites are way too small, so I asked a few of my friends to help me get more realistic figures.

Why is IPsec so Complex?

Jason Edelman wrote a great blog post after watching Ethan Banks struggle with yet another multi-vendor IPsec deployment. Some of his ideas make perfect sense (wiki-like web site documenting working configurations between vendor X and Y for every possible X and Y), others less so (tunnel broker – particularly in view of recent Tor challenges), but let’s step back a bit and ask ourselves “Why is IPsec so complex?

Forwarding Models in OpenFlow Networks

A few days ago Tom (@NetworkingNerd) Hollingsworth asked a seemingly simple question: “OpenFlow programs hop-by-hop packet forwarding, right? No tunnels?” and wasn’t satisfied with my standard answer, so here’s a longer explanation.

ATAoE Is Alive and Well

A while ago I wrote about ATAoE and why I think a layer-2-only TFTP-like protocol shouldn’t be used these days. As always, the answer to that black-and-white opinion (and I’m full of them) is “it depends” – ATAoE works great if you do it right.

VMware NSX Architecture Videos Published

The edited videos from VMware NSX Architecture webinar are published on my demo content web site and on YouTube. Enjoy!

How do you write a blog post a day?

It all started with a message from one of my Twitter friends: “how on Earth do you find the time to blog so often?” Here’s the secret recipe: a happy little thought and a bit of fairy dust. No, got it wrong, that helps you fly. The real secret ingredients: time, process, ideas, and a pinch of motivation.

Exception Routing with BGP: SDN Done Right

One of the holy grails of data center SDN evangelists is controller-driven traffic engineering (throwing more leaf-and-spine bandwidth at the problem might be cheaper, but definitely not sexier). Obviously they don’t call it traffic engineering as they don’t want to scare their audience with MPLS TE nightmares, but the idea is the same.

Interestingly, you don’t need new technologies to get as close to that holy grail as you wish; Petr Lapukhov got there with a 20 year old technology – BGP.

Can BGP Route Reflectors Really Generate Forwarding Loops?

TL&DR Summary: Yes (if you’re clumsy enough).

A while ago I read Impact of Graceful IGP Operations on BGP – an article that described how changes in IGP topology result in temporary (or sometimes even permanent) forwarding loops in networks using BGP route reflectors.

Is the problem real? Yes, it is. Could you generate a BGP RR topology that results in a permanent forwarding loop? Yes. It’s not that hard.

What Exactly Is The Control Plane?

Tassos opened an interesting can of worms in a comment to my Management, Control and Data Planes post: Is ICMP response to a forwarded packet (TTL exceeded, fragmentation needed or destination unreachable) a control- or data-plane activity?

Overlay Virtual Networks 101

My keynote speech @ PLNOG11 conference was focused on (surprise, surprise) overlay virtual networks and described the usual motley crew: The Annoying Problem, The Hated VLAN, The Overlay Unicorn, The Control-Plane Wisdom and The Ever-Skeptic Use Case. You can view the presentation on my web site; PLNOG organizers promised video recording in mid-October.

Just in case you’re wondering why I keep coming back to PLNOG: they’re not only as good as ever; they’re getting even more creative.

TTL in Overlay Virtual Networks

After we get rid of the QoS FUD, the next question I usually get when discussing overlay networks is “how should these networks treat IP TTL?

As (almost) always, the answer is “It depends.”

OpenStack Quantum (Neutron) Plug-In: There Can Only Be One

OpenStack seems to have a great architecture: all device-specific code is abstracted into plugins that have a well-defined API, allowing numerous (more or less innovative) implementations under the same umbrella orchestration system.

Looks great in PowerPoint, but to an uninitiated outsider looking at the network (Quantum, now Neutron) plugin through the lenses of OpenStack Neutron documentation, it looks like it was designed by either a vendor or a server-focused engineer using NIC device driver concepts.

The Intricacies of Optimal Layer-3 Forwarding

I must have confused a few readers with my blog posts describing Arista’s VARP and Enterasys’ Fabric Routing – I got plenty of questions along the lines of “how does it really work behind the scenes?” Let’s shed some light on those dirty details.

To ULA or not to ULA, That’s the Question

Ed Horley, an awesome IPv6 geek I had the privilege to meet at NFD6, wrote an interesting blog post arguing against IPv6 ULA usage (particularly when combined with NPT66). We would all love to get rid of NAT, however ...

IPv6-Only Data Center: Q&A Time

Not surprisingly, the unorthodox ideas of Tore Anderson generated plenty of questions, so he spent ~20 minutes answering them.

OpenFlow and Fermi Estimates

Fast advances in networking technologies (and the pixie dust sprinkled on them) blinded us – we lost our gut feeling and rule-of-thumb. Guess what, contrary to what we love to believe, networking isn’t unique. Physicists faced the same challenge for a long time; one of them was so good that they named the whole problem category after him.

Every time someone tries to tell you what your problem is, and how their wonderful new gizmo will solve it, it’s time for another Fermi estimate.

Know Thy Boundaries

Every mid-sized company usually has legal counsel on staff (we have two lawyers for ~100 employees, but we might be a bit specific), that will escalate to an external law firm as necessary. Usually this would be when dealing with extraordinary events such as lawsuits or negotiation of a complex agreement.

Configure physical firewalls based on VM groups? Sure, use DSE from Plexxi

Plexxi has an interesting problem. They have a shiny new solution that requires unorthodox approaches to network forwarding and allows them to implement potentially cool concepts like affinities (traffic engineering in disguise). They also need to sell these new concepts to the customers, and the first question I would ask after recovering from a hefty dose of cool-aid is "and how do I configure these affinities?"

Net::Beer @ PLNOG 11

I'll be talking about overlay virtual networks and clouds @ PLNOG 11 (September 30th - October 1st in Krakow) - if you're there and would like to chat with me, send me an email or a tweet.

Also, the organizers sent me this great promotional picture - I simply have to publish it ;)

Do You REALLY Want to Program Your Network?

The primary benefit of SDN (as claimed by its promoters) is a geek’s dream: the ability to program the network (or, more precisely, control network’s behavior through programmable tools).

Sounds great – we’ll finally be able to fix that pesky detail the vendor never wanted to implement (probably for a good reason). But should we REALLY do that?

SIIT – The Magic Behind IPv6-only Data Center

Remember Tore Anderson’s IPv6-only data center design he described in last June’s webinar? Wondered how he got it done? The secret sauce he used is SIIT – the stateless IPv6-to-IPv4 translation technology. His trick: turning it around.

Layer-2 Extension (OTV) Use Cases

I was listening to the fantastic OTV Deep Dive PQ Packet Pushers podcast while biking around the wonderful Slovenian forests. They started the podcast by discussing OTV use cases, Ethan throwing in long-distance vMotion (the usual long-distance L2 extension selling point), but refreshingly some of the engineers said “well, that’s not really the use case we see in real life.”

So what were the use cases they were mentioning?

Plexxi PSI: MAU at Gigabit Speed

Regardless of the advantages of photonic switching (David Husak claims it’s 20.000 times more effective than electronic switching), the programmable optical components remain ludicrously expensive, prompting Plexxi to launch a cost-optimized fixed-topology version of their data center products.

Monitor Public SaaS Providers with ThousandEyes

If you’ve ever tried to troubleshoot web application performance issues, you’ve probably seen it all – browser waterfall diagrams, visual traceroute tools, network topologies produced by network management systems … but I haven’t seen them packaged in a comprehensive, easy-to-use and visually compelling package before. Welcome to ThousandEyes.

Test virtual appliance throughput with Spirent Avalanche NEXT

During the Networking Tech Field Day 6 Spirent showed us Avalanche NEXT – another great testing tool that generates up to 10Gbps of perfectly valid application-level traffic that you can push through your network devices to test their performance, stability or impact of feature mix on maximum throughput.

Not surprisingly, as soon as they told us that you could use Avalanche NEXT to replay captured traffic we started getting creative ideas.

Dizzy from the Kool-Aid

I spent most of the last week in day-long geek parties better known as Networking Tech Field Day. Talking with startups doing cool things or vendors launching cool products, and bouncing barely-feasible ideas off fellow geeks is intoxicating. No wonder I was so excited after the previous events.

Migrating a cold VM into a foreign subnet

Moving a running VM into a foreign subnet is Mission Impossible due to stale ARP entries (anyone telling you otherwise is handwaving over a detail or two - maybe their VM doesn't communicate with other VMs in the same subnet), but it's entirely feasible to migrate a cold VM into a foreign subnet if you can fix IP routing. Here's how you can do the trick with Enterasys switches.

Overlay Networks and QoS FUD

One of the usual complaints I hear whenever I mention overlay virtual networks is “with overlay networks we lose all application visibility and QoS functionality” ... that worked so phenomenally in the physical networks, right?

How Much Data Center Bandwidth Do You Really Need?

Networking vendors are quick to point out how the opaqueness (read: we don’t have the HW to look into it) of overlay networks presents visibility problems and how their favorite shiny gizmo (whatever it is) gives you better results (they usually forget to mention the lock-in that it creates).

Now let’s step back and ask a fundamental question: how much bandwidth do we need?

Why Is Network Virtualization So Hard?

We’ve been hearing how the networking is the last bastion of rigidity in the wonderful unicorn-flavored virtual world for the last few years. Let’s see why it’s so much harder to virtualize the networks as opposed to compute or storage capacities (side note: it didn’t help that virtualization vendors had no clue about networking, but things are changing).

OpenFlow Fabric Controllers Are Light-years Away from Wireless Ones

When talking about OpenFlow and the whole idea of controller-based networking, people usually say “well, it’s nothing radically new, we’ve been using wireless controllers for years and they work well, so the OpenFlow ones will work as well.”

Unfortunately the comparison is totally misleading.

DF bit explained

Finally a good explanation of what DF bit does ;)

Layer-2 DCI with Enterasys Switches

The second half of the Enterasys DCI Solutions webinar focused on real-life case studies. First the less interesting one: long-distance live VM migration (you know my feelings about the whole concept, but sometimes you just have to do it) and the role of fabric routing and host routing in the process.

Chatting about VMware NSX

Matt Oswalt and Matthew Stone kindly invited me to an interesting NSX-focused chat that turned into an hour-long podcast. Here are the results. Enjoy!

Sooner or later someone will pay for the complexity of the kludges you use

I loved listening to OTV/FabricPath/LISP Packet Pushers podcast. Ron Fuller and Russ White did a great job explaining the role of OTV, FabricPath and LISP in a stretched (inter-DC) subnet deployment scenario and how the three pieces fit together … but I couldn't stop wondering whether there is a better method to solve the underlying business need than throwing three new pretty complex technologies and associated equipment (or VDC contexts or line cards) into the mix.

Extending Layer-2 Connection into a Cloud

Carlos Asensio was facing an “interesting” challenge: someone has sold a layer-2 extension into their public cloud to one of the customers. Being a good engineer, he wanted to limit the damage the customer could do to the cloud infrastructure and thus immediately rejected the idea to connect the customer straight into the layer-2 network core ... but what could he do?

The Plexxi Challenge (or: Don’t Blame the Tools)

Plexxi has an incredibly creative data center fabric solution: they paired data center switching with CWDM optics, programmable ROADMs and controller-based traffic engineering to get something that looks almost like distributed switched version of FDDI (or Token Ring for the FCoTR fans). Not surprisingly, the tools we use to build traditional networks don’t work well with their architecture.

In a recent blog post Marten Terpstra hinted at shortcomings of Shortest Path First (SPF) approach used by every single modern routing algorithm. Let’s take a closer look at why Plexxi’s engineers couldn’t use SPF.

Combining DMVPN with Existing MPLS/VPN Network

One of the Expert Express sessions focused on an MPLS/VPN-based WAN network using OSPF as the routing protocol. The customer wanted to add DMVPN-based backup links and planned to retain OSPF as the routing protocol. Not surprisingly, the initial design had all sorts of unexpectedly complex kludges (see the case study for more details).

Having a really smart engineer on the other end of the WebEx call, I had to ask a single question: “Why don’t you use BGP everywhere” and after a short pause got back the expected reply “wow ... now it all makes sense.”

Enterasys Host Routing – Optimal L3 Forwarding with VM Mobility

I spent the last few weeks blogging about the brave new overlay worlds. Time to return to VLAN-based physical reality and revisit one of the challenges of VM mobility: mobile IP addresses.

A while ago I speculated that you might solve inter-subnet VM mobility with Mobile ARP. While Mobile ARP isn’t the best idea ever invented it just might work reasonably well for environments with dozens (not millions) of virtual servers.

Enterasys decided to go down that route and implement host routing in their data center switches. For more details, watch the video from the Enterasys DCI webinar.

Virtual Appliance Routing – Network Engineer’s Survival Guide

Routing protocols running on virtual appliances significantly increase the flexibility of virtual-to-physical network integration – you can easily move the whole application stack across subnets or data centers without changing the physical network configuration.

Major hypervisor vendors already support the concept: VMware NSX Edge Services Router can run OSPF, BGP or IS-IS, and BGP is coming to Hyper-V gateways. Like it or not, we’ll have to accept these solutions in the near future – here’s a quick survival guide.

Are Overlay Networking Tunnels a Scalability Nightmare?

Every time I mention overlay virtual networking tunnels someone starts worrying about the scalability of this approach along the lines of “In a data center with hundreds of hosts, do I have an impossibly high number of GRE tunnels in the full mesh? Are there scaling limitations to this approach?

Not surprisingly, some ToR switch vendors abuse this fear to the point where they look downright stupid (but I guess that’s their privilege), so let’s set the record straight.

Routing Protocols on NSX Edge Services Router

VMware gave me early access to NSX hands-on lab a few days prior to VMworld 2013. The lab was meant to demonstrate the basics of NSX, from VXLAN encapsulation to cross-subnet flooding, but I quickly veered off the beaten path and started playing with routing protocols in NSX Edge appliances.

What is VMware NSX?

Answer#1: An overlay virtual networking solution providing logical bridging (aka layer-2 forwarding or switching), logical routing (aka layer-3 switching), distributed or centralized firewalls, load balancers, NAT and VPNs.

Answer#2: A merger of Nicira NVP and VMware vCNS (a product formerly known as vShield).

Oh, and did I mention it’s actually two products, not one?

SDDC Interview on SDNCentral

SDN Central published a short SDDC-focused interview with Brent Salisbury and myself. Enjoy ... and don't forget to register for the SDDC symposium if you're in Silicon Valley in September.

Design Options in Dual-Stack Data Centers

Tore Anderson started his part of the IPv6-Only Data Centers webinar with a comprehensive analysis of numerous design options you have when implementing dual-stack access to your data center.

Unless you decided to live under a rock for the next 20 years or plan to drop out of networking in the very near future, you simply (RFC 2119) MUST watch this video.

50 Shades of Statefulness

A while ago Greg Ferro wrote a great article describing integration of overlay and physical networks in which he wrote that “an overlay network tunnel has no state in the physical network”, triggering an almost-immediate reaction from Marten Terpstra (of RIPE fame, now @ Plexxi) arguing that the network (at least the first ToR switch) knows the MAC and IP address of hypervisor host and thus has at least some state associated with the tunnel.

Marten is correct from a purely scholastic perspective (using his argument, the network keeps some state about TCP sessions as well), but what really matters is how much state is kept, which device keeps it, how it’s created and how often it changes.

How big is a big private cloud?

During the UCS Director Overview Packet Pushers Podcast I listened to recently the participants started discussing the use cases and someone mentioned that UCS Director might not be applicable for small shops with only a few thousand VMs. Let's put that in perspective.

Networking Enhancements in Windows Server 2012 R2

The “What’s coming in Hyper-V Network Virtualization (Windows Server 2012 R2)” blog post got way too long, so I had to split it in two parts: Hyper-V Network Virtualization and the rest of the features (this post).

Nicira NVP Control Plane

In the previous posts I described how a typical overlay virtual networking data plane works and what technologies vendors use to implement the associated control plane that maps VM MAC addresses to transport IP addresses. Now let’s walk through the details of a particular implementation: Nicira Network Virtualization Platform (NVP), part of VMware NSX.

Skip the transitions with IPv6-only data center deployment

Before Tore Anderson, the rock star behind the IPv6-only data center, started explaining the interesting details of his ideas, I did a short intro explaining the need for IPv4+IPv6 access to your content and the steps you have to take to get there.

You might decide to proceed down the more traditional path (doing 5-6 transitions in the next few years) or deploy IPv6-only data center and be done with it.

2013-08-18: Fixed the video links

RSVP over DMVPN

A while ago Tomasz Kacprzynski asked me whether I'd ever run RSVP over DMVPN. I hadn't - after all, you'd only need that in VoIP environments and I try to stay as far away from voice as possible.

In the meantime, Tomasz solved the problem (short summary: you have to turn Phase 3 DMVPN into Phase 2 DMVPN) and wrote a lengthy blog post describing the problem (RSVP split horizon rule) and his solution (including numerous debugging printouts). Definitely worth reading if there's a non-zero chance you'll have to get the two working together.

We should teach the network how to serve the applications. Really?

In a recent blog post Marten Terpstra wrote:

We are teaching our applications how to behave uniformly. Or normal. And that's not normal. We should teaching the network how to serve the applications instead. However demanding or quirky they decide to be.

That’s definitely a noble engineering goal, the “only” problem is that I don’t know many customers who would be willing to foot the bill.

Control Plane Protocols in Overlay Virtual Networks

Multiple overlay network encapsulations are nothing more than a major inconvenience (and religious wars based on individual bit fields close to meaningless) for anyone trying to support more than one overlay virtual networking technology (just ask F5 or Arista).

The key differentiator between scalable and not-so-very-scalable architectures and technologies is the control plane – the mechanism that maps (at the very minimum) remote VM MAC address into a transport network IP address of the target hypervisor (see A Day in a Life of an Overlaid Virtual Packet for more details).

Management, Control and Data Planes in Network Devices and Systems

Every single network device (or a distributed system like QFabric) has to perform at least three distinct activities:

  • Process the transit traffic (that’s why we buy them) in the data plane;
  • Figure out what’s going on around it with the control plane protocols;
  • Interact with its owner (or NMS) through the management plane.

What’s Coming in Hyper-V Network Virtualization (Windows Server 2012 R2)

Right after Microsoft’s TechEd event CJ Williams kindly sent me links to videos describing new features in upcoming Windows Server (and Hyper-V) release. I would strongly recommend you watch What’s New in Windows Server 2012 R2 Networking and Deep Dive on Hyper-V Network Virtualization in Windows Server 2012 R2, and here’s a short(er) summary.

This blog post is describing futures that will ship in 2H2013. However, as all the videos mentioned above included live demos, and the preview release shipped on June 24th, it’s obvious they’re past the “it works so great in PowerPoint” stage.

All it takes is a single misdirected STP packet ...

... and the rest is history ;)

Optimal Layer-3 Forwarding with Active/Active VRRP (Enterasys Fabric Routing)

Enterasys implemented optimal layer-3 forwarding with an interesting trick: they support VRRP like any other switch vendor, but allow you to make all members of a VRRP group active forwarders regardless of their status.

Apart from a slightly more synchronized behavior, their implementation doesn’t differ much from Arista’s Virtual ARP, and thus shares the same design and deployment caveats.

For more information, watch the Fabric Routing video from the Enterasys Robust Data Center Interconnect Solutions webinar.

3 & 5 Years Ago (July 2013)

In July 2008 I was writing about ARP tables, router configuration partitioning, QoS Policing in Cisco IOS and unequal-bandwidth EBGP load balancing.

July 2010 was all about differences: between bridging and routing (part II), IP and MPLS and TCP/IP and mainframe.

A Day in a Life of an Overlaid Virtual Packet

I explain the intricacies of overlay network forwarding in every overlay-network-related webinar (Cloud Computing Networking, VXLAN deep dive, Overlay Virtual Networking, VMware NSX Architecture), but never wrote a blog post about them. Let’s fix that.

First of all, remember that most mainstream overlay network implementations (Cisco Nexus 1000V, VMware vShield, Microsoft Hyper-V) don’t change the intra-hypervisor network behavior: a virtual machine network interface card (VM NIC) is still connected to a layer-2 hypervisor switch. The magic happens between the internal layer-2 switch and the physical (server) NIC.

Can I Use Shared (RFC 6598) IPv4 Address Space Within My Network?

Andrew sent me the following question: “I'm pushing to start a conversation about IPv6 in my organization, but meanwhile I've no RFC 1918 space left. What's your take on 100.64.0.0/10 - it's seems like this is available for RFC 1918 purposes, even if not intentionally?

Short answer: Don’t even think about that!

Temper Your MacGyver Streak

Microseconds after VXLAN was launched at VMworld 2011, someone started promoting it as a data center extension solution. Even though layer-2 DCI doesn’t make much sense (even to server people) and VXLAN is really not a DCI solution, the lure of misusing a technology was irresistible.

Dual-Stack Security Exposures

Dual-stack exposures were the last topic Eric Vyncke and myself addressed in the IPv6 security webinar. They range from missing ip6tables on Linux hosts to unintentional split-tunnel VPNs and missing access classes on Cisco IOS devices.

September: The Month of Overlay Networks

September definitely seems to be the month of overlay networks (at least from my perspective). The fun starts on September 4th with the Overlay Virtual Networking webinar which will describe overlay networking solutions from (in alphabetical order) Cisco, Microsoft, Midokura, OpenStack, VMware, and whichever startup sends me product documentation of a shipping product in the next two weeks.

3 & 5 Years Ago (June 2013)

Top hits of June 2008: explicit IPv4 address family in BGP configuration and static DHCP assignment for clients without client-id.

Two evergreens from June 2010 that still generate loads of traffic: Manipulating EIGRP metrics and GETVPN in 20 seconds.

And here are the other popular posts published a bit more than three and five years ago.

Update: Migrating my blog to ipspace.net

Update 2013-07-30 13:30 GMT - domain switched over, everything should be back to normal within an hour.

In the next 24 hours (to make sure all high-TTL DNS records expire) I'll migrate blog.ioshints.info to blog.ipspace.net. The old domain will remain active with permanent redirects to the new domain so you shouldn't have to change anything on your end. However, if you experience any problems access the blog or the main web site (www.ipspace.net) please let me know. Thank you!

More private AS numbers

Have you ever tried to implement a large-scale DMVPN or MPLS/VPN network using BGP as the routing protocol? If you tried to stitch more than ~1000 sites together you’re well aware of all the pain caused by a small range of private AS numbers defined in RFC 1930. We can kludge our way around the limitation by reusing the same AS number on multiple sites (and using allowas-in when we need full routing information on every site), but such a design clearly sucks.

Cloud-as-an-Appliance Design

The original idea behind cloud-as-an-appliance design came from Brad Hedlund’s blog post in which he described how he’d build a greenfield Hadoop or private cloud cluster with servers connected to a Clos fabric. Throw virtual appliances into the mix and you get an extremely simple and versatile architecture:

Live VM Mobility Network Requirements

Every time someone mentions how awesome new technologies solve live VM mobility across WAN networks, I start muttering unmentionables. Live VM mobility across disjoint layer-2 subnets works great in demos, but usually fails in real life due to stale ARP caches. The only way to solve this problem for good is to implement EC2-like layer-3 forwarding in hypervisor soft switches.

Update: LISP Host Mobility seems to be a potential exception; see the comment from Nico.

For more details, watch the VM Mobility Requirements video (part of Enterasys-sponsored DCI webinar), read the Hot and Cold VM Mobility blog post or watch the recording of NFD4 session with Cisco’s Victor Moreno.

Unreadable IPv6 Addresses Might Be Good For Us in the Long Run

One of the first arguments used by networking engineers living in IPv6 denial and trying to justify their stance is “IPv6 addresses are unreadable. We will never migrate to IPv6; it’s much easier to deal with IPv4 addresses.”

That’s absolutely true. If you use RFC 1918 addresses in a small(ish) network, the first two octets don’t change, and it’s easy to remember the remaining two numbers … but the unreadable IPv6 addresses just might change the way we approach network configuration and monitoring.

3 & 5 Years Ago (May 2013)

In May 2008 I wrote about asymmetric routing, OSPF across a PIX/ASA firewall and control plane protection.

The most popular article from May 2010 is still Tunneling VPNs and Zone-Based Firewalls with NAT64 and DNS64 in 30 minutes being close second. I also wrote about iSCSI and Moore’s Law and IPv6 myths.

Where’s the Revolutionary Networking Innovation?

In his recent blog post Joe Onisick wrote “What network virtualization doesn’t provide, in any form, is a change to the model we use to deploy networks and support applications. [...] All of the same broken or misused methodologies are carried forward. [...] Faithful replication of today’s networking challenges as virtual machines with encapsulation tunnels doesn’t move the bar for deploying applications.

Unicast-Only VXLAN Finally Shipping

The long-promised unicast-only VXLAN has finally shipped with the Nexus 1000V release 4.2(1)SV2(2.1) (there must be some logic behind those numbers, but they all look like madness to me). The new Nexus 1000V release brings two significant VXLAN enhancements: unicast-only mode and MAC distribution mode.

Downloadable Recording of Enterasys Data Center Interconnect Solutions Webinar

The recording of recent Enterasys Robust Data Center Solutions is available on ipSpace.net demo web site.

You can watch (or download) the following videos:

First-Hop IPv6 Security Features in Cisco IOS

I wanted to figure out how to use IPv6 DAD proxy in PVLAN environments during my seaside vacations, and as I had no regular Internet access decided to download the whole set of IPv6 configuration guides while enjoying the morning cup of coffee in an Internet café. Opening the IPv6 First-Hop Security Configuration Guide was one of the most pleasant (professional) surprises I had recently.

One word summary: Awesome.

3 & 5 Years Ago (April 2013)

Not surprisingly, the top posts from April 2008 include BGP essentials: Non-transit AS, What is CLNS? and The impact of tx-ring-limit.

April 2010 was a bit more introspective – I wrote about how networking is like physics or math, not history and small steps to large complexity.

Doodling on a Napkin

Two more books you must read during your summer vacation: The Back of the Napkin and Unfolding the Napkin ... and here are a few drawing tools that work for me.

If you’re anything like me, your first results will be disastrous. Keep practicing; I’m able to draw understandable (if not pretty) diagrams by now (and I’m still trying to stay away from Pictionary).

Smart Fabrics Versus Overlay Virtual Networks

With the recent plethora of overlay networking startups and Cisco Live Dynamic Fabric Architecture announcements it’s time to revisit a blog post I wrote a bit more than a year ago, comparing virtual networks and voice technologies.

They say a picture is worth a thousand words – here are a few slides from my Interop 2013 Overlay Virtual Networking Explained presentation.

ProgrammableFlow Typical Use Cases

The last part of the ProgrammableFlow webinar described typical use cases including Cloud-as-an-Appliance, traffic steering (used by appliances like Radware DefenseFlow) and hypervisor switching with PF1000. Predictably, the use cases were followed by a lengthy Q&A session.

The Tools That I Use (Drawings)

Continuing from the previous Tools That I Use post, here’s what I’m using to generate the hand drawings in blog posts and case studies.

3 & 5 Years Ago (March 2013)

Summer seems to be the perfect time to revisit old posts. Here are a few from 2008 and 2010.

The top hit of March 2008 was the UDP flood in Perl closely followed by the mysteries of the “Internet” BGP community. I was also writing about tracking the DHCP default route and detecting router restarts.

March 2010 was way more interesting: I was writing about MPLS TE autoroute basics, the difference between CLNS and CLNP, QPPB in MPLS VPN networks and IPv6 myths.

Cloudbursting, the Wally Way

Priceless! (source: Dilbert.com)

iSCSI with PFC?

Nicolas VermandĂ© sent me a really interesting question: “I've been looking for answers to a simple question that even different people at Cisco don't seem to agree on: Is it a good idea to class IP traffic (iSCSI or NFS over TCP) in pause no-drop class? What is the impact of having both pauses and TCP sliding windows at the same time?

IPv6 Address Assignment and Tracking

One of the significant challenges of IPv6 is definitely the host address assignment and tracking (for logging/auditing reasons), more so if you use SLAAC or (even worse) SLAAC privacy extensions. Not surprisingly, Eric Vyncke and myself spent a significant amount of time addressing this topic in the IPv6 Security webinar.

Fallacies of GUI

I love Greg Ferro’s characterization of CLI:

We need to realise that the CLI is a “power tools” for specialist tradespeople and not a “knife and fork” for everyday use.

However, you do know that most devices’ GUI offers nothing more than what CLI does, don’t you? Where’s the catch?

Summer seems to have arrived

The current weather around Central Europe doesn’t exactly support this conclusion, but I do get many more “I’m on vacation” responses than usual, so it’s time to reduce the blogging frequency to keep your RSS reader from overloading (you did switch from Google Reader to something like Feedly, didn’t you?).

However, if you’re looking for some really heavy reading, do pick up The Hidden Reality and explore various multiverse proposals. There’s also a beach-friendly version of multiverse discussion: The Long Earth by the one-and-only Terry Pratchett.

Data Center Fabrics Built with Plexxi Switches

During the recent Data Center Fabrics Update webinar Dan Backman from Plexxi explained how their innovative use of CWDM technology and controller-assisted forwarding simplifies deployment and growth of reasonably-sized data center fabrics.

I would highly recommend that you watch the video – the start is a bit short on details, but he does cover all the juicy aspects later on.

Real-Life SDN/OpenFlow Applications

NEC and a slew of its partners demonstrated an interesting next step in the SDN saga @ Interop Las Vegas 2013: multi-vendor SDN applications. Load balancing, orchestration and security solutions from A10, Silver Peak, Red Hat and Radware were happily cooperating with ProgrammableFlow controller.

A curious mind obviously wants to know what’s behind the scenes. Masterpieces of engineering? Large integration projects ... or is it just a smart application of API glue? In most cases, it’s the latter. Let’s look at the ProgrammableFlow – Radware integration.

The Tools That I Use (Webinars)

Andreas was watching my recent Enterasys DCI webinar and got intrigued by the quick hand drawings I made, so he asked me “What kind of tool do you use to make the hand drawings during your presentations? It must be something different than a mouse.

In case you weren’t watching one of my recent webinars, here’s a sample to get you started:

CLI and API Myths

Greg Ferro published a great blog post explaining why he decided to use node.js to build his cloud automation platform. While I agree with most things he wrote, this one prickled me the wrong way:

In my view, an Application Programmable Interface(API) is the fundamental change that makes Software Defined Networking (SDN) a “thing”. We need to realise that the CLI is a “power tools” for specialist tradespeople and not a “knife and fork” for everyday use.

While I agree with his view on CLI, keep in mind that API is no different.

EIBGP Load Balancing

The next small step in my MPLS/VPN update project: EIBGP load balancing – why is it useful, how it works, why can’t you use it without full-blown MPLS/VPN, and what the alternatives are.

MPLS/VPN Carrier’s Carrier – Myth or Reality?

Andrew is struggling with MPLS/VPN providers and sent me the following question:

Is "carriers carrier" a real service? I'm having a bit of an issue at the moment with too many MPLS providers […] Carrier’s carrier would be an answer to many of them, but none of the carriers admit to being able to do this, so I was wondering if it's simply that I'm speaking to the wrong people, or whether they really don't...

Short answer: I have yet to see this particular unicorn roaming the meadows of reality.

Arista EOS Virtual ARP (VARP) Behind the Scenes

In the "Optimal L3 Forwarding with VARP and Active/Active VRRP" blog post I made a remark along the lines of "Things might get nasty [in Arista EOS Virtual ARP world] if you have configuration mismatches", resulting in a lengthy and amazingly insightful email exchange with Lincoln Dale during which we ventured deeper and deeper down the Virtual ARP (VARP) rabbit hole. Here's what I learned during out trip:

Implementing Control-Plane Protocols with OpenFlow

The true OpenFlow zealots would love you to believe that you can drop whatever you’ve been doing before and replace it with a clean-slate solution using dumbest (and cheapest) possible switches and OpenFlow controllers.

In real world, your shiny new network has to communicate with the outside world … or you could take the approach most controller vendors did, decide to pretend STP is irrelevant, and ask people to configure static LAGs because you’re also not supporting LACP.

Network Virtualization and Spaghetti Wall

I was reading What Network Virtualization Isn’t from Jon Onisick the other day and started experiencing all sorts of unpleasant flashbacks caused by my overly long exposure to networking industry missteps and dead ends touted as the best possible solutions or architectures in the days of their glory:

Will SPDY Solve Web Application Performance Issues?

In the TCP, HTTP and SPDY webinar I described the web application performance roadblocks caused by TCP and HTTP and HTTP improvements that remove most of them. Google went a step further and created SPDY, a totally redesigned HTTP. What is SPDY? Is it really the final solution? How much does it help? Hopefully you’ll find answers to some of these questions in the last part of the webinar.

The whole webinar is also available on Udemy - it’s free but you’ll have to register (or log in with Facebook) to get access.

The Difference between Access Lists and Prefix Lists

A while ago someone asked what the difference between access and prefix lists is on the Network Engineering Stack Exchange web site (a fantastic resource brought to life primarily by sheer persistence of Jeremy Stretch, who had to fight troves of naysayers with somewhat limited insight claiming everything one would want to discuss about networking falls under server administration web site).

The question triggered a lengthy wandering down the memory lane … and here's the history of how the two came into being (and why they are the way they are).

Response: SDN’s Casualties

An individual focused more on sensationalism than content deemed it appropriate to publish an article declaring networking engineers an endangered species on an industry press web site that I considered somewhat reliable in the past.

The resulting flurry of expected blog posts included an interesting one from Steven Iveson in which he made a good point: it’s easy for the cream-of-the-crop not to be concerned, but what about others lower down the pile. As always, it makes sense to do a bit of reality check.

Watch Tore Anderson Describing IPv6-Only Data Centers

Could you run a data center exclusively on IPv6? What would you need to do to interact with the IPv4 side of the Internet? Tore Anderson from Redpill Linpro figured out that IPv4 addresses are better used to enable the service for the servers hosted in their datacenter instead of “wasting” them for network infrastructure and data center connectivity, and he’ll describe how he’s doing it (in production environment) in tomorrow’s free IPv6-Only Data Centers webinar.

Dynamic Routing with Virtual Appliances

Meeting Brad Hedlund in person was definitely one of the highlights of my Interop 2013 week. We had an awesome conversation and quickly realized how closely aligned our views of VLANs, overlay networks and virtual appliances are.

Not surprisingly, Brad quickly improved my ideas with a radical proposal: running BGP between the virtual and the physical world.

Long Live Just Good Enough!

Today's Dilbert is dedicated to every networking and security vendor selling us just good enough solutions.

Huge "Thank you!" to Scott Adams for another well-explained documentary!

IPv6 uRPF and Neighbor Discovery Throttling

IPv6 source address spoofing should be old news – it’s no different from its IPv4 counterpart. Neighbor discovery exhaustion attack is an IPv6-only phenomenon, enabled by huge IPv6 subnet sizes.

During the IPv6 Security webinar Eric Vyncke described Cisco IOS mechanisms you can use to cope with both. Enjoy!

Dirty Details of Inter-DC VM- and IP Address Mobility

The upcoming Data Center Interconnect webinar (register) is sponsored by Enterasys Networks, so it’s obvious that I’ll also mention how you can use their technology to solve particular data center interconnect problems, but that’s not all. The webinar will focus primarily on Whys, Hows and Whats of solving VM- and IP address mobility challenges in multi-data center environments.

Here are a few of the topics we’ll cover:

Network Virtualization at ToR switches? Makes as much sense as IP-over-APPN

One of my blogger friends sent me an interesting observation:

After talking to networking vendors I'm inclined to think they are going to focus on a mesh of overlays from the TOR, with possible use of overlays between vswitch and TOR too if desired - drawing analogies to MPLS with ToR a PE and vSwitch a CE. Aside from selling more hardware for this, I'm not drawn towards a solution like this bc it doesn't help with full network virtualization and a network abstraction for VMs.

The whole situation reminds me of the good old SNA and APPN days with networking vendors playing the IBM part of the comedy.

VRRP, Anycasts, Fabrics and Optimal Forwarding

The Optimal L3 Forwarding with VARP/VRRP post generated numerous comments, ranging from technical questions about VARP (more about that in a few days) to remarks along the lines of “you can do that with X” or “vendor Y supports Z, which does the same thing.” It seems I’ve opened yet another can of worms, let’s try to tame and sort them.

Scott Shenker on OpenFlow and SDN

Brent Salisbury sent me a link to a fantastic OpenFlow/SDN presentation Scott Shenker did @ Stanford University a few days ago. It’s a perfect introduction to the fundamental ideas behind SDN and therefore a must-see for everyone vaguely involved in networking.

Here are some of the highlights (from my highly biased perspective):

What is Network Virtualization

Brad Hedlund wrote another great article, this one explaining the fundamentals of network virtualization. As you'll see, VMware (and everyone else) aims way higher than replacing VLANs with overlay networks. Highly recommended!

Simplify Your Disaster Recovery with Virtual Appliances

Regardless of what the vendors are telling you, it’s hard to get data center disaster recovery right (unless you’re running regular fire drills), and your job usually gets harder due to the intricate (sometimes undocumented) intertwining of physical and virtual worlds. For example, do you know how to get the firewall and load balancer configurations from the failed site implemented in the equipment currently used at disaster recovery site?

Imagine a simple application stack with a few web servers, app servers and two database servers. There’s a firewall in front of the web servers and a load balancer tying all the segments together.

BGP Best External Explained

Loads of niche features got crammed into (MP)BGP and MPLS since I wrote my MPLS books, most of them trying to tweak BGP (a scalable and reasonably slow routing protocol dealing with behemoth tables) to behave more like an IGP would.

It looks like we’ll never see updated versions of the books, so I’ll try to cover the new features with short videos. The first one on the list: BGP Best External – a mechanism that speeds up MP-IBGP convergence in primary/backup PE-CE scenarios using EBGP.

Optimal L3 Forwarding with VARP and Active/Active VRRP

I’ve blogged about the need for optimal L3 forwarding across the whole data center almost a year ago when I introduced it as one of the interesting requirements in Data Center Fabrics webinar. A year later, there are still only a few companies that can deliver this functionality.

Could IXPs Use OpenFlow to Scale?

The SDN industry probably considers me an old and grumpy naysayer (and I’m positive Mrs Y has a special place in their hearts after her recent blog post), so I tried really hard to find a real-life example where OpenFlow could be used to solve mid-market innovator’s dilemma to balance my usual OpenFlow and SDN presentation.

Enhance Your Summer Bootcamp with ipSpace Recordings

The networking department at one of the leading US universities got a great idea last year: they were organizing a summer networking bootcamp for undergraduate students and decided to use some of my webinars as background/supplementary material. As we've been working with Central European universities for over a decade, we were able to add yet another product to the mix: IP Primer (introduction to TCP/IP) which includes hands-on exercises on live Cisco IOS routers.

Their idea was obviously pretty successful – they'll do the same thing this year. Would you like to do something similar? Contact me, tell me more about your project, and I'm positive we'll find a way to help you.

Hyper-V 3.0 Extensible Virtual Switch

It took years before the rumored Cisco vSwitch materialized (in form of Nexus 1000V), several more years before there was the first competitor (IBM Distributed Virtual Switch), and who knows how long before the third entrant (recently announced HP vSwitch) jumps out of PowerPoint slides and whitepapers into the real world.

Compare that to the Hyper-V environment, where we have at least two virtual switches (Nexus 1000V and NEC's PF1000) mere months after Hyper-V's general availability.

OpenFlow and SDN – Do You Want to Build Your Own Racing Car?

The OpenFlow zealots are quick to point out the beauties of the centralized control plane, and the huge savings you can expect from using commodity hardware and open-source software. What they usually forget to tell you is that you also have to reinvent all the wheels the networking industry has invented in the last 30 years.

Celebrating 40 years of Ethernet ... at south pole

Did you know Ethernet turned 40 today? I didn't (I was never good at tracking anniversaries), but Kris Amundson (the engineer keeping his network up and running in pitch dark Antarctica) quickly brought it to my attention with wonderful photos of South Pole Ethernet network built @ -69C (that's -92F if you're still ignoring the metric system).

Even better, they still have a thick coax cable with transceiver screwed into it!

Thanks for sharing, Kris! Really appreciated ;)

The Dangers of Ignoring IPv6

I was sitting next to a really nice security engineer during the fantastic dinner-in-a-wine-cellar @ Troopers 13 and as we started talking about security implications of ignoring IPv6, I was quickly able to persuade him that it's dangerous to pretend IPv6 doesn't exist and that even though you might choose not to deploy it, you still have to acknowledge it exists and take protective measures.

It’s always great fun to explain the dangers of ignoring IPv6 to a networking or security audience, and see some people muttering “oh, ****”

Multi-Vendor OpenFlow – Myth or Reality?

NEC demonstrated multi-vendor OpenFlow network @ Interop Las Vegas, linking physical switches from Arista, Brocade, Centec, Dell, Extreme, Intel and NEC, and virtual switches in Linux (OVS) and Hyper-V (PF1000) environments in a leaf-and-spine fabric controlled by ProgrammableFlow controller (watch the video of Samrat Ganguly demonstrating the network).

Does that mean we’ve entered the era of multi-vendor OpenFlow networking? Not so fast.

Troopers 13 – a must-visit security conference

If you live in Europe and happen to be interested in security, make sure you put Troopers on the list of must-attend events. Like many things coming from Europe it’s a boutique event (limited to 200 attendees even if it means it’s sold out – that would never happen in some other parts of the world) with some great content.

Enno Rey, the mastermind behind the event, was kind enough to invite me to talk about virtual firewall architectures – you can view my presentation or watch the video – and of course I used the opportunity to visit a not-so-well-known Heidelberg attraction ;)

Expert Beginners

Erik Dietrich obviously hates the self-proclaimed (usually clueless) “experts” – he devoted a whole series of blog posts to them:

I’m positive you know at least a few people that would match his descriptions. Enjoy!

How Networking Is Changing – Interview with Stu Miniman

Stu Miniman kindly invited me to do an interview for the SiliconANGLE during the Interop/EMC World week. Here are the results:

Tail-f Network Control System – the First Impressions

One of the most pleasant surprises of the recent Interop show was the Tail-f's Network Control System (NCS). I “knew” Carl Moberg (of the NETCONF and YANG fame) for a long time and had the privilege to meet him in person just before the SDN Buyer's Guide panel that I co-hosted with Kurt Marko (who did an excellent job putting the buyer's guide together). Anyhow, what Carl presented during the panel totally blew me away.

Data Has Mass and Gravity

A while ago, while listening to an interesting CloudCast podcast (my second favorite podcast - the best one out there is still the Packet Pushers), I stumbled upon an interesting idea “Data has gravity”. The podcast guest used that idea to explain how data agglomerates in larger and larger chunks and how it makes sense to move the data processing (application) closer to the data.