Category: IP routing

When describing Hyper-V Network Virtualization packet forwarding I briefly mentioned that the hypervisor switches create (an equivalent of) a host route for every VM they need to know about, prompting some readers to question the scalability of such an approach. As it turns out, layer-3 switches did the same thing under the hood for years.

If you’re building a Greenfield private cloud, you SHOULD consider using virtual network services appliances (firewalls, load balancers, IPS/IDS systems), removing the need for additional hard-to-scale hardware devices. But can we go a step further? Can we replace all networking hardware with x86 servers and virtual appliances?

The layer-3-only Hyper-V Network Virtualization forwarding model implemented in Windows Server 2012 R2 thoroughly confuses engineers used to deal with traditional layer-2 subnets connected via layer-3 switches.

As always, it helps to take a few steps back, focus on the principles, and the “unexpected” behavior becomes crystal clear.

2014-02-05: HNV routing details updated based on feedback from Praveen Balasubramanian. Thank you!

All overlay virtual networking solutions look similar from far away: many provide layer-2 segments, most of them have some sort of distributed layer-3 forwarding, gateways to physical world are ubiquitous, and you might find security features in some products.

The implementation details (usually hidden behind the scenes) vary widely, and I’ll try to document at least some of them in a series of blog posts, starting with VMware NSX.

Almost a year ago rumors started circulating about a Deutsche Telekom pilot network utilizing some crazy new optic technology. In spring I’ve heard about them using NFV and Tail-f NCS for service provisioning … but it took a few more months till we got the first glimpses into their architecture.

TL&DR summary: Good design always beats bleeding-edge technologies

TL&DR Summary: Yes (if you’re clumsy enough).

A while ago I read Impact of Graceful IGP Operations on BGP – an article that described how changes in IGP topology result in temporary (or sometimes even permanent) forwarding loops in networks using BGP route reflectors.

Is the problem real? Yes, it is. Could you generate a BGP RR topology that results in a permanent forwarding loop? Yes. It’s not that hard.

Tassos opened an interesting can of worms in a comment to my Management, Control and Data Planes post: Is ICMP response to a forwarded packet (TTL exceeded, fragmentation needed or destination unreachable) a control- or data-plane activity?

After we get rid of the QoS FUD, the next question I usually get when discussing overlay networks is “how should these networks treat IP TTL?”

As (almost) always, the answer is “It depends.”

I must have confused a few readers with my blog posts describing Arista’s VARP and Enterasys’ Fabric Routing – I got plenty of questions along the lines of “how does it really work behind the scenes?” Let’s shed some light on those dirty details.

Moving a running VM into a foreign subnet is Mission Impossible due to stale ARP entries (anyone telling you otherwise is handwaving over a detail or two - maybe their VM doesn't communicate with other VMs in the same subnet), but it's entirely feasible to migrate a cold VM into a foreign subnet if you can fix IP routing. Here's how you can do the trick with Enterasys switches.

I spent the last few weeks blogging about the brave new overlay worlds. Time to return to VLAN-based physical reality and revisit one of the challenges of VM mobility: mobile IP addresses.

A while ago I speculated that you might solve inter-subnet VM mobility with Mobile ARP. While Mobile ARP isn’t the best idea ever invented it just might work reasonably well for environments with dozens (not millions) of virtual servers.

Enterasys decided to go down that route and implement host routing in their data center switches. For more details, watch the video from the Enterasys DCI webinar.

Routing protocols running on virtual appliances significantly increase the flexibility of virtual-to-physical network integration – you can easily move the whole application stack across subnets or data centers without changing the physical network configuration.

Major hypervisor vendors already support the concept: VMware NSX-T edge nodes can run BGP or OSPF¹, and Hyper-V gateways can run BGP. Like it or not, we’ll have to accept these solutions in the near future – here’s a quick survival guide.

VMware gave me early access to NSX hands-on lab a few days prior to VMworld 2013. The lab was meant to demonstrate the basics of NSX, from VXLAN encapsulation to cross-subnet flooding, but I quickly veered off the beaten path and started playing with routing protocols in NSX Edge appliances.

Enterasys implemented optimal layer-3 forwarding with an interesting trick: they support VRRP like any other switch vendor, but allow you to make all members of a VRRP group active forwarders regardless of their status.

Apart from a slightly more synchronized behavior, their implementation doesn’t differ much from Arista’s Virtual ARP, and thus shares the same design and deployment caveats.

For more information, watch the Fabric Routing video from the Enterasys Robust Data Center Interconnect Solutions webinar.

A while ago someone asked what the difference between access and prefix lists is on the Network Engineering Stack Exchange web site (a fantastic resource brought to life primarily by sheer persistence of Jeremy Stretch, who had to fight troves of naysayers with somewhat limited insight claiming everything one would want to discuss about networking falls under server administration web site).

The question triggered a lengthy wandering down the memory lane … and here's the history of how the two came into being (and why they are the way they are).