Building Network Automation Solutions
6 week online course starting in September 2017

That’s It for 2014

A dozen webinars, tens of public presentations and on-site workshops, numerous highly interesting ExpertExpress sessions, three books and over 250 blog posts. That should be enough for a year; it’s time to go offline.

I hope your company has a New Year freeze (and not let’s upgrade everything over New Year policy), so you’ll be able to do the same and enjoy some time during the rest of the year with your loved ones. See you in 2015!

MPLS Tech Talks: RSVP-TE 101

After discussing the basics of MPLS, MPLS-TE and LDP, and the relationship between FECs, LDP and BGP, Seamus and myself focused on another interesting topic: how MPLS protocol stack uses RSVP to implement traffic engineering.

VRF Lite on Nexus 5600

One of the networking engineers using my ExpertExpress to validate their network design had an interesting problem: he was building a multi-tenant VLAN-based private cloud architecture with each tenant having multiple subnets, and wanted to route within the tenant network as close to the VMs as possible (in the ToR switch).

He was using Nexus 5600 as the ToR switch, and although there’s conflicting information on the number of VRFs supported by that switch (verified topology: 25 VRFs, verified maximum: 1000 VRFs, configuration guide: 64 VRFs), he thought 25 VRFs (tenant routing domains) might be enough.

Just Published: Scaling Overlay Virtual Networking Videos

The edited videos for Scaling Overlay Virtual Networking webinar are available on Content site. Nuage Networks sponsored the webinar; the videos are thus publicly available (without registration).

Webinars in 2014, and a Quick Peek Into 2015

I promise engineers who renew their subscription 4-6 new webinars a year. It’s time to see whether I kept that promise in 2014.

TL&DR summary: it was a great year, but I still missed a few things.

L2VPN over IPv6 with Snabb Switch on Software Gone Wild

Highly customizable high-speed virtual switch written in Lua sounds great, but is it really that easy to use? Simon Leinen was kind enough to get me in touch with Alex Gall, his colleague at Switch, who's working on an interesting project: implementing L2VPN over IPv6 with Snabb Switch.

Facebook Next-Generation Fabric

Facebook published their next-generation data center architecture a few weeks ago, resulting in the expected “revolutionary approach to data center fabrics” echoes from the industry press and blogosphere.

In reality, they did a great engineering job using an interesting twist on pretty traditional multi-stage leaf-and-spine (or folded Clos) architecture.

Performance Tests and Out-of-Box Performance

Simonp made a perfectly valid point in a comment to my latest OVS blog post:

Obviously the page you're referring to is a quick-and-dirty benchmark. If you wanted the optimal numbers, you would have to tune quite a few parameters just like for hardware benchmarks (sysctl kernel parameters, Jumbo frames, ...).

While he’s absolutely right, this is not the performance data a typical user should be looking for.

Last Call: Overlay Virtual Networks in Software Defined Data Centers

If you want to get a free copy of my Overlay Virtual Networks in Software-Defined Data Centers book, download it now. The offer will expire by December 15th.

Just published: Enterprise IPv6 videos

The edited videos for my Enterprise IPv6 webinars have been published on Enjoy!

Load Balancing in Google Network

Todd Hoff (of the HighScalability fame) sent me a link to an interesting video describing load-balancing mechanisms used at Google and how they evolved over time.

If the rest of the blog post feels like Latin, you SHOULD watch the Load Balancing and Scale-Out Application Architecture webinar.

The beginning of the story resembles traditional enterprise solutions:

Scaling Distributed Systems Is Hard

Stumbled upon a hilarious description of challenges encountered when trying to scale distributed systems (cluster of controllers running centralized control plane comes to mind).

It starts with “If someone tells you that scaling out a distributed system is easy they are either lying or drunk, and possibly both,” and gets better and better. Enjoy!

Hotel California Effects of Public Clouds

In his The Case for Hybrids blog post Mat Mathews described the Hotel California effect of public clouds as: “One of the most oft mentioned issues with public cloud is the difficulty in getting out.” Once you start relying on cloud provider APIs to provide DNS, load balancing, CDN, content hosting, security groups, and a plethora of other services, it’s impossible to get out.

Interestingly, the side effects of public cloud deployments extend into the realm of application programming, as I was surprised to find out during one of my Expert Express engagements.

Should I Really Program My Network?

In my presentation @ SDN Meetup in Stockholm, I tried to answer a simple question: “Should I really program my network?” and obviously had to start with an even simpler one: “What is SDN?

The video of the presentation is already available on YouTube, and you can watch the slides on my content web site.

Also, make sure you watch other presentations from that event, particularly David Barroso’s SDN Internet Router.

MPLS P-Router, Router or Layer-3 Switch?

One of my readers is struggling with the aftermath of marketing gimmicks:

We will be implementing a new network soon, and we're discussing P-routers versus regular routers versus switches. I'm looking for arguments to go one way or the other.

TL&DR: there’s no difference between router and L3 switch.

Can You AS-Prepend a Single Host Route?

Someone recently sent me this question:

Is it possible to prepend one IP address from a public IPv4 segment?

I don’t want to know what crazy stunt this engineer was forced to pull off, but just in case you land in a similar quandary here’s how you shoelace yourself out of it.

Thanks for being there!

A third of my readers are celebrating Thanksgiving today, and I’d like to use the opportunity to say what I always wanted to say but somehow never got to it. Let’s make it short: Thank you! Without you, there would be no

Transactional Thoughts on a Stormy Night

It was a dark stormy autumn night and three networking engineers had nothing better to do than ponder the heavy topics of transactional consistency in a distributed SDN environment in Episode 16 of Software Gone Wild podcast.

Here are a few of the topics that crossed our minds:

FECs, LDP and BGP in the MPLS World

After discussing the basics of MPLS and LDP in our Tech Talks chat, Seamus Gilchrist and myself focused on a concept that perplexes many networking engineers entering the MPLS world: the relationship between Forward Equivalence Classes (FEC), LDP and BGP.

Quick Peek: Juniper vMX Router

While the industry press deliberates the disaggregation of Arista and Cisco, and Juniper’s new CEO, Juniper launched a virtual version of its vMX router, which is supposed to have up to 160 Gbps of throughput (as compared to 10 Gbps offered by Vyatta 5600 and Cisco CSR). Can Juniper really deliver on that promise?

Moving Workloads to the Clouds

David Spark published 16 tips for moving your workloads to the clouds. Contrary to the usual useless nonsense coming down from hybrid cloud evangelists (you know, the people who moved from “VMs following the sun” to “seamless hybrid cloud workload mobility”) some of the tips actually make sense, starting with “Have a real reason for the migration”. Enjoy!

Open vSwitch Performance Revisited

A while ago I wrote about performance bottlenecks of Open vSwitch. In the meantime, the OVS team drastically improved OVS performance resulting in something that Andy Hill called Ludicrous Speed at the latest OpenStack summit (slide deck, video).

Let’s look at how impressive the performance improvements are.

Network Programmability Phase 2: the Provisioned Network

After describing the current state of affairs in his Network Programmability 101 webinar, Matt Oswald moved to the low-hanging fruits: automating repetitive tasks in baby steps, from VLAN provisioning to consistent device configurations.

Just Published: PCI DSS Videos

The edited videos of the fantastic PCI DSS webinar Michele Chubirka presented in early July have finally been published (yes, there’s a huge backlog that’s getting cleaned up). Enjoy!

Coping with Byzantine Routing Failures

One of my readers sent me an interesting challenge:

We have two MPLS providers sending us default routes and it seems like whenever we have problem with SP1 our failover is not happening properly and actually we have to go in manually and influence our traffic to forward via another path.

Welcome to the wondrous world of byzantine routing failures ;)

Do We Have Too Many Knobs?

The last day of Interop New York found me sitting in the Speaker Center with a few friends pondering the hype and reality of SDN and brokenness of traditional network products. One of the remarks during that conversation was very familiar: “we have too many knobs to configure”, and I replied “and how many knobs do you think there are in Windows registry?" (or Linux kernel and configuration files).

Just Published: Overlay Virtual Networks in Software Defined Data Centers

Overlay virtual networks are one of my favorite topics – it seems I wrote over a hundred blog posts describing various aspects of this emerging (or is it reinvented) technology since Cisco launched VXLAN in 2011.

During the summer of 2014 I organized my blog posts on overlay networks and SDDC into a digital book. I want to make this information as useful and as widely distributed as possible – for a limited time you can download the PDF free of charge.

Learn more about the book

Reinventing the wheel (or RFC 1925 sect 2.11)

Simon Wardley is another old-timer with low tolerance for people reinventing the broken wheels. I couldn’t resist sharing part of his blog post because it applies equally well to what we’re seeing in the SDN world:

No, I haven't read Gartner's recent research on this subject (I'm not a subscriber) and it seems weird to be reading "research" about stuff you've done in practice a decade ago (sounds familiar). Maybe they've found some magic juice? Experience however dictates that it'll be snake oil […]. I feel like the old car mechanic listening to the kid saying that his magic pill turns water into gas. I'm sure it doesn't ... maybe this time it will ... duh, suckered again.

Meanwhile the academics already talk about SDN 2.0.

Viptela SEN: Hybrid WAN Connectivity with an SDN Twist

Like many of us Khalid Raza wasted countless hours sitting in meetings discussing hybrid WAN connectivity designs using a random combination of DMVPN, IPsec, PfR, and one or more routing protocols… and decided to try to create a better solution to the problem.

Viptela Secure Extensible Network (SEN) doesn’t try to solve every networking problem ever encountered, which is why it’s simpler to use in the use case it is designed to solve: multi-provider WAN connectivity.

Just Published: VXLAN 2.0 Videos

Last week I ran the second part of the updated (4-hour) VXLAN webinar. The raw videos are already online and cover these topics:

  • VXLAN-related technologies, including encapsulation, IP multicast use, unicast VXLAN, and VXLAN-over-EVPN;
  • VXLAN implementations, including Cisco Nexus 1000v, VMware vCNS, VMware NSX, Nuage VSP and Juniper Contrail;
  • VXLAN gateways, including Arista, Brocade, Cisco and Juniper;
  • Hardware VTEP integration with OVSDB and EVPN;
  • VXLAN-based data center fabrics, including Cisco’s ACI.

Scaling the Cloud Security Groups

Most overlay virtual networking and cloud orchestration products support security groups more-or-less-statefulish ACLs inserted between VM NIC and virtual switch.

The lure of security groups is obvious: if you’re willing to change your network security paradigm, you can stop thinking in subnets and focus on specifying who can exchange what traffic (usually specified as TCP/UDP port#) with whom.

A Month of SDN

My calendar for the following four weeks is jam-packed with SDN events:

All the travel might affect my blogging frequency, but I still have a few podcasts in the editing queue, so you’ll have something to listen to in the meantime ;)

Handling the Bottom of MPLS Stack

MPLS bottom-of-stack bit confused one of my readers. In particular, he had a problem with the part where the egress MPLS Label Switch Router (LSR) should go from labeled (MPLS) to unlabeled (IPv4, IPv6) packets and has to figure out what’s in the packet.

Response: Midokura Open-Sources MidoNet

Last week Midokura decided to follow Juniper’s lead and make MidoNet an open-source product (after all, a similar move worked really well for Contrail, right?).

I may be too skeptical (again), but I fail to see how this could possibly work.

FlipIT Cloud: Orchestrating IT-as-a-Service on Software Gone Wild

Imagine being an IT administrator running a multi-tenant enterprise environment (example: an SMB business center). How many things would you have to configure to add a new tenant? How about adding a new user for an existing tenant?

The engineers behind the scenes of FlipIT cloud service ended up with a 40-page configuration guide when they started the service years ago… and obviously decided full-blown automation is the only way to go.

So You’re an Open Source Shop? Really?

I carried out an interesting quiz during one of my Interop workshop:

  • How many use Linux-based servers? Almost everyone raised their hands;
  • How many use Apache or Tomcat web servers? Yet again, almost everyone.
  • How many run applications written in PHP, Python, Ruby…? Same crowd (probably even a bit more).
  • How many use Nginx, Squid or HAProxy for load balancing? Very few.

Is there a rational explanation for this seemingly nonsensical result?

Just Published: HP and Dell Force10 Data Center Switches

Want to know how HP IRF works? What its limitations are? Which data center protocols HP 5900 supports? How Dell Force10 switches handle MLAG? How well are HP and Dell supporting OpenFlow?

You’ll get answers to all these questions in the videos recently published in the Data Center Fabric Architecture webinar (also available as part of yearly subscription).

Does a Cloud Orchestration System Need an Underlying SDN Controller?

A while ago I had an interesting discussion with a fellow SDN explorer, in which I came to a conclusion that it makes no sense to insert an overlay virtual networking SDN controller between cloud orchestration system and virtual switches. As always, I missed an important piece of the puzzle: federation of cloud instances.

2014-11-04 16:48Z: CJ Williams sent me an email with information on SDN controller in upcoming Windows Server release. Thank you!

Use a Disaster Recovery Project to Build Your New Cloud

It doesn’t make sense to build a new data center network to support legacy bare-metal server infrastructure. You’ll have to use relatively expensive 1G/10G ports to be able to connect the current and future servers, and once the server and virtualization engineers wake up and do hardware refresh you’ll end up with way too many ports (oh, and you do know that transceivers could cost more than the switching hardware, right?).

Make Rip-and-Replace a bit less Creepy

It was a dark and stormy Halloween night and a networking engineer was stuck in a data center facing a Mission Impossible project: replace a failing Cat6500 with a brand-new Nexus 7000. Shouldn’t have been a problem, if only the cables were labeled.

Overlay-to-Underlay Network Interactions: Document Your Hidden Assumptions

If you listen to the marketing departments of overlay virtual networking vendors, it looks like the world is a simple place: you deploy their solution on top of any IP fabric, and it all works.

You’ll hear a totally different story from the physical hardware vendors: they’ll happily serve you a healthy portion of FUD, hoping you swallow it whole, and describe in gory details all the mishaps you might encounter on your virtualization quest.

The funny thing is they’re all right (not to mention the really fun part when FUDders change sides ;).

Bad Ideas and Abominations

This post SHOULD have been published on April 1st, but I need to define the terminology for another upcoming post, so here it is ;)

RFC 2119 defines polite words to use when something really shouldn’t be done. Some network designs I see deserve more colorful terminology.

2014-11-02: Updated with reference to RFC 6919 (/HT to @LapTop006)

New Webinar: Scaling Overlay Virtual Networks

You can get an overlay virtual networking solution from almost every major hypervisor- and data center networking vendor. Do you ever wonder which one to choose for your large-scale environment? I’m positive you’d get all of them up and running in a one-rack environment, but what if you happen to be larger than that?

We’ll try to address scalability hiccups and roadblocks you might encounter on your growth path in Scaling Overlay Virtual Networks webinar (get your free ticket here).

Cumulus Linux in Real Life on Software Gone Wild

A year ago Matthew Stone first heard about Cumulus Linux when I ranted about it on a Packet Pushers podcast (which only proves that any publicity is good publicity even though some people thought otherwise at that time), and when his cloud service provider company started selecting ToR switches he considered Cumulus together with Cisco and Arista… and chose Cumulus.

Tech Talks: Introduction to Label Distribution Protocol (LDP)

In the third part of MPLS Tech Talks we focused on the role of label distribution protocol (LDP) and its operation in frame-mode MPLS. You can watch the video on the Tech Talks web page.

IPv6 in a Global Company – a Real-World Example

More than a year ago I wrote a response to a comment Pascal wrote on my Predicting the IPv6 BGP table size blog post. I recently rediscovered it and figured out that it’s (unfortunately) as relevant as it was almost 18 months ago.

Other people have realized we have this problem in the meantime, and are still being told to stop yammering because the problem is not real. Let’s see what happens in a few years.

All You Need Are Two Top-of-Rack Switches

Every time I’m running a classroom version of my Designing the Cloud Infrastructure workshop, I start with a simple question: “Who has more than 2000 VMs or bare-metal servers in the data center?

I might see three hands on a good day; 90-95% of the audience have smaller data centers… and some of them get disappointed when I tell them they don’t need more than two ToR switches in their data center.

Network Programmability Phase 1: the Configured Network

During his Network Programmability 101 webinar Matt Oswalt described three phases of network programmability. The first level in the pyramid of programmable awesomeness (his words, not mine) is described in today’s video.

Micro-BFD: BFD over LAG (Port Channel)

The discussion in the comments to my LAG versus ECMP post took a totally unexpected turn when someone mentioned BFD failure detection over port channels (link aggregation groups – LAGs).

What’s the big deal?

Just Published: Juniper Data Center Switches

Want to know what the difference between Virtual Chassis and Virtual Chassis Fabric is? How Local Link Bias works? How ISSU on QFX 5100 works even though the box doesn’t have two supervisor boards? You’ll find answers to all these questions in new videos describing Juniper data center switches.

Workload Mobility and Reality: Bandwidth Constraints

People talking about long-distance workload mobility and cloudbursting often forget the physical reality documented in the fallacies of distributed computing. Today we’ll focus on bandwidth, in a follow-up blog post we’ll deal with its ugly cousin latency.

TL&DR summary: If you plan to spread application components across the network without understanding their network requirements, you’ll get the results you deserve.

Border6 Non-Stop Internet: a Commercial BGP-Based SDN

Several SDN solutions that coexist with the traditional control- and data planes instead of ripping them out and replacing them with the new awesomesauce use BGP to modify the network’s forwarding behavior.

Border6 decided to turn that concept into a commercial product that we dissected in Episode 12 of Software Gone Wild podcast.

Enjoy the show (this time in video format).

Networking Is Not as Special as We Think It Is

I was listening to the Packet Pushers show #203 – an interesting high-level discussion of policies (if you happen to be interested in those things) – and unavoidably someone had to mention how the networking is all broken because different devices implement the same functionality in different ways and use different CLI/API syntax.

Last Call: Free Version of SDN and OpenFlow – The Hype and the Harsh Reality

If you want to get a free copy of my SDN and OpenFlow – The Hype and the Harsh Reality book, download it now. The offer will expire by October 20th.

Packet Reordering and Service Providers

My “Was it bufferbloat?” blog post generated an unexpected amount of responses, most of them focusing on a side note saying “it looks like there really are service providers out there that are clueless enough to reorder packets within a TCP session”. Let’s walk through them.

How to Get into the Top N%

Michael Church wrote an interesting answer on Quora, describing a logarithmic scale of programming skills and (even more importantly) hints to follow to get from n00b into the top N% (for some small value of N):

  • Budget 7–14 years;
  • Study voraciously;
  • Build things when you don’t know that you’ll succeed;
  • Network to get new ideas;
  • Job hop when you stop learning.

Replace “programmer” with “networking engineer” and read the whole answer ;)

IPv6 High Availability Strategies on NIL TV

I had a shorter version of my IPv6 High Availability talk @ Slovenian IPv6 summit this spring. The video is online, but wouldn’t be of much use to anyone but both Slovenian readers of this blog.

The English version of that same talk is now available on NIL TV (or you could decide to go for the full webinar or whole IPv6 track).

VXLAN and OTV: The Saga Continues

Randall Greer left a comment on my Revisited: Layer-2 DCI over VXLAN post saying:

Could you please elaborate on how VXLAN is a better option than OTV? As far as I can see, OTV doesn't suffer from the traffic tromboning you get from VXLAN. Sure you have to stretch your VLANs, but you're protected from bridging failures going over your DCI. OTV is also able to have multiple edge devices per site, so there's no single failure domain. It's even integrated with LISP to mitigate any sub-optimal traffic flows.

Before going through the individual points, let’s focus on the big picture: the failure domains.

Data Center Design Case Studies on Amazon – Take 2

In July I wrote about an Amazon Kindle version of my Data Center Design Case Studies book and complained about their royalties model. Someone quickly pointed out how to adapt to their system: split the book into multiple volumes and charge $9.99 for each.

It took me months to get there, but the first two volumes are finally on Amazon:

We Need Consistency more than Controllers

I was listening to the I2RS Packet Pushers podcast a while ago and was more than glad that when Greg Ferro yet again mentioned the complexity of OSPF, someone simply pointed out that controllers would not reduce the complexity; if anything they would increase it.

LAG versus ECMP

Bryan sent me an interesting question:

When you have the opportunity to use LAG or ECMP, what are some things you should consider?

He already gathered some ideas (thank you!) and I expanded his list and added a few comments.

Interop New York: It Was Great Fun

Last week’s Interop New York was hard work (three workshops in two days), but also lots of nerdy fun. I love doing workshops with smart participants who bring their real-life problems to the room and challenge my assumptions and conclusions, and I had plenty of these interactions during the week. Thank you all (you know who you are)!

Network Automation Tools with Jason Edelman on Sofware Gone Wild

The stars have finally aligned, and after months of scheduling Jason and myself found time to chat about network automation tools and all the other exciting things Jason is doing (and blogging about).

We started with easy topics:

Bufferbloat Killed my HTTP Session… or not?

Every now and then I get an email from a subscriber having video download problems. Most of the time the problem auto-magically disappears (and there’s no indication of packet loss or ridiculous latency in traceroute printout), but a few days ago Henry Moats managed to consistently reproduce the problem and sent me exactly what I needed: a pcap file.

TL&DR summary: you have to know a lot about application-level protocols, application servers and operating systems to troubleshoot networking problems.

Tech Talks: MPLS Traffic Engineering Basics

After covering the basics of MPLS, the discussion I had with Seamus Gilchrist turned to the basics of MPLS Traffic Engineering.

The video of that discussion is available online on the Tech Talks web page.

Replacing a Central Firewall

During one of my ExpertExpress engagements I got an interesting question: “could we replace a pair of central firewalls with iptables on the Linux server?

Short answer: Maybe (depending on your security policy), but I’d still love to see some baseline scrubbing before the traffic hits the server – after all, if someone pwns your server, he’ll quickly turn off iptables.

Building a Small Cloud with UCS Mini

During the last round of polishing of my Designing Infrastructure for Private Clouds Interop New York session (also available in webinar format) I wondered whether one could use the recently-launched UCS Mini to build my sample private cloud.

It’s the Application Development, Stupid

I love reading blog posts on Plexxi blog (you SHOULD add them to your RSS reader) and the “It’s the Application, Stupid” series from Mat Matthews is no exception. What pleasantly surprised me was that a large enterprise came to the same conclusions I’m preaching for the last few years.

TCP Is a Stream Protocol

I hope you know TCP provides a reliable stream service not reliable packet delivery, but you might not have realized all the implications – I found an old post by Robert Graham explaining how things really work and how you can use them to bypass quick-and-dirty IDS that rely on signatures instead of doing proper protocol decodes.

Schprokits with Jeremy Schulman on Software Gone Wild

Jeremy Schulman was the driving force behind the Puppet agent that Juniper implemented on some Junos switches (one of the first fully supported Puppet-on-a-switch implementations). In the meantime, he quit Juniper and started his own company focused on a network automation product – more than enough reasons to chat with him on Software Gone Wild.

Quick Guide to my Interop New York Sessions

I’m running or participating in five workshops or sessions during next week’s Interop New York. Three of them build on each other, so you might want to attend all of them in sequence:

Designing Infrastructure for Private Clouds starts with requirements gathering phase and focuses on physical infrastructure design decisions covering compute, storage, physical and virtual networking, and network services. If you plan to build a private (or a reasonable small public) cloud, start here.

Network Programmability 101: The Problem

In the first part of the Network Programmability webinar Matt Oswalt described some of the major challenges most networks are facing today:

  • Why is everyone claiming that the network is so slow to change?
  • Is that really the case? Why?
  • Why is the manual configuration culture so widespread in networking?
  • How does the holistic thinking in the design phase dissolve into the box mentality of CLI commands?
  • How does the box mentality limit the scalability of network deployments?

Connecting Virtual Routers to the Outside World

Stefan de Kooter (@sdktr) sent me a follow-up question to my Going All Virtual with Virtual WAN Edge Routers blog post:

How would one interface with external Internet in this scenario? I totally get the virtual network assets mantra, but even a virtual BGP router would need to get a physical interconnect one way or another.

As always, there are plenty of solutions depending on your security needs.

SDN Deployment Considerations

Are you lucky enough to be one of the 87% of North American enterprises that plan to have SDN in production by 2016 or one of the 53% of the companies that plan to have SDN deployed in the near future? Even though we all know how inflated these claims are, you might have to start considering the deployment aspects of a solution a $vendor will persuade your CIO to buy.

Formal Announcement: Software Gone Wild Podcast

If you’ve been reading my blog in the last few months, you might have noticed that I started a new podcast focused on software-defined everything (hence the name: Software Gone Wild – thanks to Jason Edelman).

The latest episodes are always available on this page; you can also subscribe to the podcast feed in RSS, Atom or iTunes format… and if you wonder why we need yet-another podcast, read the About Software Gone Wild document.

Virtual Networking in CloudStack

If you mention open-source cloud orchestration tools these days, everyone immediately thinks about OpenStack (including the people who spent months or years trying to make it ready for production use). In the meantime, there are at least two other comparable open-source products (CloudStack and Eucalyptus) that nobody talks about. Obviously having a working product is not as sexy as having 50+ vendors and analysts producing press releases.

Dynamic FCoE – Sparse-Mode FCoE Strikes Again

A while ago Cisco added dynamic FCoE support to Nexus 5000 switches. It sounded interesting and I wanted to talk about it in my Data Center Fabrics update session, but I couldn’t find any documentation at that time.

In the meantime, the Configuring Dynamic FCoE Using FabricPath configuration guide appeared on Cisco’s web site and J Metz wrote a lengthly blog post explaining how it all works, triggering a severe attack of déjà vu.

You’ve been doing the same thing for the last 20 years

When we were discussing my autumn travel plans, my lovely wife asked me “What are you going to talk about in Bern?” She has a technical background, but I didn’t feel like going into the intricacies of SDN, SDDC and NetOps, so I told her the essence of my keynote speech:

The Four Paths to SDN

After the initial onslaught of SDN washing, four distinct approaches to SDN have started to emerge, from centralized control plane architectures to smart reuse of existing protocols.

As always, each approach has its benefits and drawbacks, and there’s no universally best solution. You just got four more (somewhat immature) tools in your toolbox. And now for the details.

SIGS & Carrier’s Lunch DC Day: An Event Definitely worth Visiting

I spent last Tuesday in Bern attending the SIGS DC Day Event, and came back home extremely pleasantly surprised. The conference was nice and cozy, giving everyone plenty of opportunities to chat about data center technical challenges (thanks for all the wonderful conversations we had – you know who you are!).

Having the opportunity to meet fellow networking engineers and compare notes is great, but it’s even better to combine that with new knowledge, and that’s where the event really excelled.

Tech Talks: The Essence of MPLS

Seamus Gilchrist sent me a fantastic list of MPLS- and MPLS-TE-related questions. Instead of starting an email exchange we agreed on something that should benefit a wider community: a lengthy whiteboard session discussing the basics of MPLS, MPLS-TE, load balancing and QoS in MPLS networks…

The first part of our conversation is already online: The Essence of MPLS.

Open-Source Hybrid Cloud Reference Architecture on Software Gone Wild

A while ago Rick Parker told me about his amazing project: he started a meetup group that will build a reference private/hybrid cloud heavily relying on virtualized network services, and publish all documentation related to their effort, from high-level architecture to device and software configurations, and wiring plans.

In Episode 8 of Software Gone Wild Rick told us more about his project, and we simply couldn’t avoid a long list of topics including:

IPv6 Neighbor Discovery (ND) and Multicast Listener Discovery (MLD) Challenges

A few days ago Garrett Wollman published his exasperating experience running IPv6 on large L2 subnets with Juniper Ex4200 switches, concluding that “… much in IPv6 design and implementation has been botched by protocol designers and vendors …” (some of us would forcefully agree) making IPv6 “…simply unsafe to run on a production network…

The resulting debate on Hacker News is quite interesting (and Andrew Yourtchenko is trying hard to keep it close to facts) and definitely worth reading… but is ND/MLD really as broken as some people claim it is?

vMotion Enhancements in vSphere 6

VMware announced several vMotion enhancements in vSphere 6, ranging from “finally” to “interesting”.

vMotion across virtual switches. Finally. The tricks you had to use previous were absolutely bizarre.

Controller Cluster Is a Single Failure Domain

Some OpenFlow-focused startups are desperately trying to tell you how redundant their architecture is. Unfortunately all the whitepapers (and the prancing unicorns) cannot change a simple fact: an SDN controller (OpenFlow-based or otherwise) is in some aspects a single failure domain.

Is Anyone Using DMVPN-over-IPv6?

One of my readers sent me an interesting challenge: they’re deploying a new DMVPN WAN, and as they cannot expect all locations to have native (non-NAT) IPv4 access, they plan to build the new DMVPN over IPv6. He was wondering whether it would work.

Apart from “you’re definitely going in the right direction” all I could tell him was “looking at the documentation I couldn’t see why it wouldn’t work” Has anyone deployed DMVPN over IPv6 in a production network? Any hiccups? Please share your experience in the comments. Thank you!

See You in Bern on September 9th

TL;DR: I'll be in Bern on September 9th. If you'd like to drop by and discuss network design or automation challenges, read on…

Scalability Enhancements in Cisco Nexus 1000V

The latest release of Cisco Nexus 1000V for vSphere can handle twice as many vSphere hosts as the previous one (250 instead of 128). Cisco probably did a lot of code polishing to improve Nexus 1000V scalability, but I’m positive most of the improvement comes from interesting architectural changes.

Snabb Switch Deep Dive on Software Gone Wild

The pilot episode of Software Gone Wild podcast featuring Snabb Switch created plenty of additional queries (and thousands of downloads) – it was obviously time for another deep dive episode discussing the intricate innards of this interesting virtual switch.

During the deep dive Luke Gorrie, the mastermind behind the Snabb Switch, answered a long list of questions, including:

Just Published: SDN and OpenFlow – The Hype and the Harsh Reality

If you’re a regular reader of my blog, you know that I spent a lot of time during the last three years debunking SDN myths, explaining the limitations of OpenFlow and pointing out other technologies one could use to program the network.

During the summer of 2014 I organized my SDN- and OpenFlow-related blog posts into a digital book. I want to make this information as useful and as widely distributed as possible – for a limited time you can download the PDF free of charge.

Learn more about the book

Network Infrastructure as Database

A while ago I wrote about the idea of treating network infrastructure (and all other infrastructure) as code, and using the same processes application developers are using to write, test and deploy code to design and implement networks.

That approach clearly works well if you can virtualize (and clone ad infinitum) everything. We can virtualize appliances or even routers, but installed equipment and high-speed physical infrastructure remain somewhat resistant to that idea. We need a different paradigm, and the best analogy I could come up with is a database.

Is Data Center Trilogy Package the Right Fit to Understand Long Distance vMotion Challenges?

A reader sent me this question:

My company will have 10GE dark fiber across our DCs with possibly OTV as the DCI. The VM team has also expressed interest in DC-to-DC vMotion (<4ms). Based on your blogs it looks like overall you don't recommend long-distance vMotion across DCI. Will the "Data Center trilogy" package be the right fit to help me better understand why?

Unfortunately, long-distance vMotion seems to be a persistent craze that peaks with a predicable period of approximately 12 months, and while it seems nothing can inoculate your peers against it, having technical arguments might help.

Fate Sharing in IP Networks

My good friend Tiziano complained about the fact that BGP considers next hop reachable if there’s an entry in the IP routing table even though the router cannot even ping the next hop.

That behavior is one of the fundamental aspects of IP networks: networks built with IP routing protocols rely on fate sharing between control and data planes instead of path liveliness checks.

Toolsmith @ Netflix on Software Gone Wild

I first met Elisa Jasinska when she had one of the coolest job titles I ever saw: Senior Packet Herder. Her current job title is almost as cool: Senior Network Toolsmith @ Netflix – obviously an ideal guest for the Software Gone Wild podcast.

In our short chat she described some of the tools she’s working on, including an adaptation of pmacct to environments with numerous BGP exit points (more details in her NANOG presentation).

VMware EVO:RAIL – One Stop Shopping for Your Private Cloud

Building a private cloud infrastructure tends to be a cumbersome process: even if you do it right, you oft have to deal with four to six different components: orchestration system, hypervisors, servers, storage arrays, networking infrastructure, and network services appliances.

Interview: Reduce Costs and Gain Efficiencies with SDDC

A few days ago I had an interesting interview with Christoph Jaggi discussing the challenges, changes in mindsets and processes, and other “minor details” one must undertake to gain something from the SDDC concepts. The German version of the interview is published on; you’ll find the English version below.

Finally: a Virtual Switch Supports BPDU Guard

Nexus 1000V release 5.2(1)SV3(1.1) was published on August 22nd (I’m positive that has nothing to do with VMworld starting tomorrow) and I found this gem in the release notes:

Enabling BPDU guard causes the Cisco Nexus 1000V to detect these spurious BPDUs and shut down the virtual machine adapters (the origination BPDUs), thereby avoiding loops.

It took them almost three years, but we finally have BPDU guard on a layer-2 virtual switch (why does it matter). Nice! Is on CloudFlare (and IPv6)

After a week of testing, I decided to move the main web site ( as well as some of the resource servicing hostnames to CloudFlare CDN. Everything should work fine, but if you experience any problems with my web site, please let me know ASAP.

2014-08-27: Had to turn off CloudFlare (and thus IPv6). They don't seem to support HTTP range requests, which makes video startup time unacceptable. Will have to move all video URLs (where the HTTP range requests are expected coming from streaming clients) to a different host name, which will take time.

Collateral benefit: is now fully accessible over IPv6 – register for the Enterprise IPv6 101 webinar if you think that doesn’t matter ;)

Cloud Orchestration System Is an Ideal Controller Use Case

A while ago I explained why OpenFlow might be a wrong tool for some jobs, and why centralized control plane might not make sense, and quickly got misquoted as saying “controllers don’t scale”. Nothing could be further from the truth, properly architected controller-based architectures can reach enormous scale – Amazon VPC is the best possible example.

The Impact of Data Gravity: a Campfire Story

Here’s an interesting story illustrating the potential pitfalls of multi-DC deployments and the impact of data gravity on application performance.

Long long time ago on a cloudy planet far far away, a multinational organization decided to centralize their IT operations and move all workloads into a central private cloud.

Pmacct: the Traffic Analysis Tool with Unpronounceable Name

SDN evangelists talking about centralized traffic engineering, flow steering or bandwidth calendaring sometimes tend to gloss over the first rule of successful traffic engineering: Know Thy Traffic.

In a world ruled by OpenFlow you’d expect the OpenFlow controller to know all the traffic; in more traditional networks we use technologies like NetFlow, sFlow or IPFIX to report the traffic statistics – but regardless of the underlying mechanism, you need a tool that will collect the statistics, aggregate them in a way that makes them usable to the network operators, report them, and potentially act on the deviations.

Revisited: Layer-2 DCI over VXLAN

I’m still getting questions about layer-2 data center interconnect; it seems this particular bad idea isn’t going away any time soon. In the face of that sad reality, let’s revisit what I wrote about layer-2 DCI over VXLAN.

VXLAN hasn’t changed much since the time I explained why it’s not the right technology for long-distance VLANs.

Do you really need to see all 512K Internet routes?

Last week the global routing table (as seen from some perspectives) supposedly exceeded 512K routes, and weird things started to happen to some people that are using old platforms that by default support 512K IPv4 routes in the switching hardware.

I’m still wondering whether the BGP table size was the root cause of the observed outages. Cisco’s documentation (at least this document) is pretty sloppy when it comes to the fact that usually 1K = 1024, not 1000 – I’d expect the hard limit to be @ 524.288 routes … but then maybe Cisco’s hardware works with decimal arithmetic.

Just Published: Brocade VCS Fabric Videos

The Data Center Fabric Architectures update session in late June included a whole new section on Brocade’s VCS fabric and new features they added in Network OS 4.0. The edited videos have been published and cover these topics:

What Is a Valid BGP Route?

Carlos Mendioroz sent me a seemingly simple question: when is a BGP route invalid? My knee-jerk reaction: when the next hop is not reachable (and I’m not the only one). WRONG – BGP routes with unreachable next hop are still valid, as shown in the following printout:

MPLS Load Sharing – Data Plane Considerations

In a previous blog post I explained how load sharing across LDP-controlled MPLS core works. Now let’s focus on another detail: how are the packets assigned to individual paths across the core?

2014-08-14: Additional information was added to the blog post based on comments from Nischal Sheth, Frederic Cuiller and Tiziano Tofoni. Thank you!

Should I Go for CCIE or Some Other Certification?

One of my readers sent me this question:

I am already CCIE and work as a network engineer with pretty good salary. But I think that I am losing some passion for Cisco networking and have interests in many other technologies. Currently I am very interested in Linux and Python development. Is it worth to add some Red Hat certification along CCIE or should I pursue another CCIE?

I think “should I go for CCIE or RHCE” is the wrong question.

VXLAN Encapsulation in Juniper Contrail

VXLAN is becoming de-facto encapsulation standard for overlay virtual networks (at least according to industry pundits and marketing gurus working for companies with VXLAN-based products) – even Juniper Contrail, which was traditionally a pure MPLS/VPN architecture uses it.

Not so fast – Contrail is using VXLAN packet format to carry MPLS labels between hypervisors and ToR switches.

Load Sharing in MPLS Core

Here’s a question that bothered me for years till I finally gave up and labbed it: does ECMP load sharing work in an MPLS core? More specifically, will an LSP split into multiple LSPs?

STP and Expert Beginners

Maxim and myself continued our STP discussion and eventually agreed that while STP might not be the best protocol out there (remember: it had to run on Z80 CPU), it’s the only standardized thing that prevents nasty forwarding loops, prompting Maxim to ask another seemingly simple question:

What's so wrong with STP, that there are STP haters out there turning it off wherever they see it?

Welcome to the wonderful world of Expert Beginners.

Rate-Limit Console Logging

Someone made a really interesting remark on my Disable Console Logging blog post: Cisco IOS has log message rate limiting – all you need is the logging rate-limit configuration command.

VMware vSwitch and 802.1p CoS Value

One of my readers opened another can of VMware vSwitch worms. He sent me this question:

If a VM were to set a COS value, would the vSwitch reset it to 0 as part of its process of building the dot1q header?

The nasty detail (as you probably know) is that 802.1p CoS value resides in the 802.1q (VLAN) tag.

Is STP Really Evil?

Maxim Gelin sent me an interesting question:

Can you please explain to me, why is STP supposed to be evil? What's wrong with STP?

STP’s fundamental problem is that it’s a fail-close, not a fail-open protocol.

Could You Replace MPLS/VPN with IPSec-over-Internet?

Someone recently sent me this scenario:

Our CIO has recently told us that he wants to get rid of MPLS because it is too costly and is leaning towards big Internet lines running IPSEC VPNs to connect the whole of Africa.

He was obviously shopping around for free advice (my friend Jeremy Stretch posted his answers to exactly the same set of questions not so long ago); here are the responses I wrote to his questions:

Campfire story: Using the wrong tool for the job

Summer is the perfect time for campfire stories – here’s one about using the wrong tool for the job.

A Long time ago in an IT organization far, far away Artificial Intelligence (AI) was the coolest kid on the block.

Layer-3 Switching over VXLAN Revisited

My Trident 2 Chipset and Nexus 9500 blog post must have hit a raw nerve or two – Bruce Davie dedicated a whole paragraph in his Physical Networks in Virtualized Networking World blog post to tell everyone how the whole thing is a non-issue and how everything’s good in the NSX land.

It’s always fun digging into more details to figure out what’s really going on behind the scenes; let’s do it.

Next Chapter in Data Center Design Case Studies

When I published the Data Center Design Case Studies book almost exactly a month ago, three chapters were still missing – but that was the only way to stop the procrastination and ensure I’ll write them (I’m trying to stick to published deadlines ;).

The first one of the missing chapters is already finished and available to subscribers and everyone who bought the book or Designing Private Cloud Infrastructure webinar (you’ll also get a mailing on Sunday to remind you to download the fresh copy of the PDF).

The Amazon Kindle version will be updated in a few days.

Network Automation @ Spotify on Software Gone Wild

What can you do if you have a small team of networking engineers responsible for four ever-growing data centers (with several hundred network devices in each of them)? There’s only one answer: you try to survive by automating as much as you can.

In the fourth episode of Software Gone Wild podcast David Barosso from Spotify explains how they use network automation to cope with the ever-growing installed base without increasing the size of the networking team.

There Is no Paradigm Shift – Good Applications Were Always Network-Aware

Someone left the following comment on one of my blog posts:

There is a paradigm shift that I don’t think most application developers understand. In a traditional enterprise model, the network is built around the application requirements, now we are saying the application has to build around the network.

I would say there’s no paradigm shift – developers of well-performing applications were always aware of laws of physics.

How Big Will Your Cloud Be?

The last generations of high-end servers are amazing: they can have terabyte (or more) of RAM, dozens of CPU cores, and four (or more) 10GE uplinks. It’s easy to pack 100+ well-behaved VMs on a single server, reducing the whole data center into a private cloud that fits into a single rack.

The F-Script with John Herbert on Software Gone Wild

The use of tools has accelerated human evolution and made us what we are today. Networking is no different, and yet there aren’t that many tool builders among the networking engineers… or maybe all you need is a nudge and some hints on how to get started.

The Virtual Design Master Is Starting Today

A while ago I was listening to a podcast describing Project Runway for geeks – the Virtual Design Master – and decided to do what I could to help them ;)

The second season is starting today – the list of participants is already online (and you might watch the videos of the first season while waiting for the first challenge).

Data Center Design Case Studies on Amazon Kindle

If you’re an avid Kindle user, you can buy the Data Center Design Case Studies book on Amazon. Here’s the link, search should find it on most other Amazon marketplaces.

However, if you can survive reading the PDF version, please buy it straight off my web site. Here’s why:

The summer office is finally operational...

Expect a significantly reduced blogging frequency ;)

Build a Cloud in Three Easy Steps

Occasionally I get a question about some totally impossible implementation detail (example: can we use OpenStack OVS plugin on VMware to avoid buying NSX?). These questions are often coming from people who painted themselves into a corner and are now desperately looking for MacGyver’s shoelaces to pull themselves out.

It’s easy to blame the engineer who tries to do the obviously impossible, but it’s often not his fault – these days a lot of technical people get pulled into the game of Build a Cloud in Three Easy Steps.

Network Programmability with David Gee on Software Gone Wild

For the second episode of Software Gone Wild I got a truly interesting guest: David Gee, a network engineer already working on numerous network programmability and orchestration deployment.

During our half-hour chat we couldn’t avoid the question of whether every networking engineer will become a programmer and David provided an interesting answer: you don’t have to program, but you’ll definitely have to start thinking more like a good programmer.

What Is This API Thingy?

A reader sent me this question:

I am hearing a lot about API in reference to SDN. I do not have any software or programming background but would like to understand this API in practical way. Could you help me?

TL&DR: API is CLI for program-to-program communication

Mice, Elephants and Virtual Switches

The Mice and Elephants is a traditional QoS fable – latency-sensitive real time traffic (or request-response protocol like HTTP) stuck in the same queue behind megabytes of file transfer (or backup or iSCSI) traffic.

The solution is also well known – color the elephants pink (aka DSCP marking) and sort them into a different queue – until the reality intervenes.

A Long Trip down the Memory Lane

True old-timers might appreciate the analogies I got while writing the Network Infrastructure as Code article. Let’s start with “do you remember this thingy?”

Source: Wikipedia

If you recognized the state-of-the-art (in those days) box in the picture, you might be able to relate to this screenshot:

It works, but is it scalable?

This is pretty close to some SDN architectures I was privileged to see in the last three years.

Virtual Routers 101

I was asked to do a presentation at the recent Slovenian NOG (SINOG) meeting. I did an SDN one at the previous meeting, making NFV the next obvious choice… but I decided to put an interesting spin on it and focused on virtual routers.

Infrastructure as Code Actually Makes Sense

When I heard people talking about “networking infrastructure as code” I dismissed that as yet another Software-Defined-Everything one-controller-to-rule-it-all hype. Boy was I wrong.

Snabb Switch and NFV on OpenStack in Software Gone Wild

Last September I received a peculiar tweet from Luke Gorrie pointing me to a software switch pushing 200 Gbps through an Intel server literally hours after I’d watched the Deutsche Telekom Terastream presentation, so I mentioned Luke’s Snabb Switch as a potential performance solution in an email to Ian Farrer… and before Ian managed to reply, Luke was already working for Deutsche Telekom.

Unnumbered OSPF Interfaces in Quagga (and Cumulus)

Carlos Mendioroz sent me an interesting question about unnumbered interfaces in Cumulus Linux and some of the claims they make in their documentation.

TL&DR: Finally someone got it! Kudos for realizing how to use an ancient trick to make data center fabrics easier to deploy (and, BTW, the claims are exaggerated).

Why is IPv6 layer-2 security so complex (and how to fix it)

After the excellent IPv6 security presentation Eric Vyncke had @ 9th Slovenian IPv6 summit someone asked me: “Why is IPv6 first-hop security so complex? It looks like the developers of IPv6 protocol stack tried to make users anonymous and made everyone’s life complex while doing that.

Well, he was totally surprised by my answer: “The real reason IPv6 first-hop security is so complex is the total mess we made of L2/L3 boundary.”

Bikeshed painting

Tired of endless debates discussing trivial matters? You're not alone (bonus point: a few pop-up windows every mail client should have).

Virtual Switch 101

One of my readers sent me this question:

I am a CCNP-certified L3/L4 network engineer working purely on Cisco products and network technologies for the last 10 years of experience. How long will it take for a typical person like me to get a foothold into VM/SAN?

It obviously depends on how deep you want to go, but the virtual networking part of it shouldn’t be too hard.

To Get a Job Done Well, You Need Proper Training

The “bring Amazon Web Services mentality back home” blog post generated the expected comments, from “developers have no clue about networking or network services” to “we went through the whole thing and failed badly.”

Well, even though it might have seemed so, I didn’t advocate letting the developers go unchecked, I was just pointing out that double standards make no sense.

Just Out: Data Center Design Case Studies

After months of being on a back burner, my first self-publishing book is out: Data Center Design Case Studies are available on my web site (Amazon Kindle version coming whenever I find time to figure out all the details).

Subscribers can download the book immediately (you’ll find it in the new Books section), as can anyone who bought the Designing Private Cloud Infrastructure webinar (you’ll find it in the webinar materials), and you can buy a copy on my web site.

More information

Updated: Metro Ethernet and Carrier Ethernet Encryption

Update 2014-06-18 05:13Z - We slashdotted Christoph's site yesterday. He moved to a new server during the night; the links should work now.

Christoph Jaggi focused on analyzing Metro Ethernet and Carrier Ethernet encryption gear. The introductory part of this year’s report has just been published and it’s definitely worth reading even if you have no immediate plans to buy such gear – it’s a nice overview document covering numerous encryption technologies, key distribution systems, network topologies, and operational aspects. If you want to get in-depth evaluation of individual vendors or solutions, you’ll obviously have to contact Christoph.

Speaking of security, if you wonder how to secure your credit card handling environment, register for Michele Chubirka's PCI DSS webinar.

Trident 2 Chipset and Nexus 9500

Most recently launched data center switches use the Trident 2 chipset, and yet we know almost nothing about its capabilities and limitations. It might not work at linerate, it might have L3 lookup challenges when faced with L2 tunnels, there might be other unpleasant surprises… but we don’t know what they are, because you cannot get Broadcom’s documentation unless you work for a vendor who signed an NDA.

Interestingly, the best source of Trident 2 technical information I found so far happens to be the Cisco Live Nexus 9000 Series Switch Architecture presentation (BRKARC-2222). Here are a few tidbits I got from that presentation and Broadcom’s so-called datasheet.

Just Found Two Data Center / Virtualization Blogs

I found two interesting data center/virtualization blogs this week (can’t remember the last time I found two new blogs in a week):

  • Pedro Marques’ blog – it focuses primarily on Open Contrail, but you’ll find interesting gems questioning the wisdom of OpenFlow and Segment Routing (among other things);
  • Aldrin Isaac’s blog – unfortunately he stopped writing in the last months, but some of his older posts are pure gold.

Enjoy – and poke Aldrin to continue writing ;)

Programming the Way I Remember It

Jure Špiler, an old friend of mine (and my first boss) sent me a link to a fantastic YouTube video documenting how we did programming on PDP-11 computers when I started my programming career (note: I did not enter compiled assembly code with console switches on PDP-11, but I did do that on an 6800-based box with 2K of RAM).

Related videos tab revealed another gem: PDP-11/34 with RL-01 disk drives – the very same system where I started my IT career as a high-school kid. Enjoy!

IP Addresses Considered Harmful

A long while ago I wrote about the brokenness of socket API and lack of session layer in TCP/IP stack. Looks like I’m not the only one with heretic opinions; Fred Baker reached similar conclusions in his Happier Eyeballs draft and Brian Carpenter recently published a long article title IP Addresses Considered Harmful which documents (among other things) the history of socket API and the reasons DNS isn’t tightly integrated with it. Both documents are definitely worth reading.

On the other hand, if you love listening to old rants, here’s a video I recorded more than 3 years ago (part of the Upcoming Internet Challenges webinar). The truly sad part: nothing changed in the meantime.

Is OpenFlow the Best Tool for Overlay Virtual Networks?

Overlay virtual networks were the first commercial-grade OpenFlow use case – Nicira’s Network Virtualization Platform (NVP – now VMware NSX for Multiple Hypervisors) used OpenFlow to program the hypervisor virtual switches (Open vSwitches – OVS).

OpenStack is using the same approach in its OVS Neutron plugin, and it seems Open Daylight aims to reinvent that same wheel, replacing OVS plugin running on the hypervisor host agent with central controller.

Does that mean one should always use OpenFlow to implement overlay virtual networks? Not really, OpenFlow is not exactly the best tool for the job.

Will Network Engineers Become Programmers?

I know numerous engineers who decided to pursue a career in networking because they didn’t want to deep-dive into programming. Will that change when the Software Defined Everything takes over the world?

TL&DR summary: Of course not.

Can We Just Throw More Bandwidth at a Problem?

One of my readers sent me an interesting question:

I have been reading at many places about "throwing more bandwidth at the problem." How far is this statement valid? Should the applications(servers) work with the assumption that there is infinite bandwidth provided at the fabric level?

Moore’s law works in our favor. It’s already cheaper (in some environments) to add bandwidth than to deploy QoS.

30 Years of Yammering

Some of the comments I get every time I write about the idea of merging network services deployment with application deployments, and making application developers responsible for the results of their code (aka DevOps) remind me of a very long list of “this will never work” sentiments I encountered in the 30 years I spent in IT and networking. Here are just a few of them:

Use Webinars for Your Summer Bootcamp

A year ago I wrote about a university that used my webinars in a summer bootcamp. In the meantime, I formalized the program and added a simple form that allows you to:

  • Calculate how much academic workgroup subscriptions or webinar technology tracks cost;
  • Request a quote once you find the product that meets your budget.

If you have any questions about this program, just send me an email.

Virtual Networking Implementation Taxonomy

I’m not sure I wrote about the taxonomy of numerous virtual networking implementations. Just in case, here it is ;)

Layer-2 or layer-3 networks?

Some virtual networking solutions emulate thick coax cable (more precisely, layer-2 switch), giving their users the impression of having regular VLAN-like layer-2 segments.

New Webinar: PCI DSS for Engineers

Every now and then I get a complaint that someone cannot buy the webinar subscription because I’m using Google Wallet to process the credit card transactions. There’s a single reason I’m using Google Wallet: I’m trying to stay as far away as possible from the morass called PCI DSS.

FCoE and Nexus 1000v QoS

One of my readers wanted to deploy FCoE on UCS in combination with Nexus 1000v and wondered how the FCoE traffic impacts QoS on Nexus 1000v. He wrote:

Let's say I want 4Gb for FCoE. Should I add bandwidth shares up to 60% in the nexus 1000v CBWFQ config so that 40% are in the default-class as 1kv is not aware of FCoE traffic? Or add up to 100% with the assumption that the 1kv knows there is only 6Gb left for network? Also, will the Nexus 1000v be able to detect contention on the uplink even if it doesn't see the FCoE traffic?

As always, things aren’t as simple as they look.

Building Scalable Web Applications – Final Presentations

Last Friday my students attending this year’s Designing Scalable Web Applications course presented their semester-long assignments. I can’t tell you how pleasantly surprised I was – the results were much better and more polished than what I’ve seen during the previous years.

What Are Linux Containers?

Everyone (and The Register) talks about Linux containers these days like they would be the hottest thing invented this spring. In reality, it’s a seven year old technology that was heavily used by some smart web hosting companies for years (but of course some people think mentioning Google makes everything look sexier).

If you’re interested in a high-level overview of differences between Linux containers and more traditional virtual machines, watch the video from the Introduction to Virtual Networking webinar.

It’s OK to Let Developers Go @ Amazon Web Services, but Not at Home? You Must Be Kidding!

Recently I was discussing the benefits and drawbacks of virtual appliances, software-defined data centers, and self-service approach to application deployment with a group of extremely smart networking engineers.

After the usual set of objections, someone said “but if we won’t become more flexible, the developers will simply go to Amazon. In fact, they already use Amazon Web Services.

How Line-rate Is Line-rate?

During yesterday’s Data Center Fabrics Update presentation, one of the attendees sent me this question while I was talking about the Arista 7300 series switches:

Is the 7300 really non-blocking at all packet sizes? With only 2 x Trident-2 per line card it can't support non-blocking for small packets based on Trident-2 architecture.

It was an obvious example of vendor bickering, so I ignored the question during the presentation, but it still intrigued me, so I decided to do some more research.

Queuing Mechanisms in Modern Switches

A long while ago there was an interesting discussion started by Brad Hedlund (then at Dell Force10) comparing leaf-and-spine (Clos) fabrics built from fixed-configuration pizza box switches with high-end chassis switches. The comments made by other readers were all over the place (addressing pricing, wiring, power consumption) but surprisingly nobody addressed the queuing issues.

This blog post focuses on queuing mechanisms available within a switch; the next one will address end-to-end queuing issues in leaf-and-spine fabrics.

The best spam comment (so far)

Idiots posting random comments with (not-so-very) hidden links to whatever warez they're selling are utterly annoying, but there's always one-in-a-million chance for a hilarious one. This is what I got on the Traffic Trombone post:

The traffic across the network core and the end-to-end latency would be minimal (the same packet would traverse the core only once), increasing visits to my adult site.

Data Center Protocols in HP Switches

HP representatives made some pretty bold claims during Networking Tech Field Day 1, including “our switches will support EVB, FCoE, SPB and TRILL.” I took them three years to deliver on those promises (and the hardware they had at that time doesn’t exactly support all features they promised), but their current protocol coverage is impressive.

Benefits of SDN (or: SDN is like IPv6)

A while ago Paul Stewart wrote a fantastic blog post listing the potential business benefits of SDN (as promoted by SDN evangelists and SDN-washing vendors).

Here’s his list:

How Do I Start My First Overlay Virtual Networking Project?

After the Designing Private Cloud Infrastructure workshop I had in Slovenia last week (in a packed room of ~60 people), someone approached me with a simple question: “I like the idea of using overlay virtual networks in my private cloud, but where do I start?

vMotion and VXLAN

A while ago I wrote “vMotion over VXLAN is stupid and unnecessary” in a comment to a blog post by Duncan Epping, assuming everyone knew the necessary background details. I was wrong (again).

All Operations Engineers Should Have Firefighting Training

Recently I had a fantastic conversation with Erich Hohermuth, a networking engineer with an unusual hobby: he’s a professional firefighting instructor (teaching firefighters across the country how to do their job).

Volunteer fire departments are pretty popular in Central European countries, and so he’s not the only one on his team with that skillset. The (not so unexpected) side effect: these people are the best ones when it comes to fighting IT disasters.

OpenFlow Support in Data Center Switches

Good news: In the last few months, almost all major data center Ethernet switching vendors (Arista, Cisco, Dell Force 10, HP, and Juniper) released documented GA version of OpenFlow on some of their data center switches.

Bad news: no two vendors have even remotely comparable functionality.

Load Balancing Across IP Subnets

One of my readers sent me this question:

I have a data center with huge L2 domains. I would like to move routing down to the top of the rack, however I’m stuck with a load-balancing question: how do load-balancers work if you have routed network and pool members that are multiple hops away? How is that possible to use with Direct Return?

There are multiple ways to make load balancers work across multiple subnets:

Whitebox Switching and Fermi Estimates

Craig Matsumoto recently quoted some astonishing claims from Dell’Oro Group analyst Alan Weckel:

  • Whitebox switches (combined) will be the second largest ToR vendor;
  • Whitebox 10GE ports will cost around $100.

Let’s try to guestimate how realistic these claims are.

Connecting Legacy Servers to Overlay Virtual Networks

I wrote (and spoke) at length about layer-2 and layer-3 gateways between VLANs and overlay virtual networks, but I still get questions along the lines of “how will you connect legacy servers to the new cloud infrastructure that uses VXLAN?

Optimizing OpenFlow Hardware Tables

Initial OpenFlow hardware implementations used a simplistic approach: install all OpenFlow entries in TCAM (the hardware that’s used to implement ACLs and PBR) and hope for the best.

That approach was good enough to get you a tick-in-the-box on RFP responses, but it fails miserably when you try to get OpenFlow working in a reasonably sized network. On the other hand, many problems people try to solve with OpenFlow, like data center fabrics, involve simple destination-only L2 or L3 switching.

Is OpenFlow Useful?

The Does Centralized Control Plane Make Sense post triggered several comments along the lines of “do you think there’s no need for OpenFlow?

TL;DR version: OpenFlow is just a low-level tool. Don’t blame it for how it’s being promoted.

New content feeds (and why I love Perl)

I’m slowly moving away from Feedburner, and started the process by creating a new web page listing all my content feeds.

Sounds great, right? Well, this isn’t how this particular yak shaving really started.

Does Centralized Control Plane Make Sense?

A friend of mine sent me a challenging question:

You've stated a couple of times that you don't favor the OpenFlow version of SDN due to a variety of problems like scaling and latency. What model/mechanism do you like? Hybrid? Something else?

Before answering the question, let’s step back and ask another one: “Does centralized control plane, as evangelized by ONF, make sense?

It Doesn’t Make Sense to Virtualize 80% of the Servers

A networking engineer was trying to persuade me of importance of hardware VXLAN VTEPs. We quickly agreed physical-to-virtual gateways are the primary use case, and he tried to illustrate his point by saying “Imagine you have 1000 servers in your data center and you manage to virtualize 80% of them. How will you connect them to the other 200?” to which I replied, “That doesn’t make any sense.” Here’s why.

Marketing Grammar

We all know how IT marketing works – unless you exaggerate your claims at least as much as your competitors do (the activity politely called “Bulls**t bidding war” by Tom Nolle) you’re soon just a footnote in the IT history. However, you don’t have to use the same approach in technical conversations.

SDN, OpenFlow, NFV and SDDC: Hype and Reality (2-day Workshop)

There are tons of SDN workshops, academies, and webinars out there, many of them praising the almost-magic properties of the new technologies, or the shininess of vendors’ new gadgets and strategic alliances. Not surprisingly, the dirty details of real-life deployments aren’t their main focus.

As you might expect, my 2-day workshop isn’t one of them.

Declarative and Procedural Programming (and How I Got It all Wrong)

During a recent NetOps-focused discussion trying to figure out where Puppet/Chef/Ansible/… make sense in the brave new SDN-focused networking world I made this analogy: “Puppet manifest is like Prolog, router configuration is like Java or C++.” It’s a nice sound bite. It’s also totally wrong.

If you never met Prolog, you might consider yourself lucky. Or you might want to figure out what it is (warning: it might make your head explode). Just joking, I actually quite liked it in my programming days.

IPv6 Resources on

As expected, ARIN wasn’t that far behind APNIC and RIPE in IPv4 allocations and is now down to the last /8. Maybe it’s time for the last denialists to wake up and start considering IPv6 (or not – consultants love panicking customers)… and the new IPv6 resources page on might help you get IPv6-fluent (hint: don’t miss the must-read documents section).