Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

9 module online course

Start now!

That’s It for 2014

A dozen webinars, tens of public presentations and on-site workshops, numerous highly interesting ExpertExpress sessions, three books and over 250 blog posts. That should be enough for a year; it’s time to go offline.

I hope your company has a New Year freeze (and not let’s upgrade everything over New Year policy), so you’ll be able to do the same and enjoy some time during the rest of the year with your loved ones. See you in 2015!

see 3 comments

VRF Lite on Nexus 5600

One of the networking engineers using my ExpertExpress to validate their network design had an interesting problem: he was building a multi-tenant VLAN-based private cloud architecture with each tenant having multiple subnets, and wanted to route within the tenant network as close to the VMs as possible (in the ToR switch).

He was using Nexus 5600 as the ToR switch, and although there’s conflicting information on the number of VRFs supported by that switch (verified topology: 25 VRFs, verified maximum: 1000 VRFs, configuration guide: 64 VRFs), he thought 25 VRFs (tenant routing domains) might be enough.

read more see 3 comments

L2VPN over IPv6 with Snabb Switch on Software Gone Wild

Highly customizable high-speed virtual switch written in Lua sounds great, but is it really that easy to use? Simon Leinen was kind enough to get me in touch with Alex Gall, his colleague at Switch, who's working on an interesting project: implementing L2VPN over IPv6 with Snabb Switch.

read more see 1 comments

Facebook Next-Generation Fabric

Facebook published their next-generation data center architecture a few weeks ago, resulting in the expected “revolutionary approach to data center fabrics” echoes from the industry press and blogosphere.

In reality, they did a great engineering job using an interesting twist on pretty traditional multi-stage leaf-and-spine (or folded Clos) architecture.

read more see 7 comments

Performance Tests and Out-of-Box Performance

Simonp made a perfectly valid point in a comment to my latest OVS blog post:

Obviously the page you're referring to is a quick-and-dirty benchmark. If you wanted the optimal numbers, you would have to tune quite a few parameters just like for hardware benchmarks (sysctl kernel parameters, Jumbo frames, ...).

While he’s absolutely right, this is not the performance data a typical user should be looking for.

read more Add comment

Load Balancing in Google Network

Todd Hoff (of the HighScalability fame) sent me a link to an interesting video describing load-balancing mechanisms used at Google and how they evolved over time.

If the rest of the blog post feels like Latin, you SHOULD watch the Load Balancing and Scale-Out Application Architecture webinar.

The beginning of the story resembles traditional enterprise solutions:

read more see 2 comments

Scaling Distributed Systems Is Hard

Stumbled upon a hilarious description of challenges encountered when trying to scale distributed systems (cluster of controllers running centralized control plane comes to mind).

It starts with “If someone tells you that scaling out a distributed system is easy they are either lying or drunk, and possibly both,” and gets better and better. Enjoy!

see 1 comments

Hotel California Effects of Public Clouds

In his The Case for Hybrids blog post Mat Mathews described the Hotel California effect of public clouds as: “One of the most oft mentioned issues with public cloud is the difficulty in getting out.” Once you start relying on cloud provider APIs to provide DNS, load balancing, CDN, content hosting, security groups, and a plethora of other services, it’s impossible to get out.

Interestingly, the side effects of public cloud deployments extend into the realm of application programming, as I was surprised to find out during one of my Expert Express engagements.

read more see 4 comments

Should I Really Program My Network?

In my presentation @ SDN Meetup in Stockholm, I tried to answer a simple question: “Should I really program my network?” and obviously had to start with an even simpler one: “What is SDN?

The video of the presentation is already available on YouTube, and you can watch the slides on my content web site.

Also, make sure you watch other presentations from that event, particularly David Barroso’s SDN Internet Router.

see 2 comments

MPLS P-Router, Router or Layer-3 Switch?

One of my readers is struggling with the aftermath of marketing gimmicks:

We will be implementing a new network soon, and we're discussing P-routers versus regular routers versus switches. I'm looking for arguments to go one way or the other.

TL&DR: there’s no difference between router and L3 switch.

read more see 5 comments

Quick Peek: Juniper vMX Router

While the industry press deliberates the disaggregation of Arista and Cisco, and Juniper’s new CEO, Juniper launched a virtual version of its vMX router, which is supposed to have up to 160 Gbps of throughput (as compared to 10 Gbps offered by Vyatta 5600 and Cisco CSR). Can Juniper really deliver on that promise?

read more see 9 comments

Moving Workloads to the Clouds

David Spark published 16 tips for moving your workloads to the clouds. Contrary to the usual useless nonsense coming down from hybrid cloud evangelists (you know, the people who moved from “VMs following the sun” to “seamless hybrid cloud workload mobility”) some of the tips actually make sense, starting with “Have a real reason for the migration”. Enjoy!

see 1 comments

Open vSwitch Performance Revisited

A while ago I wrote about performance bottlenecks of Open vSwitch. In the meantime, the OVS team drastically improved OVS performance resulting in something that Andy Hill called Ludicrous Speed at the latest OpenStack summit (slide deck, video).

Let’s look at how impressive the performance improvements are.

read more see 8 comments

Coping with Byzantine Routing Failures

One of my readers sent me an interesting challenge:

We have two MPLS providers sending us default routes and it seems like whenever we have problem with SP1 our failover is not happening properly and actually we have to go in manually and influence our traffic to forward via another path.

Welcome to the wondrous world of byzantine routing failures ;)

read more Add comment

Do We Have Too Many Knobs?

The last day of Interop New York found me sitting in the Speaker Center with a few friends pondering the hype and reality of SDN and brokenness of traditional network products. One of the remarks during that conversation was very familiar: “we have too many knobs to configure”, and I replied “and how many knobs do you think there are in Windows registry?" (or Linux kernel and configuration files).

read more see 6 comments

Just Published: Overlay Virtual Networks in Software Defined Data Centers

Overlay virtual networks are one of my favorite topics – it seems I wrote over a hundred blog posts describing various aspects of this emerging (or is it reinvented) technology since Cisco launched VXLAN in 2011.

During the summer of 2014 I organized my blog posts on overlay networks and SDDC into a digital book. I want to make this information as useful and as widely distributed as possible – for a limited time you can download the PDF free of charge.

Learn more about the book

see 1 comments

Reinventing the wheel (or RFC 1925 sect 2.11)

Simon Wardley is another old-timer with low tolerance for people reinventing the broken wheels. I couldn’t resist sharing part of his blog post because it applies equally well to what we’re seeing in the SDN world:

No, I haven't read Gartner's recent research on this subject (I'm not a subscriber) and it seems weird to be reading "research" about stuff you've done in practice a decade ago (sounds familiar). Maybe they've found some magic juice? Experience however dictates that it'll be snake oil […]. I feel like the old car mechanic listening to the kid saying that his magic pill turns water into gas. I'm sure it doesn't ... maybe this time it will ... duh, suckered again.

Meanwhile the academics already talk about SDN 2.0.

Add comment

Viptela SEN: Hybrid WAN Connectivity with an SDN Twist

Like many of us Khalid Raza wasted countless hours sitting in meetings discussing hybrid WAN connectivity designs using a random combination of DMVPN, IPsec, PfR, and one or more routing protocols… and decided to try to create a better solution to the problem.

Viptela Secure Extensible Network (SEN) doesn’t try to solve every networking problem ever encountered, which is why it’s simpler to use in the use case it is designed to solve: multi-provider WAN connectivity.

read more see 1 comments

Just Published: VXLAN 2.0 Videos

Last week I ran the second part of the updated (4-hour) VXLAN webinar. The raw videos are already online and cover these topics:

  • VXLAN-related technologies, including encapsulation, IP multicast use, unicast VXLAN, and VXLAN-over-EVPN;
  • VXLAN implementations, including Cisco Nexus 1000v, VMware vCNS, VMware NSX, Nuage VSP and Juniper Contrail;
  • VXLAN gateways, including Arista, Brocade, Cisco and Juniper;
  • Hardware VTEP integration with OVSDB and EVPN;
  • VXLAN-based data center fabrics, including Cisco’s ACI.
Add comment

Scaling the Cloud Security Groups

Most overlay virtual networking and cloud orchestration products support security groups more-or-less-statefulish ACLs inserted between VM NIC and virtual switch.

The lure of security groups is obvious: if you’re willing to change your network security paradigm, you can stop thinking in subnets and focus on specifying who can exchange what traffic (usually specified as TCP/UDP port#) with whom.

read more see 4 comments

A Month of SDN

My calendar for the following four weeks is jam-packed with SDN events:

All the travel might affect my blogging frequency, but I still have a few podcasts in the editing queue, so you’ll have something to listen to in the meantime ;)

see 2 comments

FlipIT Cloud: Orchestrating IT-as-a-Service on Software Gone Wild

Imagine being an IT administrator running a multi-tenant enterprise environment (example: an SMB business center). How many things would you have to configure to add a new tenant? How about adding a new user for an existing tenant?

The engineers behind the scenes of FlipIT cloud service ended up with a 40-page configuration guide when they started the service years ago… and obviously decided full-blown automation is the only way to go.

read more Add comment

So You’re an Open Source Shop? Really?

I carried out an interesting quiz during one of my Interop workshop:

  • How many use Linux-based servers? Almost everyone raised their hands;
  • How many use Apache or Tomcat web servers? Yet again, almost everyone.
  • How many run applications written in PHP, Python, Ruby…? Same crowd (probably even a bit more).
  • How many use Nginx, Squid or HAProxy for load balancing? Very few.

Is there a rational explanation for this seemingly nonsensical result?

read more see 9 comments

Just Published: HP and Dell Force10 Data Center Switches

Want to know how HP IRF works? What its limitations are? Which data center protocols HP 5900 supports? How Dell Force10 switches handle MLAG? How well are HP and Dell supporting OpenFlow?

You’ll get answers to all these questions in the videos recently published in the Data Center Fabric Architecture webinar (also available as part of yearly subscription).

see 1 comments

Does a Cloud Orchestration System Need an Underlying SDN Controller?

A while ago I had an interesting discussion with a fellow SDN explorer, in which I came to a conclusion that it makes no sense to insert an overlay virtual networking SDN controller between cloud orchestration system and virtual switches. As always, I missed an important piece of the puzzle: federation of cloud instances.

2014-11-04 16:48Z: CJ Williams sent me an email with information on SDN controller in upcoming Windows Server release. Thank you!

read more see 1 comments

Use a Disaster Recovery Project to Build Your New Cloud

It doesn’t make sense to build a new data center network to support legacy bare-metal server infrastructure. You’ll have to use relatively expensive 1G/10G ports to be able to connect the current and future servers, and once the server and virtualization engineers wake up and do hardware refresh you’ll end up with way too many ports (oh, and you do know that transceivers could cost more than the switching hardware, right?).

read more see 1 comments

Overlay-to-Underlay Network Interactions: Document Your Hidden Assumptions

If you listen to the marketing departments of overlay virtual networking vendors, it looks like the world is a simple place: you deploy their solution on top of any IP fabric, and it all works.

You’ll hear a totally different story from the physical hardware vendors: they’ll happily serve you a healthy portion of FUD, hoping you swallow it whole, and describe in gory details all the mishaps you might encounter on your virtualization quest.

The funny thing is they’re all right (not to mention the really fun part when FUDders change sides ;).

read more Add comment

Bad Ideas and Abominations

This post SHOULD have been published on April 1st, but I need to define the terminology for another upcoming post, so here it is ;)

RFC 2119 defines polite words to use when something really shouldn’t be done. Some network designs I see deserve more colorful terminology.

2014-11-02: Updated with reference to RFC 6919 (/HT to @LapTop006)

read more see 3 comments

New Webinar: Scaling Overlay Virtual Networks

You can get an overlay virtual networking solution from almost every major hypervisor- and data center networking vendor. Do you ever wonder which one to choose for your large-scale environment? I’m positive you’d get all of them up and running in a one-rack environment, but what if you happen to be larger than that?

We’ll try to address scalability hiccups and roadblocks you might encounter on your growth path in Scaling Overlay Virtual Networks webinar (get your free ticket here).

read more Add comment

Cumulus Linux in Real Life on Software Gone Wild

A year ago Matthew Stone first heard about Cumulus Linux when I ranted about it on a Packet Pushers podcast (which only proves that any publicity is good publicity even though some people thought otherwise at that time), and when his cloud service provider company started selecting ToR switches he considered Cumulus together with Cisco and Arista… and chose Cumulus.

read more see 11 comments

IPv6 in a Global Company – a Real-World Example

More than a year ago I wrote a response to a comment Pascal wrote on my Predicting the IPv6 BGP table size blog post. I recently rediscovered it and figured out that it’s (unfortunately) as relevant as it was almost 18 months ago.

Other people have realized we have this problem in the meantime, and are still being told to stop yammering because the problem is not real. Let’s see what happens in a few years.

read more see 14 comments

All You Need Are Two Top-of-Rack Switches

Every time I’m running a classroom version of my Designing the Cloud Infrastructure workshop, I start with a simple question: “Who has more than 2000 VMs or bare-metal servers in the data center?

I might see three hands on a good day; 90-95% of the audience have smaller data centers… and some of them get disappointed when I tell them they don’t need more than two ToR switches in their data center.

read more see 9 comments

Just Published: Juniper Data Center Switches

Want to know what the difference between Virtual Chassis and Virtual Chassis Fabric is? How Local Link Bias works? How ISSU on QFX 5100 works even though the box doesn’t have two supervisor boards? You’ll find answers to all these questions in new videos describing Juniper data center switches.

see 1 comments

Workload Mobility and Reality: Bandwidth Constraints

People talking about long-distance workload mobility and cloudbursting often forget the physical reality documented in the fallacies of distributed computing. Today we’ll focus on bandwidth, in a follow-up blog post we’ll deal with its ugly cousin latency.

TL&DR summary: If you plan to spread application components across the network without understanding their network requirements, you’ll get the results you deserve.

read more see 2 comments

Border6 Non-Stop Internet: a Commercial BGP-Based SDN

Several SDN solutions that coexist with the traditional control- and data planes instead of ripping them out and replacing them with the new awesomesauce use BGP to modify the network’s forwarding behavior.

Border6 decided to turn that concept into a commercial product that we dissected in Episode 12 of Software Gone Wild podcast.

Enjoy the show (this time in video format).

Add comment

Networking Is Not as Special as We Think It Is

I was listening to the Packet Pushers show #203 – an interesting high-level discussion of policies (if you happen to be interested in those things) – and unavoidably someone had to mention how the networking is all broken because different devices implement the same functionality in different ways and use different CLI/API syntax.

read more see 9 comments

How to Get into the Top N%

Michael Church wrote an interesting answer on Quora, describing a logarithmic scale of programming skills and (even more importantly) hints to follow to get from n00b into the top N% (for some small value of N):

  • Budget 7–14 years;
  • Study voraciously;
  • Build things when you don’t know that you’ll succeed;
  • Network to get new ideas;
  • Job hop when you stop learning.

Replace “programmer” with “networking engineer” and read the whole answer ;)

see 1 comments

IPv6 High Availability Strategies on NIL TV

I had a shorter version of my IPv6 High Availability talk @ Slovenian IPv6 summit this spring. The video is online, but wouldn’t be of much use to anyone but both Slovenian readers of this blog.

The English version of that same talk is now available on NIL TV (or you could decide to go for the full webinar or whole IPv6 track).

Add comment

VXLAN and OTV: The Saga Continues

Randall Greer left a comment on my Revisited: Layer-2 DCI over VXLAN post saying:

Could you please elaborate on how VXLAN is a better option than OTV? As far as I can see, OTV doesn't suffer from the traffic tromboning you get from VXLAN. Sure you have to stretch your VLANs, but you're protected from bridging failures going over your DCI. OTV is also able to have multiple edge devices per site, so there's no single failure domain. It's even integrated with LISP to mitigate any sub-optimal traffic flows.

Before going through the individual points, let’s focus on the big picture: the failure domains.

read more see 15 comments

Data Center Design Case Studies on Amazon – Take 2

In July I wrote about an Amazon Kindle version of my Data Center Design Case Studies book and complained about their royalties model. Someone quickly pointed out how to adapt to their system: split the book into multiple volumes and charge $9.99 for each.

It took me months to get there, but the first two volumes are finally on Amazon:

read more Add comment

Interop New York: It Was Great Fun

Last week’s Interop New York was hard work (three workshops in two days), but also lots of nerdy fun. I love doing workshops with smart participants who bring their real-life problems to the room and challenge my assumptions and conclusions, and I had plenty of these interactions during the week. Thank you all (you know who you are)!

read more see 1 comments

Bufferbloat Killed my HTTP Session… or not?

Every now and then I get an email from a subscriber having video download problems. Most of the time the problem auto-magically disappears (and there’s no indication of packet loss or ridiculous latency in traceroute printout), but a few days ago Henry Moats managed to consistently reproduce the problem and sent me exactly what I needed: a pcap file.

TL&DR summary: you have to know a lot about application-level protocols, application servers and operating systems to troubleshoot networking problems.

read more see 22 comments

Replacing a Central Firewall

During one of my ExpertExpress engagements I got an interesting question: “could we replace a pair of central firewalls with iptables on the Linux server?

Short answer: Maybe (depending on your security policy), but I’d still love to see some baseline scrubbing before the traffic hits the server – after all, if someone pwns your server, he’ll quickly turn off iptables.

read more see 1 comments

Schprokits with Jeremy Schulman on Software Gone Wild

Jeremy Schulman was the driving force behind the Puppet agent that Juniper implemented on some Junos switches (one of the first fully supported Puppet-on-a-switch implementations). In the meantime, he quit Juniper and started his own company focused on a network automation product – more than enough reasons to chat with him on Software Gone Wild.

read more Add comment

Quick Guide to my Interop New York Sessions

I’m running or participating in five workshops or sessions during next week’s Interop New York. Three of them build on each other, so you might want to attend all of them in sequence:

Designing Infrastructure for Private Clouds starts with requirements gathering phase and focuses on physical infrastructure design decisions covering compute, storage, physical and virtual networking, and network services. If you plan to build a private (or a reasonable small public) cloud, start here.

read more Add comment

Network Programmability 101: The Problem

In the first part of the Network Programmability webinar Matt Oswalt described some of the major challenges most networks are facing today:

  • Why is everyone claiming that the network is so slow to change?
  • Is that really the case? Why?
  • Why is the manual configuration culture so widespread in networking?
  • How does the holistic thinking in the design phase dissolve into the box mentality of CLI commands?
  • How does the box mentality limit the scalability of network deployments?
see 1 comments

Connecting Virtual Routers to the Outside World

Stefan de Kooter (@sdktr) sent me a follow-up question to my Going All Virtual with Virtual WAN Edge Routers blog post:

How would one interface with external Internet in this scenario? I totally get the virtual network assets mantra, but even a virtual BGP router would need to get a physical interconnect one way or another.

As always, there are plenty of solutions depending on your security needs.

read more Add comment

SDN Deployment Considerations

Are you lucky enough to be one of the 87% of North American enterprises that plan to have SDN in production by 2016 or one of the 53% of the companies that plan to have SDN deployed in the near future? Even though we all know how inflated these claims are, you might have to start considering the deployment aspects of a solution a $vendor will persuade your CIO to buy.

read more Add comment

Formal Announcement: Software Gone Wild Podcast

If you’ve been reading my blog in the last few months, you might have noticed that I started a new podcast focused on software-defined everything (hence the name: Software Gone Wild – thanks to Jason Edelman).

The latest episodes are always available on this page; you can also subscribe to the podcast feed in RSS, Atom or iTunes format… and if you wonder why we need yet-another podcast, read the About Software Gone Wild document.

see 1 comments

Virtual Networking in CloudStack

If you mention open-source cloud orchestration tools these days, everyone immediately thinks about OpenStack (including the people who spent months or years trying to make it ready for production use). In the meantime, there are at least two other comparable open-source products (CloudStack and Eucalyptus) that nobody talks about. Obviously having a working product is not as sexy as having 50+ vendors and analysts producing press releases.

read more Add comment

Dynamic FCoE – Sparse-Mode FCoE Strikes Again

A while ago Cisco added dynamic FCoE support to Nexus 5000 switches. It sounded interesting and I wanted to talk about it in my Data Center Fabrics update session, but I couldn’t find any documentation at that time.

In the meantime, the Configuring Dynamic FCoE Using FabricPath configuration guide appeared on Cisco’s web site and J Metz wrote a lengthly blog post explaining how it all works, triggering a severe attack of déjà vu.

read more see 6 comments

The Four Paths to SDN

After the initial onslaught of SDN washing, four distinct approaches to SDN have started to emerge, from centralized control plane architectures to smart reuse of existing protocols.

As always, each approach has its benefits and drawbacks, and there’s no universally best solution. You just got four more (somewhat immature) tools in your toolbox. And now for the details.

read more see 9 comments

SIGS & Carrier’s Lunch DC Day: An Event Definitely worth Visiting

I spent last Tuesday in Bern attending the SIGS DC Day Event, and came back home extremely pleasantly surprised. The conference was nice and cozy, giving everyone plenty of opportunities to chat about data center technical challenges (thanks for all the wonderful conversations we had – you know who you are!).

Having the opportunity to meet fellow networking engineers and compare notes is great, but it’s even better to combine that with new knowledge, and that’s where the event really excelled.

read more Add comment

Tech Talks: The Essence of MPLS

Seamus Gilchrist sent me a fantastic list of MPLS- and MPLS-TE-related questions. Instead of starting an email exchange we agreed on something that should benefit a wider community: a lengthy whiteboard session discussing the basics of MPLS, MPLS-TE, load balancing and QoS in MPLS networks…

The first part of our conversation is already online: The Essence of MPLS.

Add comment

Open-Source Hybrid Cloud Reference Architecture on Software Gone Wild

A while ago Rick Parker told me about his amazing project: he started a meetup group that will build a reference private/hybrid cloud heavily relying on virtualized network services, and publish all documentation related to their effort, from high-level architecture to device and software configurations, and wiring plans.

In Episode 8 of Software Gone Wild Rick told us more about his project, and we simply couldn’t avoid a long list of topics including:

read more Add comment

IPv6 Neighbor Discovery (ND) and Multicast Listener Discovery (MLD) Challenges

A few days ago Garrett Wollman published his exasperating experience running IPv6 on large L2 subnets with Juniper Ex4200 switches, concluding that “… much in IPv6 design and implementation has been botched by protocol designers and vendors …” (some of us would forcefully agree) making IPv6 “…simply unsafe to run on a production network…

The resulting debate on Hacker News is quite interesting (and Andrew Yourtchenko is trying hard to keep it close to facts) and definitely worth reading… but is ND/MLD really as broken as some people claim it is?

read more see 6 comments

Is Anyone Using DMVPN-over-IPv6?

One of my readers sent me an interesting challenge: they’re deploying a new DMVPN WAN, and as they cannot expect all locations to have native (non-NAT) IPv4 access, they plan to build the new DMVPN over IPv6. He was wondering whether it would work.

Apart from “you’re definitely going in the right direction” all I could tell him was “looking at the documentation I couldn’t see why it wouldn’t work” Has anyone deployed DMVPN over IPv6 in a production network? Any hiccups? Please share your experience in the comments. Thank you!

see 6 comments

Scalability Enhancements in Cisco Nexus 1000V

The latest release of Cisco Nexus 1000V for vSphere can handle twice as many vSphere hosts as the previous one (250 instead of 128). Cisco probably did a lot of code polishing to improve Nexus 1000V scalability, but I’m positive most of the improvement comes from interesting architectural changes.

read more Add comment

Snabb Switch Deep Dive on Software Gone Wild

The pilot episode of Software Gone Wild podcast featuring Snabb Switch created plenty of additional queries (and thousands of downloads) – it was obviously time for another deep dive episode discussing the intricate innards of this interesting virtual switch.

During the deep dive Luke Gorrie, the mastermind behind the Snabb Switch, answered a long list of questions, including:

read more see 1 comments

Just Published: SDN and OpenFlow – The Hype and the Harsh Reality

If you’re a regular reader of my blog, you know that I spent a lot of time during the last three years debunking SDN myths, explaining the limitations of OpenFlow and pointing out other technologies one could use to program the network.

During the summer of 2014 I organized my SDN- and OpenFlow-related blog posts into a digital book. I want to make this information as useful and as widely distributed as possible – for a limited time you can download the PDF free of charge.

Learn more about the book

see 4 comments

Network Infrastructure as Database

A while ago I wrote about the idea of treating network infrastructure (and all other infrastructure) as code, and using the same processes application developers are using to write, test and deploy code to design and implement networks.

That approach clearly works well if you can virtualize (and clone ad infinitum) everything. We can virtualize appliances or even routers, but installed equipment and high-speed physical infrastructure remain somewhat resistant to that idea. We need a different paradigm, and the best analogy I could come up with is a database.

read more see 11 comments

Is Data Center Trilogy Package the Right Fit to Understand Long Distance vMotion Challenges?

A reader sent me this question:

My company will have 10GE dark fiber across our DCs with possibly OTV as the DCI. The VM team has also expressed interest in DC-to-DC vMotion (<4ms). Based on your blogs it looks like overall you don't recommend long-distance vMotion across DCI. Will the "Data Center trilogy" package be the right fit to help me better understand why?

Unfortunately, long-distance vMotion seems to be a persistent craze that peaks with a predicable period of approximately 12 months, and while it seems nothing can inoculate your peers against it, having technical arguments might help.

read more see 3 comments

Fate Sharing in IP Networks

My good friend Tiziano complained about the fact that BGP considers next hop reachable if there’s an entry in the IP routing table even though the router cannot even ping the next hop.

That behavior is one of the fundamental aspects of IP networks: networks built with IP routing protocols rely on fate sharing between control and data planes instead of path liveliness checks.

read more see 3 comments

Toolsmith @ Netflix on Software Gone Wild

I first met Elisa Jasinska when she had one of the coolest job titles I ever saw: Senior Packet Herder. Her current job title is almost as cool: Senior Network Toolsmith @ Netflix – obviously an ideal guest for the Software Gone Wild podcast.

In our short chat she described some of the tools she’s working on, including an adaptation of pmacct to environments with numerous BGP exit points (more details in her NANOG presentation).

Add comment

VMware EVO:RAIL – One Stop Shopping for Your Private Cloud

Building a private cloud infrastructure tends to be a cumbersome process: even if you do it right, you oft have to deal with four to six different components: orchestration system, hypervisors, servers, storage arrays, networking infrastructure, and network services appliances.

read more Add comment

Interview: Reduce Costs and Gain Efficiencies with SDDC

A few days ago I had an interesting interview with Christoph Jaggi discussing the challenges, changes in mindsets and processes, and other “minor details” one must undertake to gain something from the SDDC concepts. The German version of the interview is published on; you’ll find the English version below.

read more see 2 comments

Finally: a Virtual Switch Supports BPDU Guard

Nexus 1000V release 5.2(1)SV3(1.1) was published on August 22nd (I’m positive that has nothing to do with VMworld starting tomorrow) and I found this gem in the release notes:

Enabling BPDU guard causes the Cisco Nexus 1000V to detect these spurious BPDUs and shut down the virtual machine adapters (the origination BPDUs), thereby avoiding loops.

It took them almost three years, but we finally have BPDU guard on a layer-2 virtual switch (why does it matter). Nice!

see 3 comments Is on CloudFlare (and IPv6)

After a week of testing, I decided to move the main web site ( as well as some of the resource servicing hostnames to CloudFlare CDN. Everything should work fine, but if you experience any problems with my web site, please let me know ASAP.

2014-08-27: Had to turn off CloudFlare (and thus IPv6). They don't seem to support HTTP range requests, which makes video startup time unacceptable. Will have to move all video URLs (where the HTTP range requests are expected coming from streaming clients) to a different host name, which will take time.

Collateral benefit: is now fully accessible over IPv6 – register for the Enterprise IPv6 101 webinar if you think that doesn’t matter ;)

see 4 comments

Cloud Orchestration System Is an Ideal Controller Use Case

A while ago I explained why OpenFlow might be a wrong tool for some jobs, and why centralized control plane might not make sense, and quickly got misquoted as saying “controllers don’t scale”. Nothing could be further from the truth, properly architected controller-based architectures can reach enormous scale – Amazon VPC is the best possible example.

read more see 6 comments

The Impact of Data Gravity: a Campfire Story

Here’s an interesting story illustrating the potential pitfalls of multi-DC deployments and the impact of data gravity on application performance.

Long long time ago on a cloudy planet far far away, a multinational organization decided to centralize their IT operations and move all workloads into a central private cloud.

read more see 6 comments

Pmacct: the Traffic Analysis Tool with Unpronounceable Name

SDN evangelists talking about centralized traffic engineering, flow steering or bandwidth calendaring sometimes tend to gloss over the first rule of successful traffic engineering: Know Thy Traffic.

In a world ruled by OpenFlow you’d expect the OpenFlow controller to know all the traffic; in more traditional networks we use technologies like NetFlow, sFlow or IPFIX to report the traffic statistics – but regardless of the underlying mechanism, you need a tool that will collect the statistics, aggregate them in a way that makes them usable to the network operators, report them, and potentially act on the deviations.

read more Add comment

Revisited: Layer-2 DCI over VXLAN

I’m still getting questions about layer-2 data center interconnect; it seems this particular bad idea isn’t going away any time soon. In the face of that sad reality, let’s revisit what I wrote about layer-2 DCI over VXLAN.

VXLAN hasn’t changed much since the time I explained why it’s not the right technology for long-distance VLANs.

read more see 24 comments

Do you really need to see all 512K Internet routes?

Last week the global routing table (as seen from some perspectives) supposedly exceeded 512K routes, and weird things started to happen to some people that are using old platforms that by default support 512K IPv4 routes in the switching hardware.

I’m still wondering whether the BGP table size was the root cause of the observed outages. Cisco’s documentation (at least this document) is pretty sloppy when it comes to the fact that usually 1K = 1024, not 1000 – I’d expect the hard limit to be @ 524.288 routes … but then maybe Cisco’s hardware works with decimal arithmetic.

read more see 6 comments

MPLS Load Sharing – Data Plane Considerations

In a previous blog post I explained how load sharing across LDP-controlled MPLS core works. Now let’s focus on another detail: how are the packets assigned to individual paths across the core?

2014-08-14: Additional information was added to the blog post based on comments from Nischal Sheth, Frederic Cuiller and Tiziano Tofoni. Thank you!

read more see 6 comments

Should I Go for CCIE or Some Other Certification?

One of my readers sent me this question:

I am already CCIE and work as a network engineer with pretty good salary. But I think that I am losing some passion for Cisco networking and have interests in many other technologies. Currently I am very interested in Linux and Python development. Is it worth to add some Red Hat certification along CCIE or should I pursue another CCIE?

I think “should I go for CCIE or RHCE” is the wrong question.

read more see 9 comments

VXLAN Encapsulation in Juniper Contrail

VXLAN is becoming de-facto encapsulation standard for overlay virtual networks (at least according to industry pundits and marketing gurus working for companies with VXLAN-based products) – even Juniper Contrail, which was traditionally a pure MPLS/VPN architecture uses it.

Not so fast – Contrail is using VXLAN packet format to carry MPLS labels between hypervisors and ToR switches.

read more see 4 comments

STP and Expert Beginners

Maxim and myself continued our STP discussion and eventually agreed that while STP might not be the best protocol out there (remember: it had to run on Z80 CPU), it’s the only standardized thing that prevents nasty forwarding loops, prompting Maxim to ask another seemingly simple question:

What's so wrong with STP, that there are STP haters out there turning it off wherever they see it?

Welcome to the wonderful world of Expert Beginners.

read more see 8 comments

Could You Replace MPLS/VPN with IPSec-over-Internet?

Someone recently sent me this scenario:

Our CIO has recently told us that he wants to get rid of MPLS because it is too costly and is leaning towards big Internet lines running IPSEC VPNs to connect the whole of Africa.

He was obviously shopping around for free advice (my friend Jeremy Stretch posted his answers to exactly the same set of questions not so long ago); here are the responses I wrote to his questions:

read more see 5 comments

Layer-3 Switching over VXLAN Revisited

My Trident 2 Chipset and Nexus 9500 blog post must have hit a raw nerve or two – Bruce Davie dedicated a whole paragraph in his Physical Networks in Virtualized Networking World blog post to tell everyone how the whole thing is a non-issue and how everything’s good in the NSX land.

It’s always fun digging into more details to figure out what’s really going on behind the scenes; let’s do it.

read more see 4 comments

Next Chapter in Data Center Design Case Studies

When I published the Data Center Design Case Studies book almost exactly a month ago, three chapters were still missing – but that was the only way to stop the procrastination and ensure I’ll write them (I’m trying to stick to published deadlines ;).

The first one of the missing chapters is already finished and available to subscribers and everyone who bought the book or Designing Private Cloud Infrastructure webinar (you’ll also get a mailing on Sunday to remind you to download the fresh copy of the PDF).

The Amazon Kindle version will be updated in a few days.

Add comment

Network Automation @ Spotify on Software Gone Wild

What can you do if you have a small team of networking engineers responsible for four ever-growing data centers (with several hundred network devices in each of them)? There’s only one answer: you try to survive by automating as much as you can.

In the fourth episode of Software Gone Wild podcast David Barosso from Spotify explains how they use network automation to cope with the ever-growing installed base without increasing the size of the networking team.

read more Add comment

There Is no Paradigm Shift – Good Applications Were Always Network-Aware

Someone left the following comment on one of my blog posts:

There is a paradigm shift that I don’t think most application developers understand. In a traditional enterprise model, the network is built around the application requirements, now we are saying the application has to build around the network.

I would say there’s no paradigm shift – developers of well-performing applications were always aware of laws of physics.

read more see 1 comments

The Virtual Design Master Is Starting Today

A while ago I was listening to a podcast describing Project Runway for geeks – the Virtual Design Master – and decided to do what I could to help them ;)

The second season is starting today – the list of participants is already online (and you might watch the videos of the first season while waiting for the first challenge).

Add comment

Build a Cloud in Three Easy Steps

Occasionally I get a question about some totally impossible implementation detail (example: can we use OpenStack OVS plugin on VMware to avoid buying NSX?). These questions are often coming from people who painted themselves into a corner and are now desperately looking for MacGyver’s shoelaces to pull themselves out.

It’s easy to blame the engineer who tries to do the obviously impossible, but it’s often not his fault – these days a lot of technical people get pulled into the game of Build a Cloud in Three Easy Steps.

read more see 4 comments

Network Programmability with David Gee on Software Gone Wild

For the second episode of Software Gone Wild I got a truly interesting guest: David Gee, a network engineer already working on numerous network programmability and orchestration deployment.

During our half-hour chat we couldn’t avoid the question of whether every networking engineer will become a programmer and David provided an interesting answer: you don’t have to program, but you’ll definitely have to start thinking more like a good programmer.

read more see 3 comments

What Is This API Thingy?

A reader sent me this question:

I am hearing a lot about API in reference to SDN. I do not have any software or programming background but would like to understand this API in practical way. Could you help me?

TL&DR: API is CLI for program-to-program communication

read more see 2 comments

Mice, Elephants and Virtual Switches

The Mice and Elephants is a traditional QoS fable – latency-sensitive real time traffic (or request-response protocol like HTTP) stuck in the same queue behind megabytes of file transfer (or backup or iSCSI) traffic.

The solution is also well known – color the elephants pink (aka DSCP marking) and sort them into a different queue – until the reality intervenes.

read more see 4 comments

Snabb Switch and NFV on OpenStack in Software Gone Wild

Last September I received a peculiar tweet from Luke Gorrie pointing me to a software switch pushing 200 Gbps through an Intel server literally hours after I’d watched the Deutsche Telekom Terastream presentation, so I mentioned Luke’s Snabb Switch as a potential performance solution in an email to Ian Farrer… and before Ian managed to reply, Luke was already working for Deutsche Telekom.

read more see 1 comments

Unnumbered OSPF Interfaces in Quagga (and Cumulus)

Carlos Mendioroz sent me an interesting question about unnumbered interfaces in Cumulus Linux and some of the claims they make in their documentation.

TL&DR: Finally someone got it! Kudos for realizing how to use an ancient trick to make data center fabrics easier to deploy (and, BTW, the claims are exaggerated).

read more see 20 comments

Why is IPv6 layer-2 security so complex (and how to fix it)

After the excellent IPv6 security presentation Eric Vyncke had @ 9th Slovenian IPv6 summit someone asked me: “Why is IPv6 first-hop security so complex? It looks like the developers of IPv6 protocol stack tried to make users anonymous and made everyone’s life complex while doing that.

Well, he was totally surprised by my answer: “The real reason IPv6 first-hop security is so complex is the total mess we made of L2/L3 boundary.”

read more see 21 comments

Virtual Switch 101

One of my readers sent me this question:

I am a CCNP-certified L3/L4 network engineer working purely on Cisco products and network technologies for the last 10 years of experience. How long will it take for a typical person like me to get a foothold into VM/SAN?

It obviously depends on how deep you want to go, but the virtual networking part of it shouldn’t be too hard.

read more see 1 comments

To Get a Job Done Well, You Need Proper Training

The “bring Amazon Web Services mentality back home” blog post generated the expected comments, from “developers have no clue about networking or network services” to “we went through the whole thing and failed badly.”

Well, even though it might have seemed so, I didn’t advocate letting the developers go unchecked, I was just pointing out that double standards make no sense.

read more see 5 comments

Just Out: Data Center Design Case Studies

After months of being on a back burner, my first self-publishing book is out: Data Center Design Case Studies are available on my web site (Amazon Kindle version coming whenever I find time to figure out all the details).

Subscribers can download the book immediately (you’ll find it in the new Books section), as can anyone who bought the Designing Private Cloud Infrastructure webinar (you’ll find it in the webinar materials), and you can buy a copy on my web site.

More information

see 2 comments

Updated: Metro Ethernet and Carrier Ethernet Encryption

Update 2014-06-18 05:13Z - We slashdotted Christoph's site yesterday. He moved to a new server during the night; the links should work now.

Christoph Jaggi focused on analyzing Metro Ethernet and Carrier Ethernet encryption gear. The introductory part of this year’s report has just been published and it’s definitely worth reading even if you have no immediate plans to buy such gear – it’s a nice overview document covering numerous encryption technologies, key distribution systems, network topologies, and operational aspects. If you want to get in-depth evaluation of individual vendors or solutions, you’ll obviously have to contact Christoph.

Speaking of security, if you wonder how to secure your credit card handling environment, register for Michele Chubirka's PCI DSS webinar.

see 3 comments

Trident 2 Chipset and Nexus 9500

Most recently launched data center switches use the Trident 2 chipset, and yet we know almost nothing about its capabilities and limitations. It might not work at linerate, it might have L3 lookup challenges when faced with L2 tunnels, there might be other unpleasant surprises… but we don’t know what they are, because you cannot get Broadcom’s documentation unless you work for a vendor who signed an NDA.

Interestingly, the best source of Trident 2 technical information I found so far happens to be the Cisco Live Nexus 9000 Series Switch Architecture presentation (BRKARC-2222). Here are a few tidbits I got from that presentation and Broadcom’s so-called datasheet.

read more see 8 comments

Just Found Two Data Center / Virtualization Blogs

I found two interesting data center/virtualization blogs this week (can’t remember the last time I found two new blogs in a week):

  • Pedro Marques’ blog – it focuses primarily on Open Contrail, but you’ll find interesting gems questioning the wisdom of OpenFlow and Segment Routing (among other things);
  • Aldrin Isaac’s blog – unfortunately he stopped writing in the last months, but some of his older posts are pure gold.

Enjoy – and poke Aldrin to continue writing ;)

Add comment

Programming the Way I Remember It

Jure Špiler, an old friend of mine (and my first boss) sent me a link to a fantastic YouTube video documenting how we did programming on PDP-11 computers when I started my programming career (note: I did not enter compiled assembly code with console switches on PDP-11, but I did do that on an 6800-based box with 2K of RAM).

Related videos tab revealed another gem: PDP-11/34 with RL-01 disk drives – the very same system where I started my IT career as a high-school kid. Enjoy!

see 2 comments

IP Addresses Considered Harmful

A long while ago I wrote about the brokenness of socket API and lack of session layer in TCP/IP stack. Looks like I’m not the only one with heretic opinions; Fred Baker reached similar conclusions in his Happier Eyeballs draft and Brian Carpenter recently published a long article title IP Addresses Considered Harmful which documents (among other things) the history of socket API and the reasons DNS isn’t tightly integrated with it. Both documents are definitely worth reading.

On the other hand, if you love listening to old rants, here’s a video I recorded more than 3 years ago (part of the Upcoming Internet Challenges webinar). The truly sad part: nothing changed in the meantime.

Add comment

Is OpenFlow the Best Tool for Overlay Virtual Networks?

Overlay virtual networks were the first commercial-grade OpenFlow use case – Nicira’s Network Virtualization Platform (NVP – now VMware NSX for Multiple Hypervisors) used OpenFlow to program the hypervisor virtual switches (Open vSwitches – OVS).

OpenStack is using the same approach in its OVS Neutron plugin, and it seems Open Daylight aims to reinvent that same wheel, replacing OVS plugin running on the hypervisor host agent with central controller.

Does that mean one should always use OpenFlow to implement overlay virtual networks? Not really, OpenFlow is not exactly the best tool for the job.

read more see 4 comments

Can We Just Throw More Bandwidth at a Problem?

One of my readers sent me an interesting question:

I have been reading at many places about "throwing more bandwidth at the problem." How far is this statement valid? Should the applications(servers) work with the assumption that there is infinite bandwidth provided at the fabric level?

Moore’s law works in our favor. It’s already cheaper (in some environments) to add bandwidth than to deploy QoS.

read more see 10 comments

30 Years of Yammering

Some of the comments I get every time I write about the idea of merging network services deployment with application deployments, and making application developers responsible for the results of their code (aka DevOps) remind me of a very long list of “this will never work” sentiments I encountered in the 30 years I spent in IT and networking. Here are just a few of them:

read more Add comment

Use Webinars for Your Summer Bootcamp

A year ago I wrote about a university that used my webinars in a summer bootcamp. In the meantime, I formalized the program and added a simple form that allows you to:

  • Calculate how much academic workgroup subscriptions or webinar technology tracks cost;
  • Request a quote once you find the product that meets your budget.

If you have any questions about this program, just send me an email.

Add comment

Virtual Networking Implementation Taxonomy

I’m not sure I wrote about the taxonomy of numerous virtual networking implementations. Just in case, here it is ;)

Layer-2 or layer-3 networks?

Some virtual networking solutions emulate thick coax cable (more precisely, layer-2 switch), giving their users the impression of having regular VLAN-like layer-2 segments.

read more see 13 comments

FCoE and Nexus 1000v QoS

One of my readers wanted to deploy FCoE on UCS in combination with Nexus 1000v and wondered how the FCoE traffic impacts QoS on Nexus 1000v. He wrote:

Let's say I want 4Gb for FCoE. Should I add bandwidth shares up to 60% in the nexus 1000v CBWFQ config so that 40% are in the default-class as 1kv is not aware of FCoE traffic? Or add up to 100% with the assumption that the 1kv knows there is only 6Gb left for network? Also, will the Nexus 1000v be able to detect contention on the uplink even if it doesn't see the FCoE traffic?

As always, things aren’t as simple as they look.

read more Add comment

What Are Linux Containers?

Everyone (and The Register) talks about Linux containers these days like they would be the hottest thing invented this spring. In reality, it’s a seven year old technology that was heavily used by some smart web hosting companies for years (but of course some people think mentioning Google makes everything look sexier).

If you’re interested in a high-level overview of differences between Linux containers and more traditional virtual machines, watch the video from the Introduction to Virtual Networking webinar.

see 3 comments

It’s OK to Let Developers Go @ Amazon Web Services, but Not at Home? You Must Be Kidding!

Recently I was discussing the benefits and drawbacks of virtual appliances, software-defined data centers, and self-service approach to application deployment with a group of extremely smart networking engineers.

After the usual set of objections, someone said “but if we won’t become more flexible, the developers will simply go to Amazon. In fact, they already use Amazon Web Services.

read more see 5 comments

How Line-rate Is Line-rate?

During yesterday’s Data Center Fabrics Update presentation, one of the attendees sent me this question while I was talking about the Arista 7300 series switches:

Is the 7300 really non-blocking at all packet sizes? With only 2 x Trident-2 per line card it can't support non-blocking for small packets based on Trident-2 architecture.

It was an obvious example of vendor bickering, so I ignored the question during the presentation, but it still intrigued me, so I decided to do some more research.

read more see 6 comments

Queuing Mechanisms in Modern Switches

A long while ago there was an interesting discussion started by Brad Hedlund (then at Dell Force10) comparing leaf-and-spine (Clos) fabrics built from fixed-configuration pizza box switches with high-end chassis switches. The comments made by other readers were all over the place (addressing pricing, wiring, power consumption) but surprisingly nobody addressed the queuing issues.

This blog post focuses on queuing mechanisms available within a switch; the next one will address end-to-end queuing issues in leaf-and-spine fabrics.

read more see 8 comments

The best spam comment (so far)

Idiots posting random comments with (not-so-very) hidden links to whatever warez they're selling are utterly annoying, but there's always one-in-a-million chance for a hilarious one. This is what I got on the Traffic Trombone post:

The traffic across the network core and the end-to-end latency would be minimal (the same packet would traverse the core only once), increasing visits to my adult site.
see 1 comments

Data Center Protocols in HP Switches

HP representatives made some pretty bold claims during Networking Tech Field Day 1, including “our switches will support EVB, FCoE, SPB and TRILL.” I took them three years to deliver on those promises (and the hardware they had at that time doesn’t exactly support all features they promised), but their current protocol coverage is impressive.

read more see 17 comments

All Operations Engineers Should Have Firefighting Training

Recently I had a fantastic conversation with Erich Hohermuth, a networking engineer with an unusual hobby: he’s a professional firefighting instructor (teaching firefighters across the country how to do their job).

Volunteer fire departments are pretty popular in Central European countries, and so he’s not the only one on his team with that skillset. The (not so unexpected) side effect: these people are the best ones when it comes to fighting IT disasters.

read more Add comment

Load Balancing Across IP Subnets

One of my readers sent me this question:

I have a data center with huge L2 domains. I would like to move routing down to the top of the rack, however I’m stuck with a load-balancing question: how do load-balancers work if you have routed network and pool members that are multiple hops away? How is that possible to use with Direct Return?

There are multiple ways to make load balancers work across multiple subnets:

read more see 6 comments

Optimizing OpenFlow Hardware Tables

Initial OpenFlow hardware implementations used a simplistic approach: install all OpenFlow entries in TCAM (the hardware that’s used to implement ACLs and PBR) and hope for the best.

That approach was good enough to get you a tick-in-the-box on RFP responses, but it fails miserably when you try to get OpenFlow working in a reasonably sized network. On the other hand, many problems people try to solve with OpenFlow, like data center fabrics, involve simple destination-only L2 or L3 switching.

read more see 2 comments

Does Centralized Control Plane Make Sense?

A friend of mine sent me a challenging question:

You've stated a couple of times that you don't favor the OpenFlow version of SDN due to a variety of problems like scaling and latency. What model/mechanism do you like? Hybrid? Something else?

Before answering the question, let’s step back and ask another one: “Does centralized control plane, as evangelized by ONF, make sense?

read more see 6 comments

It Doesn’t Make Sense to Virtualize 80% of the Servers

A networking engineer was trying to persuade me of importance of hardware VXLAN VTEPs. We quickly agreed physical-to-virtual gateways are the primary use case, and he tried to illustrate his point by saying “Imagine you have 1000 servers in your data center and you manage to virtualize 80% of them. How will you connect them to the other 200?” to which I replied, “That doesn’t make any sense.” Here’s why.

read more see 13 comments

SDN, OpenFlow, NFV and SDDC: Hype and Reality (2-day Workshop)

There are tons of SDN workshops, academies, and webinars out there, many of them praising the almost-magic properties of the new technologies, or the shininess of vendors’ new gadgets and strategic alliances. Not surprisingly, the dirty details of real-life deployments aren’t their main focus.

As you might expect, my 2-day workshop isn’t one of them.

read more Add comment

Declarative and Procedural Programming (and How I Got It all Wrong)

During a recent NetOps-focused discussion trying to figure out where Puppet/Chef/Ansible/… make sense in the brave new SDN-focused networking world I made this analogy: “Puppet manifest is like Prolog, router configuration is like Java or C++.” It’s a nice sound bite. It’s also totally wrong.

If you never met Prolog, you might consider yourself lucky. Or you might want to figure out what it is (warning: it might make your head explode). Just joking, I actually quite liked it in my programming days.

read more see 8 comments

IPv6 Resources on

As expected, ARIN wasn’t that far behind APNIC and RIPE in IPv4 allocations and is now down to the last /8. Maybe it’s time for the last denialists to wake up and start considering IPv6 (or not – consultants love panicking customers)… and the new IPv6 resources page on might help you get IPv6-fluent (hint: don’t miss the must-read documents section).

see 2 comments