Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

9 module online course

Start now!

How Did We End with 1500-byte MTU?

A subscriber sent me this intriguing question:

Is it not theoretically possible for Ethernet frames to be 64k long if ASIC vendors simply bothered or decided to design/make chipsets that supported it? How did we end up in the 1.5k neighborhood? In whose best interest did this happen?

Remember that Ethernet started as a shared-cable 10 Mbps technology. Transmitting a 64k frame on that technology would take approximately 50 msec (or as long as getting from East Coast to West Coast). Also, Ethernet had no tight media access control like Token Ring, so it would be possible for a single host to transmit multiple frames without anyone else getting airtime, resulting in unacceptable delays.

read more see 4 comments

How Do You Provision a 500-Switch Network in a Few Days?

TL&DR: You automate the whole process. What else do you expect?

During the Tech Field Day Extra @ Cisco Live Europe 2019 we were taken on a behind-the-stage tour that included a chat with people who built the Cisco Live network, and of course I had to ask how they automated the whole thing. They said “well, we have the guy that wrote the whole system onsite and he’ll be able to tell you more”. Turns out the guy was my good friend Andrew Yourtchenko who graciously showed the system they built and explained the behind-the-scenes details.

read more see 1 comments

New Content: EVPN on Linux Hosts and External Azure Connectivity

Dinesh Dutt added another awesome chapter to the EVPN saga last week explaining how (and why) you could run VXLAN encapsulation with EVPN control plane on Linux hosts (TL&DR: think twice before doing it).

In the last part of current Azure Networking series I covered external VNet connectivity, including VNet peering, Internet access, Virtual Network Gateways, VPN connections, and ExpressRoute. The story continues on February 6th 2020 with Azure automation.

You’ll need Standard ipSpace.net Subscription to access both webinars.

Add comment

Video: Retransmissions and Flow Control in Computer Networks

Grouping the features needed in a networking stack in bunch of layered modules is a great idea, but unfortunately it turns out that you could place a number of important features like error recovery, retransmission and flow control in a number of different layers, from data link layer dealing with individual network segments to transport layer dealing with reliable end-to-end transmissions.

So where should we put those modules? As always, the correct answer is it depends, in this particular case on transmission reliability, latency, and cost of bandwidth. You’ll find more details in the Retransmissions and Flow Control part of How Networks Really Work webinar.

You need free ipSpace.net subscription to watch the video, or a paid ipSpace.net subscriptions to watch the whole webinar.

Add comment

Automation Solution: Network Health State Report

How nice would it be to have a fabric health dashboard displaying a summary of numerous parameters you’re interested in (number of operational uplinks, number of BGP sessions…) for every switch in your fabric.

I’m positive you could hack something together using the customization capabilities of your favorite network management system… or you could write a simple data gathering solution like Stephen Harding did while attending the Building Network Automation Solutions online course.

I collected dozens of automation solutions created by course attendees in the last few years. Enjoy!
Add comment

VMware NSX Killed My EVPN Fabric

A while ago I had an interesting discussion with someone running VMware NSX on top of VXLAN+EVPN fabric - a pretty common scenario considering:

  • NSX’s insistence on having all VXLAN uplink from the same server in the same subnet;
  • Data center switching vendors being on a lemming-like run praising EVPN+VXLAN;
  • Non-FANG environments being somewhat reluctant to connect a server to a single switch.

His fabric was running well… apart from the weird times when someone started tons of new VMs.

read more see 2 comments

The Cost of Disruptiveness and Guerrilla Marketing

A Docker networking rant coming from my good friend Marko Milivojević triggered a severe case of Deja-Moo, resulting in a flood of unpleasant memories caused by too-successful “disruptive” IT vendors.

Before moving on, please note that the following observations were made from my outsider perspective. If I got something badly wrong, please correct me in a comment.

Imagine you’re working for a startup creating a cool new product in the IT infrastructure space (if you have an oversized ego you would call yourself “disruptive thought leader” on your LinkedIn profile) but nobody is taking you seriously. How about some guerrilla warfare: advertising your product to people who hate the IT operations (today we’d call that Shadow IT).

read more see 3 comments

Optimizing Environment Setup in Ansible Playbooks

Have you ever seen an Ansible playbook where 90% of the code prepares the environment, and then all the work is done in a few template and assemble modules? Here’s an alternative way of getting that done. Is it better? You tell me ;)

You might also want to explore similar Ansible articles and our Ansible for Networking Engineers content.
see 1 comments

Worth Reading: Anycast DNS in Enterprise Networks

Anycast (advertising the same IP address from multiple servers/locations) has long been used to implement scale-out public DNS services (the whole root DNS system runs on massive anycast), but it’s not as common in enterprise networks.

The blog posts written by Tom Bowles should get you there. He started with the idea and described his implementation using Infoblox DNS.

Want to know even more? I covered numerous load balancing mechanisms including anycast in Data Centers Infrastructure for Networking Engineers webinar.

Add comment

Redundant BGP Connectivity on a Single ISP Connection

A while ago Johannes Weber tweeted about an interesting challenge:

We want to advertise our AS and PI space over a single ISP connection. How would a setup look like with 2 Cisco routers, using them for hardware redundancy? Is this possible with only 1 neighboring to the ISP?

Hmm, so you have one cable and two router ports that you want to connect to that cable. There’s something wrong with this picture ;)

read more see 2 comments

Network Automation Beyond Configuration Templating

Remember Nicky Davey describing how he got large DMVPN deployment back on track with configuration templating? In his own words…:

Configuration templating is still as big win a win for us as it was a year ago. We have since expanded the automation solution, and reading the old blog post makes me realise how far we have come. I began working with this particular customer in May 2017, so 2 years now. At that time the new WAN project was on the horizon and the approach to network configuration was entirely manual.

Here’s how far he got in the meantime:

read more Add comment

New Content: Azure Networking and Automation Source-of-Truth

Last week I covered network security groups, application security groups and user-defined routes in the second live session of Azure Networking webinar.

We also had a great guest speaker on the Network Automation course: Damien Garros explained how he used central source-of-truth based on NetBox and Git to set up a network automation stack from the grounds up.

Recordings are already online; you’ll need Standard ipSpace.net Subscription to access the Azure Networking webinar, and Expert ipSpace.net Subscription to access Damien’s presentation. Azure Networking webinar is also part of our new Networking in Public Clouds online course.

Add comment

Changing Cisco IOS BGP Policies Based on IP SLA Measurements

This is a guest blog post by Philippe Jounin, Senior Network Architect at Orange Business Services.


You could use track objects in Cisco IOS to track route reachability or metric, the status of an interface, or IP SLA compliance for a long time. Initially you could use them to implement reliable static routing (or even shut down a BGP session) or trigger EEM scripts. With a bit more work (and a few more EEM scripts) you could use object tracking to create time-dependent static routes.

Cisco IOS 15 has introduced Enhanced Object Tracking that allows first-hop router protocols like VRRP or HSRP to use tracking state to modify their behavior.

read more see 18 comments

Networking in Public Clouds - New ipSpace.net Online Course

I have exciting news I’d love to share with you: we’re launching a new online course focused on networking in public clouds starting in February 2020 (I’ve been mulling over this idea and polishing the concept for almost 18 months, and finally it all came together ;)

With Go To The Cloud becoming the answer to all questions (regardless of what the question is), you can find tons of materials describing various aspects of public clouds, so you might wonder why I decided to enter the fray. The answer is simple: with everyone being focused on developers, there’s not much that an infrastructure engineer could use to help him survive when the developers move on and he’s left to manage whatever they put in place.

read more Add comment

Worth Reading: Koding Academies

Every few weeks I stumble upon an article (or twitter storm) in which someone claims you don’t need formal education to get started as a Software Engineer (or whatever else) - all you need is a coding academy/bootcamp and you're all set.

George V. Neville-Neil wrote a hilarious rebuttal of this idea followed by some pretty good advice. Hope you’ll enjoy it as much as I did ;)

see 1 comments

Worth Reading: SDN Ate My Hamster

A long while ago Daniel Dib wrote a nice blog post on “SDN will make the networking engineers obsolete” theme. While it sounds like beating a dead horse, the SDN craze isn’t subsiding, so another healthy dose of common sense might come handy.

Hint: if you’re not following Daniel’s blog, you should… even though he decided to make old farts’ life harder by publishing on LinkedIn.

Add comment

Net2Text: Natural-Language Interface to Network Operations

Sick-and-tired of intent-based GUIs that are barely better than CiscoWorks on steroids? How about asking Siri-like assistant queries about network state in somewhat-limited English and getting replies back in full-blown sentences?

Warning: you might be reentering the land of unicorns driving flying DeLoreans... but then keep in mind what Arthur Clarke had to say on this topic ;).

Welcome to Net2Text, another proof-of-concept tool created by the group led by Laurent Vanbever… who joined us for a short chat to discuss it, resulting in Episode 105 of Software Gone Wild.

Add comment

Upcoming Events and Webinars (October 2019)

The autumn 2019 webinar season is in full swing ;) We’re almost done with Azure Networking webinar (the last session will take place on October 10th) and the network automation course is nicely chugging along – a few weeks ago Matthias Luft talked about supply-chain security in open-source software and today we’ll enjoy the start with a single source of truth presentation by Damien Garros.

Dinesh Dutt is coming back on October 8th with another installment of EVPN saga, this time focused on running EVPN on Linux hosts, and on October 22nd Donald Sharp will tell us all about the underlying magic box – the Free Range Routing software.

But there are even more open-source goodies waiting for you: on October 15th we’ll have Pete Lumbis describing the new features Cumulus Linux got in the last year, including AutoBGP and AutoMLAG.

Most everything I mentioned above apart is accessible with Standard ipSpace.net Subscription, and you’ll need Expert Subscription to enjoy the automation course contents.

Add comment

Automation Solution: Deploy BGP Routing with YANG Data Models

A while ago Ruben Tripiana tried to configure BGP on Cisco IOS using IETF YANG data models… and failed. In Spring 2019 Building Network Automation Solutions online course Chris Crook decided to deploy BGP routing on multiple platforms using YANG data models instead of configuration templates. Not only did he succeed, he also documented his work and the tools he used, and published the solution so you can replicate his efforts.

You can find many more network automation solutions created by the attendees of our automation course in solutions showcase.

see 4 comments

On the Usability of OSI Layered Networking Model

Two weeks ago I replied to a battle-scar reaction to 7-layer OSI model, this time I’ll address a much more nuanced view from Russ White. Please read his article first (as always, it’s well worth reading) and when you come back we’ll focus on this claim:

The OSI Model does not accurately describe networks.

Like with any tool in your toolbox, you can view the 7-layer OSI model in a number of ways. In the case of OSI model, it can be used:

read more see 2 comments

Just Published: Network Services Integration in EVPN Fabrics

In last week’s continuation of EVPN never-ending story Lukas Krattiger described how you could use EVPN constructs (VNIs, VRFs) to implement service insertion, and how you could combine then with policy-based routing.

TL&DW: It’s bridging and routing ;)

You’ll need Standard ipSpace Subscription to access the videos.

Add comment

Worth Following: Explaining Computer Things

People who can explain complex topics in simple terms, or focus on the essentials of a particular topic are exceedingly rare… and two of the best are Randall Munroe of the XKCD fame and Julia Evans, the mastermind behind WizardZines. I loved her recent curl and git exercises, and I’m guessing a lot of people in this industry would benefit from her latest HTTP zine.

Similarly to what I did a long time ago with ipSpace.net, Julia recently decided to go all-in, leaving her job and focusing on explaining things. I hope it will work out and we’ll keep enjoying her tidbits of wisdom for years to come.

Add comment

Video: The Need for Network Layers

After identifying some of the challenges every network solution must address (part 1, part 2, part 3) we tried to tackle an interesting question: “how do you implement this whole spaghetti mess in a somewhat-reliable and structured way?

The Roman Empire had an answer more than 2000 years ago: divide-and-conquer (aka “eating the elephant one bite at a time”). These days we call it layering and abstractions.

In the Need for Network Layers video I listed all the challenges we have to address, and then described how you could group them in meaningful modules (called networking layers).

You need free ipSpace.net subscription to watch the video, or a paid ipSpace.net subscriptions to watch the whole webinar.

Add comment

Paramiko, Netmiko, NAPALM or Nornir?

I had a fantastic chat with David Bombal a while ago in which we covered tons of network automation topics including “should I use Nornir or NAPALM or Netmiko?

The only answer one can give would be “it depends… on what you’re trying to do” as these three tools solve completely different challenges.

Paramiko is SSH implementation in Python. It’s used by most Python tools that want to use SSH to connect to other hosts (including networking devices).

read more see 1 comments

Worth Reading: TCP MSS Values in the Wild

In Never-Ending Story of IP Fragmentation I described how you could use TCP Maximum Segment Size to minimize the impact of IP fragmentation and PMTUD blackholes (more details on TCP MSS clamping)… but one has to wonder how people use TCP MSS in the wild and what values you might see.

As is often the case, Geoff Houston found a way to measure them, and published the answer: TCP MSS Values

see 1 comments

Beware the Marketing Magic of GUI-Based Programming

Someone working for a network automation startup desperately tried to persuade me how cool their product is. Here’s what he sent me:

We let network engineers build their own network automation solutions in no time without requiring coding or scripting knowledge. It’s all GUI based, specifically geared towards network engineers - they can simply model services or roll-out networks “as-designed”.

The only problem: I’ve seen that same argument numerous times…

read more see 1 comments

Just Published: High-Level Azure Networking Concepts

Last week we started the Microsoft Azure Networking saga that will eventually mirror the AWS Networking materials.

I recorded the hands-on demos in advance so we had plenty of time to discuss Azure API and CLI, geographies, regions and availability zones, high-availability concepts, and deployments models… and spent the second half of the live session focusing on virtual networks, subnets, interface, and IP addresses. The videos are already online and accessible with Standard ipSpace.net Subscription.

Next step (on September 24th): network security and user-defined routes.

Add comment

If You Travel to Slovenia, You SHOULD NOT Fly with Adria Airways

I apologize to my regular readers for a completely off-topic post, but if I manage to save a single traveller the frustrations I experienced a few weeks ago it was well worth it. Also, please help spread the word…

TL&DR: If you travel to Slovenia, DO NOT even consider flying with Adria Airways (and carefully check the code-share flights, they might be hiding under a Lufthansa or Swiss flight number). Their actual flight schedule is resembling a lottery, and while I always had great experience with the friendly, courteous and highly professional cabin crews, it’s totally impossible to reach their customer service.

2019-09-30: The agony ended sooner than I expected. On September 30th Adria Airways declared bankruptcy, ending the frustration and uncertainty of thousands of passengers they left stranded across Europe for almost 10 days. So long Adria, and thanks for all the good flights (we'll eventually forget all the mess you made in the last year)

2019-09-22: Added updates on what happened during last week. The whole thing is becoming a soap opera

read more see 6 comments

Video: Beyond Two Nodes

In the introductory videos of How Networks Really Work webinar I described the mandatory elements of any networking solution and additional challenges you have to solve when you can’t pull a cable between the adjacent nodes.

It’s time for the next bit of complexity: what if we have more than two nodes connected to the same network segment? Welcome to the world of multi-access networks and data link control.

You need free ipSpace.net subscription to watch the videos in Overview of Networking Challenges section, or a paid ipSpace.net subscriptions to watch the rest of the webinar.

Add comment

Disaster Recovery Test Faking: Another Use Case for Stretched VLANs

The March 2019 Packet Pushers Virtual Design Clinic had to deal with an interesting question:

Our server team is nervous about full-scale DR testing. So they have asked us to stretch L2 between sites. Is this a good idea?

The design clinic participants were a bit more diplomatic (watch the video) than my TL&DR answer which would be: **** NO!

Let’s step back and try to understand what’s really going on:

read more see 5 comments

Response: The OSI Model Is a Lie

Every now and then I stumble upon a blog post saying “OSI 7-layer model sucks” or “OSI 7-layer model is a lie”, most recent one coming from Robert Graham.

Before going into the details, let’s agree on the fundamentals.

Most everyone who ever tried to build a network spanning more than one transmission technology and including intermediate nodes came to the conclusion that layered approach to networking makes sense.

Whether you have three, four, five, or seven layers in your model doesn’t matter. What really matters is that your model contains all the functionality you need to implement host-to-host networking in target environment.

read more see 7 comments

Supply-Chain Security in Open-Source Software

Last week we started the Autumn 2019 Building Network Automation Solutions online course with an interesting presentation from Matthias Luft focused on open-source supply chain security

TL&DR: Can I download whatever stuff I found as my first Google hit and use it in my automation solution? ****, NO!

Matthias covered these topics:

read more Add comment

Intent-Based Networking with Batfish on Software Gone Wild

Imagine you would have a system that would read network device configurations, figure out how those devices might be connected, reverse-engineer the network topology, and be able to answer questions like “what would happen if this link fails” or “do I have fully-redundant network” or even “how will this configuration change impact my network”. Welcome to Batfish.

Interested? You’ll find more in Episode 104 of Software Gone Wild.

see 1 comments

Measure Twice, Cut Once: Ansible net_interface

As I was preparing the materials for Ansible 2.7 Update webinar sessions I wanted to dive deeper into declarative configuration modules, starting with “I wonder what’s going on behind the scenes

No problem: configure EEM applet command logging on Cisco IOS and execute an ios_interface module (more about that in another blog post)

Next step: let’s see how multi-platform modules work. Ansible has net_interface module that’s supposed to be used to configure interfaces on many different platforms significantly simplifying Ansible playbooks.

read more see 6 comments

If You Have to Simulate Your Whole Network, You're Doing It Wrong

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

Have you ever seen a presentation in which a startup is telling you how awesome their product is because it allows you to simulate your whole network in a virtual environment? Not only that, you can use that capability to build a test suite and a full-blown CI/CD pipeline and test whether your network works every time you make a change to any one box in the network.

Sounds awesome, right? It’s also dead wrong. Let me explain why that’s the case.

read more Add comment

Just Published: NSX-T Technical Deep Dive Slide Deck

Last year when I was creating the first version of VMware NSX Deep Dive content, NSX-V was mainstream and NSX-T was the new kid on the block. A year later NSX-V is mostly sidelined, and all the development efforts are going into NSX-T. Time to adapt the webinar to new reality… taking the usual staged approach:

Add comment

Video: Introducing Transmission Technologies

After discussing the challenges one encounters even in the simplest networking scenario connecting two computers with a cable we took a short diversion into an interesting complication: what if the two computers are far apart and we can’t pull a cable between them?

Trying to answer that question we entered the wondrous world of transmission technologies. It’s a topic one can spent a whole life exploring and mastering, so we were not able to do more than cover the fundamentals of modulations and multiplexing technologies.

You need free ipSpace.net subscription to watch the video, or a paid ipSpace.net subscriptions to watch the rest of the webinar.

Add comment

Upcoming Events and Webinars (September 2019)

We’re back from the summer break for real - the first autumn 2019 ipSpace.net event takes place today: I’ll talk about the fallacies of distributed computing.

September will be an intensive month:

Of course, we’ll keep going… our event calendar is fully packed till mid-November. More about that in a month.

Add comment

Updated: Never-Ending Story of IP Fragmentation

In mid 2000s I wrote a number of articles describing various TCP/IP features. Most of them are a bit outdated, so I decided to clean up, update and repost the most interesting ones on ipSpace.net, starting with Never-Ending Story of IP Fragmentation.

The first part of that article is already online, covering MTU basics and drawbacks of IP fragmentation.

see 1 comments

Recently Published: Azure Networking Demo Videos

Remember my rant about the glacial speed of Azure orchestration system? I decided I won’t allow it to derail yet another event and recorded the demos in advance of the first live session. The final videos are just over an hour long; it probably took me at least three hours to record them.

If you plan to attend the live webinar session on September 12th, you might want to watch at least the first few videos before the live session - I will not waste everyone’s time repeating the demos during the live session.

Add comment

Video: Networking Challenges

Whenever you’re discussing a complex topic it’s worth adhering to two principles: (A) identify the challenges you’re trying to solve and (B) start as simple as you can and add complexity later.

We did exactly that in the Introducing Networking Challenges part of How Networks Really Work webinar. We started with the simplest possible case of two computers connected with a cable… and even there identified a plethora of challenges that had to be solved more than half a century ago (and still have to be solved today no matter what magic software-defined technology someone pulls out of their wizard hat).

You need free ipSpace.net subscription to watch the video, or a paid ipSpace.net subscriptions to watch the rest of the webinar.

Add comment

Must Read: When Redundancy Actually Helps

Stumbled upon an excellent redundancy-focused blog post (HT: High Scalability). Here are just a few important points:

  • Don’t make things too complex;
  • Don’t add more risk than you take away;
  • You’ve got to fail over in the right direction;
  • You must be able to return to fully-redundant mode.

I’m guessing that people promoting stretched VLANs, vSphere and/or NSX clusters running across multiple sites, weird combination of EVPN and OTV, and a dozen similar shenanigans never considered any one of these points.

see 2 comments

Brief History of VMware NSX

I spent a lot of time during this summer figuring out the details of NSX-T, resulting in significantly updated and expanded VMware NSX Technical Deep Dive material… but before going into those details let’s do a brief walk down the memory lane ;)

We’re running an NSX Deep Dive workshop in Zurich in early September, followed by NSX-T update webinar in mid-November.

You might remember a startup called Nicira that was acquired by VMware in mid-2012… supposedly resulting in the ever-continuing spat between Cisco and VMware (and maybe even triggering the creation of Cisco ACI).

read more see 10 comments

The First Networking Fundamentals Videos are Online

In mid-June I started another pet project - a series of webinars focused on networking fundamentals. In the first live session on June 18th we focused on identifying the challenges one has to solve when building an end-to-end networking solution, and the role of layered approach to networking.

Not surprisingly, we quickly went down the rabbit holes of computer networking history, including SCSI cables, serial connections and modems… but that’s where it all started, and some of the concepts developed at that time are still used today… oftentimes heavily morphed by recursive application of RFC 1925 Rule 11.

read more Add comment

Migrating ipSpace.net Infrastructure to AWS

I’m too stupid to unwind and relax over summer - there’s always some janitorial task to be done, and I simply cannot leave it alone. This summer, I decided to migrate our server infrastructure to AWS.

TL&DR: It went smoother than I expected, and figuring out how AWS virtual networks, public IP addresses, and security groups work while creating AWS Networking webinar definitely helped, but it also took way longer than I expected.

read more see 8 comments

Reinventing Your Own STP Wheel...

One of my readers sent me a link to an interesting L2-over-IP "design". Someone tried to connect two data centers with redundant etherip links using home-brewed redundancy mechanism and (surprise, surprise) managed to bring both of them down. The obvious fix: patch the etherip device driver.

EtherIP is pre-VXLAN Ethernet-over-IP technology yet again proving RFC1925 Rule 11.

I don't know enough about OpenBSD to figure out whether (A) it doesn't have STP at all, (B) STP doesn't work over EtherIP, (C) host routing based on ARP entries would be too much of a hassle, (D) some people don't understand the networking fundamentals, (E) everything looks like a nail once you found a hammer, or (F) all of the above. Insightful comments would be highly appreciated.

see 8 comments

Why You Can't Fix a System from the Inside

Stumbled upon an interesting article describing numerous examples of how it's impossible to fix a system from the inside because the good guys always lose to the more aggressive (and less scrupulous) individuals.

It's amazing how well the same ideas apply to TCP-versus-UDP, P2P traffic versus everything else (this one has been fixed after a lot of pressure from the outside), latency- versus drop-based TCP congestion management and $vendor marketing.

Add comment

Software Engineers and Network Automation

I was saying “you’ll get the best network automation (or SDN) results if you pair network engineers with software engineers” for ages, but there’s always someone else saying it more eloquently, in this case Jeremy Schulman in his recent blog post.

Jeremy will talk about ChatOps in Autumn 2019 Building Network Automation Solutions online course, but of course you’re more than welcome to ask him other questions as well.

Add comment

Rant: Some Internet Service Providers Should Really Know Better...

I was listening to a nice podcast with Nick Buraglio discussing the recent BGP hijack SNAFU impacting Cloudflare (and their reaction) and while I usually totally agree with Nick, I think that he tried to be way too nice when saying (paraphrasing) “I think Cloudflare was a bit harsh - I would prefer a more community-oriented approach along the lines of how could we help you do your job better

read more see 1 comments

We Are on a Break ;)

It’s high time for another summer break (I get closer and closer to burnout every year - either I’m working too hard or I’m getting older ;).

Of course we’ll do our best to reply to support (and sales ;) requests, but it might take us a bit longer than usual. I will publish an occasional worth reading or watch out blog post, but don’t expect anything deeply technical for the new two months.

We’ll be back (hopefully refreshed and with tons of new content) in early September, starting with network automation course on September 3rd and VMware NSX workshop on September 10th.

In the meantime, try to get away from work (hint: automating stuff sometimes helps ;), turn off the Internet, and enjoy a few days in your favorite spot with your loved ones!

see 1 comments

First-hand Feedback: ipSpace.net Network Automation Course

Daniel Teycheney attended the Spring 2019 Building Network Automation Solutions online course and sent me this feedback after completing it (and creating some interesting real-life solutions on the way):


I spent a bit of time the other day reflecting on how much I’ve learn’t from the course in terms of technical skills and the amount I’ve learned has been great. I literally no idea about things like Git, Jinja2, CI testing, reading YAML files and had only briefly seen Ansible before.

I’m not an expert now, but I understand these things and have real practical experience on these subjects which has given me great confidence to push on and keep getting better.

read more Add comment

Device Configuration Synthesis with NetComplete on Software Gone Wild

When I was still at university the fourth-generation programming languages were all the hype, prompting us to make jokes along the lines “fifth generation will implement do what I don’t know how

The research team working in Networked Systems Group at ETH Zurich headed by prof. Laurent Vanbever got pretty close. The description of their tool says:

read more see 4 comments

Impact of Controller Failures in Software-Defined Networks

Christoph Jaggi sent me this observation during one of our SD-WAN discussions:

The centralized controller is another shortcoming of SD-WAN that hasn’t been really addressed yet. In a global WAN it can and does happen that a region might be cut off due to a cut cable or an attack. Without connection to the central SD-WAN controller the part that is cut off cannot even communicate within itself as there is no control plane…

A controller (or management/provisioning) system is obviously the central point of failure in any network, but we have to go beyond that and ask a simple question: “What happens when the controller cluster fails and/or when nodes lose connectivity to the controller?”

read more see 4 comments

Real-Life SD-WAN Experience

SD-WAN is the best thing that could have happened to networking according to some industry “thought leaders” and $vendor marketers… but it seems there might be a tiny little gap between their rosy picture and reality.

This is what I got from someone blessed with hands-on SD-WAN experience:

read more see 8 comments

Read Network Device Information with REST API and Store It Into a Database

One of my readers sent me this question:

How can I learn more about reading REST API information from network devices and storing the data into tables?

Long story short: it’s like learning how to drive (well) - you have to master multiple seemingly-unrelated tasks to get the job done.

read more see 2 comments

How Microsoft Azure Orchestration System Crashed My Demos

One of the first things I realized when I started my Azure journey was that the Azure orchestration system is incredibly slow. For example, it takes almost 40 seconds to display six routes from per-VNIC routing table. Imagine trying to troubleshoot a problem and having to cope with 30-second delay on every single SHOW command. Cisco IGS/R was faster than that.

If you’re old enough you might remember working with VT100 terminals (or an equivalent) connected to 300 baud modems… where typing too fast risked getting the output out-of-sync resulting in painful screen repaints (here’s an exercise for the youngsters: how long does it take to redraw an 80x24 character screen over a 300 bps connection?). That’s exactly how I felt using Azure CLI - the slow responses I was getting were severely hampering my productivity.

read more see 3 comments

Feedback: Ansible for Networking Engineers

I always love to hear from networking engineers who managed to start their network automation journey. Here’s what one of them wrote after watching Ansible for Networking Engineers webinar (part of paid ipSpace.net subscription, also available as an online course).

This webinar helped me a lot in understanding Ansible and the benefits we can gain. It is a big area to grasp for a non-coder and this webinar was exactly what I needed to get started (in a lab), including a lot of tips and tricks and how to think. It was more fun than I expected so started with Python just to get a better grasp of programing and Jinja.

In early 2019 we made the webinar even better with a series of live sessions covering new features added to recent Ansible releases, from core features (loops) to networking plugins and new declarative intent modules.

Add comment

Running OSPF in a Single Non-Backbone Area

One of my subscribers sent me an interesting puzzle:

> One of my colleagues configured a single-area OSPF process in a customer VRF customer, but instead of using area 0, he used area 123 nssa. Obviously it works, but I was thinking: “What the heck, a single OSPF area MUST be in Area 0

Not really. OSPF behaves identically within an area (modulo stub/NSSA behavior) regardless of the area number…

read more see 5 comments

Switch Buffer Sizes and Fermi Estimates

In my quest to understand how much buffer space we really need in high-speed switches I encountered an interesting phenomenon: we no longer have the gut feeling of what makes sense, sometimes going as far as assuming that 16 MB (or 32MB) of buffer space per 10GE/25GE data center ToR switch is another $vendor shenanigan focused on cutting cost. Time for another set of Fermi estimates.

Let’s take a recent data center switch using Trident II+ chipset and having 16 MB of buffer space (source: awesome packet buffers page by Jim Warner). Most of switches using this chipset have 48 10GE ports and 4-6 uplinks (40GE or 100GE).

read more see 8 comments

Use Per-Link Prefixes in Network Data Models

We got pretty far in our data deduplication in network data model journey, from initial attempts to network modeled as a graph… but we still haven’t got rid of all the duplicate information.

For example, if we have multiple devices connected to the same subnet, why should we have to specify IP address and subnet mask for every device (literally begging the operators to make input errors). Wouldn’t it be better (assuming we don’t care about exact IP addresses on core links) to assign IP addresses automatically?

see 5 comments

Repost: Automation Without Simplification

The No Scripting Required to Start Your Automation Journey blog post generated lively discussions (and a bit of trolling from the anonymous peanut gallery). One of the threads focused on “how does automation work in real life IT department where it might be challenging to simplify operations before automating them due to many exceptions, legacy support…

Here’s a great answer provided by another reader:

read more Add comment

As Expected: Where Have All the SDN Controllers Gone?

Roy Chua (SDx Central) published a blog post titled “Where Have All the SDN Controllers Gone” a while ago describing the gradual disappearance of SDN controller hype.

No surprise there - some of us were pointing out the gap between marketing and reality years ago.

It was evident to anyone familiar with how networking actually works that in a generic environment the drawbacks of orthodox centralized control plane SDN approach far outweigh its benefits. There are special use cases like intelligent patch panels where a centralized control plane makes sense.

read more see 1 comments

Stop Using GUI to Configure SDN or Intent-Based Products

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

At the end of my vNIC 2018 keynote speech I made a statement along these lines:

The moment you start using GUI with an SDN product you’re back to square one.

That claim confused a few people – Mark left this comment on my blog:

read more Add comment

Do Packet Drops Matter for TCP Performance?

Approximately two years ago I tried to figure out whether aggressive marketing of deep buffer data center switches makes sense, recorded a few podcasts on the topic and organized a webinar with JR Rivers.

Not surprisingly, the question keeps popping up, so it seems it’s time for another series of TL&DR articles. Let’s start with the basics:

read more see 12 comments

Generalize the Network-as-Graph Data Model

Remember the avoid duplicate data in network automation data models challenge and the restructuring we did to represent a network as a graph.

Well, I was not happy with the end result - I hated the complexity of supporting Jinja2 templates that had to check left- and right nodes of a link, so I generalized the data structure a bit, and all of a sudden I could model stub interfaces, P2P links and multi-access networks.

see 2 comments

Know Thy Environment Before Redesigning It

A while ago I had an interesting consulting engagement: a multinational organization wanted to migrate off global Carrier Ethernet VPN (with routers at the edges) to MPLS/VPN.

While that sounds like the right thing to do (after all, L3 must be better than L2, right?) in that particular case they wanted to combine the provider VPN with Internet-based IPsec VPN… and doing that in parallel with MPLS/VPN tends to become an interesting exercise in “how convoluted can I make my design before I give up and migrate to BGP”.

read more see 4 comments

MUST READ: Thou shalt not commit logical fallacies

Found a fantastic list of common logical fallacies. It's a must read for anyone having at least occasional interaction with non-Vulcans... and when you stop laughing (or screaming, or both) make sure you go through the companion web site to understand bugs in your wetware that sabotage your attempts at being perfectly logical.

Add comment

Upcoming Webinars and Events (June 2019)

I’m always amazed at how fast the time flies. I have no idea where May disappeared to, it seems like it was only yesterday when I was writing about webinar plans in 2019… and yet it’s only a month till ipSpace.net Summer Break™.

During June 2019 I’ll continue updating Designing the Private Cloud Infrastructure webinar, and start a new pet project: How Networks Really Work – I’m literally minutes away from traveling to a quiet spot in the middle of nowhere where I’ll work on the materials. In between these webinars you’ll find me in Zurich where I’ll run Microsoft Azure Networking workshop on June 12th in parallel with SIGS Technology Conference.

As you might expect we have plenty of things already lined up for autumn 2019… more about that in a week or two.

see 1 comments

Remember: Don’t Panic

I hate listening to “this is what we were doing this year” podcasts as they usually turn into pointless blabbering, self-congratulations and meaningless plans (think New Year resolutions). The Full Stack Journey Episode 28 with Scott Lowe was an amazing deviation from this too-common template.

If you don’t have time to listen to the podcast (but you OUGHT TO do it) here’s what I loved most: “When faced with the onslaught of new technologies, don’t panic. Wait a few months to see which ones survive”.

read more Add comment

IPv6 Support in Microsoft Azure

TL&DR: MIA

Six years ago, when I was talking about overlay virtual networks at Interop, I loved to joke that we must be living on a weird planet where Microsoft has the best overlay virtual networking implementation… at least as far as IPv6 goes.

Even then, their data plane implementation which was fully dual-stack-aware on both tenant- and underlay level was way ahead of what System Center could do.

read more see 4 comments

Model Your Network as a Graph not a Set of Boxes

Last week I explained how you could take a typical first attempt at a network automation data model and reduce the amount of duplicate data… but the data model we used was still describing a set of seemingly disconnected boxes.

How about restructuring the whole thing and describing what networks really are - graphs made of nodes (network devices) and links?

see 2 comments

It's Time for Another Pet Project

More than a decade ago I decided to start a pet project: a blog describing interesting details of networking technologies. The idea quickly morphed into vendor-neutral webinars - the first one took place in February 2010. A year or two later I had my first guest speaker and as of today we had more than 50 industry experts participating in ipSpace.net webinars and online courses.

In the meantime the ipSpace.net team grew: I had video and audio editors for years, Irena Marčetič took over marketing, logistics, and production in 2018, and we got a team of webinar moderators that will help us with guest speaker webinars (last week we ran the first guest speaker webinar where I didn’t have to be involved - hooray ;)

read more see 1 comments

If You Worry About 768K Day, You’re Probably Doing Something Wrong

A few years ago we “celebrated” 512K day - the size of the full Internet routing table exceeded 512K (for whatever value of K ;) prefixes, overflowing TCAMs in some IP routers and resulting in interesting brownouts.

We’re close to exceeding 768K mark and the beware 768K day blog posts have already started appearing. While you (RFC 2119) SHOULD check the size of your forwarding table and the maximum capabilities of your hardware, the more important question should be “Why do I need 768K forwarding entries if I’m not a Tier-1 provider

read more see 3 comments

How Hard Is It to Manage Your Intent?

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

Remember the “every device configuration is really an expression of our intent” discussion? Forgetting the wrong level of abstraction (we mostly don’t want to deal with all the idiosyncratic stuff network devices want to see in their configurations) and box-oriented thinking caused by device-level intent for the moment, let’s focus on another aspect: how hard is it to manage your intent?

read more Add comment

Don't Base Your Design on Vendor Marketing

Remember how Arista promoted VXLAN coupled with deep buffer switches as the perfect DCI solution a few years ago? Someone took Arista’s marketing too literally, ran with the idea and combined VXLAN-based DCI with traditional MLAG+STP data center fabric.

While I love that they wrote a blog post documenting their experience (if only more people would do that), it doesn’t change the fact that the design contains the worst of both worlds.

Here are just a few things that went wrong:

read more see 9 comments

Data Deduplication in Network Automation Data Models

One of the toughest challenges in the hands-on part of Building Network Automation Solutions online course is the create a data model describing your service exercise.

Networking engineers never had to think about data models describing their networks or services, and the first attempt often results in something that looks like simplified device configuration in YAML or JSON format.

I wrote a long article describing how you can slowly redesign your box-focused data model into a network-focused one. The first parts describing the problem and initial deduplication are already online.

see 3 comments

Microsoft Azure Networking Slide Deck Is Ready

After a few weeks of venting my frustrations on Twitter I finally completed Microsoft Azure Networking slide deck last week and published the related demos on GitHub.

I will use the slide deck in a day-long workshop in Zurich (Switzerland) on June 12th and run a series of live webinar sessions in autumn. If you’re a (paid) subscriber you can already download the slides and it would be great if you’d have time to attend the Zurich workshop – it’s infinitely better to discuss interesting challenges face-to-face than to type questions in a virtual classroom.

see 1 comments

Programmable Packet Forwarding Pipelines Using P4 on Software Gone Wild

Every time a new simple programming language is invented, we go through the same predictable cycle:

  • Tons of hype;
  • Unbounded enthusiasm when people who never worked in target environment realize they could get something simple done in a short time;
  • Ever-worsening headaches as the enthusiasts try to get a real job done with the shiny new tool;
  • Disappointment;
  • A more powerful language is invented to replace the old one.

A few years ago we experienced the same cycle when OpenFlow was the-one-tool-to-bind-them all.

read more Add comment

Stop the Low-Level Configuration Manipulation

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

Imagine a small bank deciding in their infinite wisdom (in reality: because their CIO attended a conference organized by a database vendor) to implement their banking software by teaching bank tellers how to type SQL transactions by hand.

For example, to transfer money from one account to another account, a bank teller could simply type:

read more Add comment

Building Fabric Infrastructure for an OpenStack Private Cloud

An attendee in my Building Next-Generation Data Center online course was asked to deploy numerous relatively small OpenStack cloud instances and wanted select the optimum virtual networking technology. Not surprisingly, every $vendor had just the right answer, including Arista:

We’re considering moving from hypervisor-based overlays to ToR-based overlays using Arista’s CVX for approximately 2000 VLANs.

As I explained in Overlay Virtual Networking, Networking in Private and Public Clouds and Designing Private Cloud Infrastructure (plus several presentations) you have three options to implement virtual networking in private clouds:

read more see 1 comments

Automating Brownfield Environments (Using an 802.1x Example)

This is a guest blog post by Albert Siersema, senior network and cloud engineer at Mediacaster.nl. He’s always busy broadening his horizons and helping his customers in (re)designing and automating their infrastructure deployment and management.


This is the second post in a series focused primarily on brownfield automation principles using 802.1x deployments as an example (you might want to read part 1 first).

Before diving into the specifics of the next 802.1x automation phase, let’s take a step back and think about why we’re going through this effort. Automation is a wonderful tool, but it’s not a goal… and neither is 802.1x a goal - it’s just another tool that can help us realize business benefits like:

read more Add comment

Worth Reading: Nothing Fails Like Success

I hope I'm still allowed to quote a paragraph from someone else's article (thank you, EU, you did a great job). Here's what Jeffrey Zeldman wrote about startup business models:

A family buys a house they can’t afford. They can’t make their monthly mortgage payments, so they borrow money from the Mob. Now they’re in debt to the bank and the Mob, live in fear of losing their home, and must do whatever their creditors tell them to do.

Read the article and think about how it applies to unicorn-based networking technologies ;)

Add comment

Feedback: Data Center Interconnects

Got this feedback from a networking engineer watching the Data Center Interconnects webinar:

This webinar is an excellent overview regarding current DCI design challenges. I would highly recommend to watch it for anyone working in the networking and datacenter space. Sober networkers should watch it thoughtfully at least two times. L2 DCI fans should watch it once in a month, until reaching a solid grasp.

If only life would be as easy as that ;) Most people prefer to be blissfully ignorant of the infrastructure supporting their business, while at the same time pretending they know an awful lot about other people's jobs (see also: Dunning-Kruger effect)

Add comment

Automation Should Prevent Operator Errors

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

One of the toughest tasks faced by networking engineers attending our Building Network Automation Solutions course is designing a data model describing network infrastructure or services. They usually think in terms of individual devices (nodes) resulting in tons of duplicated data.

I always point that out when reviewing their solutions and suggest how to minimize or eliminate duplicate data. Not surprisingly, doing that is hard, and one of the attendees started wondering whether the extra effort makes sense:

read more Add comment

Real-Life Data Center Meltdown

A good friend of mine who prefers to stay A. Nonymous for obvious reasons sent me his “how I lost my data center to a broadcast storm” story. Enjoy!


Small-ish data center with several hundred racks. Row of racks supported by an end-of-row stack. Each stack with 2 x L2 EtherChannels, one EC to each of 2 core switches. The inter-switch link details don’t matter other than to highlight “sprawling L2 domains."

VLAN pruning was used to limit L2 scope, but a few VLANs went everywhere, including the management VLAN.

read more see 3 comments

Building Automation Device Inventory with Open Source Tools

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

One of the common questions we get in the Building Network Automation Solutions online course is “how do I create device inventory if I don’t know (exactly) what devices are in my network?”… prompting one of the guest speakers to reply “could it really be that bad?” (yes, sometimes it is).

Some of the students tried to solve the challenge with Ansible. While that might eventually work (given enough effort), Ansible definitely isn’t the right tool for the job.

What you need to get the job done is a proper toolchain:

read more see 2 comments

Now Boarding: Autumn 2019 Network Automation Online Course

Ladies and gentlemen, our Autumn 2019 Building Network Automation Solutions online course is now ready for boarding. Please make sure you have your boarding passes ready, board at your convenience, and start enjoying the pre-flight perks like over hundred hours of self-study materials.

Our flight will depart on September 3rd with subsequent sessions on September 26th, October 24th and November 12th. The guest speakers will focus on security, inventory managements, and describe their production deployments. More in a few days…

The only thing you have to do at this moment is to register (if you want to get the Enthusiast price… otherwise please feel free to wait ;)

And just in case you’re wondering: yes, I was sitting at an airport while writing this blog post ;))

Add comment

Automation Solution: Find Source of STP Topology Changes

Topology changes are a bane of large STP-based networks, and when they become a serious challenge you could probably use a tool that could track down what’s causing them.

I’m sure there’s a network management tool out there that can do just that (please write a comment if you know one); Eder Gernot decided to write his own while working on a hands-on assignment in the Building Network Automation Solutions online course. Like most course attendees he published the code on GitHub and might appreciate pull requests ;)

Wonder what else course attendees created in the past? Here’s a small sample.

see 2 comments

Commentary: We’re stuck with 40 years old technology

One of my readers sent me this email after reading my Loop Avoidance in VXLAN Networks blog post:

Not much has changed really! It’s still a flood/learn bridged network, at least in parts. We count 2019 and talk a lot about “fabrics” but have 1980’s networks still.

The networking fundamentals haven’t changed in the last 40 years. We still use IP (sometimes with larger addresses and augmentations that make it harder to use and more vulnerable), stream-based transport protocol on top of that, leak addresses up and down the protocol stack, and rely on technology that was designed to run on 500 meters of thick yellow cable.

read more see 11 comments

Must Watch: History of Cisco IOS CLI

My first Cisco router was a blade for a Cabletron modular hub (anyone remembers what hubs were or a company named Cabletron?). We plugged it in, I read the documentation, figured out I had to type conf t and was faced with a blinking cursor staring back at me from an empty line.

A few years later I was invited to beta test Cisco software release 9.21 (it wasn’t called IOS yet). The best feature it had was the awesome configuration CLI with context-sensitive prompts and on-demand help.

read more see 2 comments

Automation Solution: Create Switch Stack Reports

Have you ever wondered how many free ports you have on your stackable campus switches? I’m sure there must be a wonderful network management tool that creates that reports with a click of a button… but what if the tool your PHB purchased based on awesome PowerPoint and glitzy demo can’t do that?

Nadeem Lughmani decided to solve this challenge as a hands-on assignment in the Building Network Automation Solutions online course and created an Ansible playbook and a Python plugin that counts the total number of ports and number of free ports for each switch stack specified in the device inventory.

Wonder what else course attendees created in the past? Here’s a small sample.

see 3 comments

How Common Are Data Center Meltdowns?

We all know about catastrophic headline-generating failures like AWS East-1 region falling apart or a major provider being down for a day or two. Then there are failures known only to those who care, like losing a major exchange point. However, I’m becoming more and more certain that the known failures are not even the tip of the iceberg - they seem to be the climber at the iceberg summit.

read more see 10 comments

Text Files or Relational Database?

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

One of the common questions I get once the networking engineers progress from Ansible 101 to large-scale deployments (example: generating configurations for 1000 devices) is “Can Ansible use a relational database? Text files don’t scale…”

TL&DR answer: Not directly, but there are tons of database Ansible plugins or custom Jinja2 filters out there.

read more see 3 comments

Using Faucet to Build SC18 Network with OpenFlow

Remember how Nick Buraglio tried to use OpenDaylight to build a small part of SuperComputing conference network… and ended up with a programmable patch panel?

This time he repeated the experiment using Faucet SDN Controller – an OpenFlow controller focused on getting the job done – and described his experience in Episode 101 of Software Gone Wild.

We started with the usual “what problem were you trying to solve” and quickly started teasing apart the architecture and got geekily focused on interesting things like:

read more see 2 comments

Making Cisco ACI REST API Transactional

This is a guest blog post by Dave Crown, Lead Data Center Engineer at the State of Delaware. He can be found automating things when he's not in meetings or fighting technical debt.


In a recent blog post, Ivan postulated “You’d execute a REST API call. Any one of those calls might fail. Now what? ... You’ll have absolutely no help from the orchestration system because REST API is not transactional so there’s no rollback.” Well, that depends on the orchestration system in use.

The promise of controller-based solutions (ACI, NSX, etc.) is that your unicorn powered network controller should be an all seeing, all knowing platform managing your network. We all have hopefully learned about the importance of backups very early on our careers. Backup and, more importantly, restore should be table stakes; a fundamental feature of any network device, let alone a networking system managed by a controller imbued with magical powers (if the vendor is to be believed).

read more see 5 comments

Decide How Badly You Want to Fail

Every time I’m running a data center-related workshop I inevitably get pulled into stretched VLAN and stretched clusters discussion. While I always tell the attendees what the right way of doing this is, and explain the challenges of stretched VLANs from all perspectives (application, database, storage, routing, and broadcast domains) the sad truth is that sometimes there’s nothing you can do.

You’ll find a generic version of that explanation in Building Active-Active and Disaster Recovery Data Centers webinar. Every few months I might be available for an onsite version of that same discussion, or you could engage one of the other ExpertExpress consultants.

In those sad cases, I can give the workshop attendees only one advice: face the reality, and figure out how badly you might fail. It’s useless pretending that you won’t get into a split-brain scenario - redundant equipment just makes it less likely unless you over-complicated it in which case adding redundancy reduces availability. It’s also useless pretending you won’t be facing a forwarding loop.

read more see 2 comments

REST API Is Not Transactional

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

I was walking down the infinite hallways of Cisco Live Europe chatting with the fellow Tech Field Day Extra delegates when I probably blanked out for a minute as the weirdest of thoughts hit me: “REST API is not transactional

TL&DR: Apart from using structured data and having error codes REST API is functionally equivalent to Cisco IOS CLI from 1995

read more see 4 comments

Automating 802.1x (Part One)

This is a guest blog post by Albert Siersema, senior network and cloud engineer at Mediacaster.nl. He’s always busy broadening his horizons and helping his customers in (re)designing and automating their infrastructure deployment and management.


We’d like to be able to automate our network deployment and management from a single source of truth, but before we get there from a running (enterprise, campus!) network, we’ll have to take some small steps first.

These posts are not focused on 802.1x, but it serves as a nice use case in which I’ll show you how automation can save time and bring some consistency and uniformity to the network (device) configuration.

read more Add comment

Worth Reading: There Is No Magic

I’m not the only one telling people not to bet the farm on Santa Claus and dancing unicorns. Pete Welcher wrote a nice blog post describing the implications of laws of physics and data gravity (I described the gory details in Designing Active-Active Data Centers and AWS Networking Deep Dive webinars).

Meanwhile, Russ White reviewed an article that (without admitting it) discovered that serverless is just software running on other people’s servers.

Enjoy!

see 1 comments

Intent-Based Networking Resources

Every now and then I get a question along the lines of I’m your subscriber and would like to know more about X, so I decided to start creating technology-specific pages on www.ipSpace.net that would include links to most relevant ipSpace.net blog posts, webinars, sections in our online courses, and interesting third-party resources.

The subscriber triggering this process asked me about Intent-Based Networking, so here’s the relevant resources page.

Add comment

Improved Solution: Create Network Diagram from LLDP Data

A long while ago I published a sample Ansible/NAPALM/Jinja2 solution that would take LLDP information and turn it into a network diagram (I described its details in a short video that’s accessible to anyone attending our network automation course or having an Expert subscription).

The trickiest part of that solution was detection of bidirectional links:

read more see 1 comments

Shifting Responsibility in Network Design and Operations

When I started working with Cisco routers in late 1980s all you could get were devices with a dozen or so ports, and CPU-based forwarding (marketers would call it software defined these days). Not surprisingly, many presentations in Cisco conferences (before they were called Networkers or Cisco Live) focused on good network design and split of functionality in core, aggregation (or distribution) and access layer.

What you got following those rules were stable and predictable networks. Not everyone would listen; some customers tried to be cheap and implement too many things on the same box… with predictable results (today they would be quick to blame vendor’s poor software quality).

read more Add comment

Recovering from Network Automation Failures

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

One of my readers sent me this question:

Would you write about methods for reverting from expected new state to old state in the case automation went wrong due to (un)predictable events that left a node or network in a limbo state betwixt and between.

Like always, there’s the easy and the really hard part.

read more see 1 comments

Last Week on ipSpace.net (2019W14)

Last Thursday I started another experiment: a series of live webinar sessions focused on business aspects of networking technologies. The first session expanded on the idea of three paths of enterprise IT. It covered the commoditization of IT and networking in particular, vendor landscape, various attempts at segmenting customers, and potential long-term Enterprise IT paths. Recording is already online and currently available with standard subscription.

Although the attendance was lower than usual, attendees thoroughly enjoyed it – one of them sent me this: “the value of ipSpace.net is that you cut through the BS”. Mission accomplished ;)

Add comment

Why Is MPLS Segment Routing Better than LDP?

A while ago I made a statement along the lines of “MPLS segment routing is the best thing that happened to MPLS control plane in a decade”. Obviously some MPLS-focused engineers disagree with that and a few years ago I decided to write a lengthy blog post explaining the differences between using MPLS SR with IGP (or BGP) versus more traditional IGP+LDP approach.

Obviously, I wasn’t making any progress on that front, so the only way forward was to record a short video on the topic which didn’t work well either because the end-result was a set of three videos (available with free or paid ipSpace.net subscription).

see 4 comments

Ansible Networking: From Science Fair Project toward Mature Product

When I started working with Ansible networking modules they had a distinct science fair feel: everything was in flux, every new version of Ansible would break my playbooks, modules would disappear from one release to next, documentation was sketchy and describing the latest development code not a shipped release.

In the meantime, code, documentation, and release/deprecation management improved dramatically:

read more Add comment

Don’t Sugarcoat the Challenges You Have

Last year I got into somewhat-heated discussion with a few engineers who followed the advice to run IBGP EVPN address family on top of an EBGP underlay.

My main argument was simple: this is not how BGP was designed and how it’s commonly used, and twisting it this way requires schizophrenic BGP routing process which introduces unnecessary complexity (even though it looks simple in Junos configuration) and might confuse people who have to run the network after the brilliant designer is gone.

read more Add comment

Automatic Clean-and-Updated Firewall Ruleset

This is a guest blog post by Andrea Dainese, senior network and security architect, and author of UNetLab (now EVE-NG) and  Route Reflector Labs. These days you’ll find him busy automating Cisco ACI deployments.


Following the Ivan’s post about Firewall Ruleset Automation, I decided to take a step forward: can we always have up-to-date and clean firewall policies without stale rules?

The problem

We usually configure and manage firewalls using a process like this:

read more Add comment

Upcoming Events and Webinars

In April 2019 we’re starting a new cloud security saga with Matthias Luft. The first webinar in this series will focus on the basics, subsequent live sessions spread through the rest of 2019 will cover individual technologies.

Another series we’re starting is Business Aspects of Networking, opening on April 4th with Three Paths of Enterprise IT.

We’ll also continue the math-in-networking series, this time focused on reliability functions and advanced reliability topics.

see 3 comments

From CCNA to SDN: Interview with David Bombal

A few weeks ago, I had an interesting video chat with David Bombal in which we covered a wide variety of topics including

  • What would you do if you started networking today?
  • How do you increase the value of your knowledge?
  • Networking hasn’t changed in the last 40 years and whatever you learn about networking will still be valid 20 years from now;
  • Why should I learn and implement network automation?
  • When should I start learning about network automation?

Note: David posted the whole list of topics with timestamps in the pinned comment under the video.

Add comment

Automating NSX-T

An attendee of our Building Network Automation Solutions online course decided to automate his NSX-T environment and sent me this question:

I will be working on NSX-T quite a lot these days and I was wondering how could I automate my workflow (lab + production) to produce a certain consistency in my work.
I’ve seen that VMware relies a lot on PowerShell and I’ve haven’t invested a lot in that yet … and I would like to get more skills and become more proficient using Python right now.

Always select the most convenient tool for the job, and regardless of personal preferences PowerShell seems to be the one to use in this case.

read more see 4 comments

Stateful Firewalls: When You Get to a Fork in the Road, Take It

If you’ve been in networking long enough you’d probably noticed an interesting pattern:

  • Some topic is hotly debated;
  • No agreement is ever reached even though the issue is an important one;
  • The debate dies after participants diverge enough to stop caring about the other group.

I was reminded of this pattern when I was explaining the traffic filtering measures available in private and public clouds during the Designing Infrastructure for Private Clouds workshop.

read more see 8 comments

Automation Solution: Create Network Diagram from BGP Data in Nornir

Chris Crook decided to work on a pretty typical problem for his second hands-on assignment in the Building Network Automation Solutions online course: create a network diagram from adjacency data.

He decided to rely on BGP adjacencies (I would usually use LLDP) and added an interesting twist: instead of Ansible he used Nornir with NAPALM.

read more see 1 comments

Last Week on ipSpace.net (2019W12)

Spring started for real, so it was time for some early-spring cleaning and I managed to complete two webinars during last week:

Both webinars are part of standard ipSpace.net subscription

Add comment

Multipath TCP on Software Gone Wild

I mentioned Multipath TCP (MP-TCP) numerous times in the past but I never managed to get beyond “this is the thing that might solve some TCP multihoming challenges” We fixed this omission in Episode 100 of Software Gone Wild with Christoph Paasch (software engineer @ Apple) and Mat Martineau from Open Source Technology Center @ Intel.

read more Add comment

Creating Automation Source-of-Truth from Device Configurations

Remember the previous blog post in this sequence in which I explained the need for single source-of-truth used in your network automation solution? No? Please read it first ;)

Ready for the next step? Assuming your sole source-of-truth is the actual device configuration, is there a magic mechanism we can use to transform it into something we could use in network automation?

TL&DR: No.

read more see 1 comments

Lock-In and SD-WAN: a Match Made in Heaven

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

I made a statement along these lines in an SD-WAN blog post and related email sent to our SDN and Network Automation mailing list:

The architecture of most SD-WAN products is thus much cleaner and easier to configure than traditional hybrid networks. However, do keep in mind that most of them use proprietary protocols, resulting in a perfect lock-in.

While reading that one of my readers sent me a nice email with an interesting question:

read more see 1 comments
Sidebar