Blog Posts in October 2020
After describing Cisco SD-WAN architecture and routing capabilities, David Penaloza focused on the onboarding process and tasks performed by the Cisco SD-WAN solution (encryption, tunnel establishment, and device onboarding) in it’s so-called Orchestration Plane.
Jon Kadis spent most of his life working on enterprise networks, and sadly found out that even changing jobs and moving into a public cloud environment can’t save you from people trying to lift-and-shift enterprise IT kludges into a greenfield environment.
Here’s what he sent me:
One of my readers is designing a layer-2-only data center fabric (no SVI interfaces on switches) with stringent security requirements using Cisco Nexus switches, and he wondered whether a host connected to such a fabric could attack a switch, and whether it would be possible to reach the management network in that way.
Do you think it’s possible to reach the MANAGEMENT PLANE from the DATA PLANE? Is it valid to think that there is a potential attack vector that someone can compromise to source traffic from the front of the device (ASIC) through the PCI bus across the CPU to the across the PCI bus to the Platform Controller Hub through the I/O card to spew out the Management Port onto that out-of-band network?
My initial answer was “of course there’s always a conduit from the switching ASIC to the CPU, how would you handle STP/CDP/LLDP otherwise”. I also asked Lukas Krattiger for more details; here’s what he sent me:
I should have known better, but I got pulled into another stretched VLANs for disaster recovery tweetfest. Surprisingly, most of the tweets were along the lines of you really shouldn’t be doing that and that would never work well, but then I guess I was only exposed to a small curated bubble of common sense… until this gem appeared in my timeline:
Interestingly, that’s exactly how IP works:
In her lecture you’ll find:
- maximum branching algorithms (and I couldn’t stop wondering why we don’t use them for OSPF- or IS-IS flooding)
- path algorithms including the ones used in OSPF, IS-IS, or BGP, as well as algorithms that find K shortest paths
- center problems (for example: where do I put my streaming server or my BGP route reflector)
A long while ago I wrote a blog post along the lines of “it’s ridiculous to allow developers to deploy directly to a public cloud while burdening them with all sorts of crazy barriers when deploying to an on-premises infrastructure,” effectively arguing for self-service approach to on-premises deployments.
Not surprisingly, the reality is grimmer than I expected (I’m appalled at how optimistic my predictions are even though I always come across as a die-hard grumpy pessimist), as explained in The Shared Irresponsibility Model in the Cloud by Dan Hubbard.
Carl Montanari recently published an interesting blog post on the punditry of network APIs (including hilarious fact that “SNMP is also an API"), and as someone sent me a link to that post he commented “it reminds me of a few blog posts you wrote a while ago”.
Speaking of those blog posts… last July I was getting bored and put together a list of interesting blog posts I published on that topic. Enjoy!
In mid-September Ethan Banks invited me to chat about multi-cloud networking in the Day Two Cloud podcast. It was just a few weeks after Corey Quinn published a fantastic Multi-Cloud is the Worst Practice rant, which perfectly matched my observations, so I came well prepared ;)
When I still cared about CCIE certification, I was always tripped up by the weird scenario with (A) mismatched ARP and MAC timeouts and (B) default gateway outside of the forwarding path. When done just right you could get persistent unicast flooding, and I’ve met someone who reported average unicast flooding reaching ~1 Gbps in his data center fabric.
One would hope that we wouldn’t experience similar problems in modern leaf-and-spine fabrics, but one of my readers managed to reproduce the problem within a single subnet in FabricPath with anycast gateway on spine switches when someone misconfigured a subnet mask in one of the servers.
It took me ages to gather the willpower to tame that particular beast, but I finally got there. In the next installment of the Data Models saga I described how you can use JSON Schema to validate Ansible inventory data and your own YAML- or JSON-based data structures.
The REST API calls return text results, so you can use them straight in a Bash script. For example, here’s a simple script to print a bunch of details about your current IP address:
A few weeks ago we published an interesting discussion on network operating system details based on an excellent set of questions by James Miles.
- How hard is it to virtualize network devices?
- What is the expected performance degradation?
- Does it make sense to use containers to do that?
- What are the operational implications of running virtual network devices?
- What will be the impact on hardware vendors and networking engineers?
And of course we couldn’t avoid the famous last question: “Should network engineers program network devices?”
My friend Marjan Bradeško wrote a great article describing how we tend to forget common sense and rely too much on technology. I would strongly recommend you read it and start thinking about the choices you make when building a network with magic software-intent-defined-intelligent technology from your preferred vendor.
The designers of Cumulus Linux CLI were always focused on simplifying network device configurations. One of the first features along these lines was BGP across unnumbered interfaces, then they introduced simplified EVPN configurations, and recently auto-MLAG and auto-BGP.
You can watch a short description of these features by Dinesh Dutt and Pete Lumbis in Simplify Network Configuration with Cumulus Linux and Smart Datacenter Defaults videos (part of Cumulus Linux section of Data Center Fabrics webinar).
This blog post was initially sent to the subscribers of our SDN and Network Automation mailing list. Subscribe here.
Most automation projects are gradual improvements of existing manual processes, but every now and then the stars align and you get a perfect storm, like what Adrian Giacommetti encountered during one of his automation projects.
The customer had well-defined security policies implemented in Cisco ACI environment with tenants, endpoint groups, and contracts. They wanted to recreate those tenants in a public cloud, but it took way too long as the only migration tool they had was an engineer chasing GUI screens on both platforms.
The idea of redistributing the full Internet routing table (840.000 routes at this moment) into OSPF sound as ridiculous as it is, but when fat fingers strike it should be relatively easy to recover, right? Just disable redistribution (assuming you can still log into the offending device) and move on.
And that, my friends, is why OSPF is a single failure domain, and why you should never run OSPF between your data center fabric and servers or VM appliances.
The biggest challenge we face is variable preparation and peer review process before committing variables to Git. I’d be particularly interested on how you overcome this challenge?
We spent hours describing potential solutions in Validation, Error Handling and Unit Tests part of Building Network Automation Solutions online course, but if you never built a network automation solution using Ansible YAML files as source-of-truth the above sentence might sound a lot like Latin, so let’s make it today’s task to define the problem.
In last week’s update session we covered the new features AWS introduced since the creation of AWS Networking webinar in 2019:
- AWS Local Zones, Wavelengths, and Outposts
- VPC Sharing
- Bring Your Own Addresses
- IP Multicast support
- Managed Prefix Lists in security groups and route tables
- VPC Traffic Mirroring
- Web Application Firewall
- AWS Shield
- VPC Ingress Routing
- Inter-region VPC peering with Transit Gateways
The videos are already online; you need Standard or Expert ipSpace.net subscription to watch them.
I was telling you there’s no need to become a programmer over six years ago, but of course nobody ever listens to grumpy old engineers… which didn’t stop Ethan Banks from writing another excellent advice on the same theme: Don’t Become A Developer, But Use Their Tools.
We all knew it for a long time, now it’s finally official: IP fragmentation is broken, or as the ever-so-diplomatic IETF likes to call it, IP Fragmentation is Considered Fragile.
This podcast introduction was written by Nick Buraglio, the host of today’s podcast.
In the original days of this podcast, there were heavy, deep discussions about this new protocol called “OpenFlow”. Like many of our most creative innovations in the IT field, OpenFlow came from an academic research project that aimed to change the way that we as operators managed, configured, and even thought about networking fundamentals.
For the most part, this project did what it intended, but once the marketing machine realized the flexibility of the technology and its potential to completely change the way we think about vendors, networks, provisioning, and management of networking, they were off to the races.
We all know what happened next.
Got this question from one of my long-time readers:
I am looking for commercial SDN solutions that can be deployed on top of brownfield networks built with traditional technologies (VPC/MLAG, STP, HSRP) on lower-cost networking gear, where a single API call could create a network-wide VLAN, or apply that VLAN to a set of ports. Gluware is one product aimed at this market. Are there others?
The two other solutions that come to mind are Apstra AOS and Cisco NSO. However, you probably won’t find a simple solution that would do what you want to do without heavy customization as every network tends to be a unique snowflake.
Before we start: if you’re new to my blog (or stumbled upon this blog post by incident) you might want to read the Considerations for Host-Based Firewalls for a brief overview of the challenge, and my explanation why flow-tracking tools cannot be used to auto-generate firewall policies.
As expected, the “you cannot do it” post on LinkedIn generated numerous comments, ranging from good ideas to borderline ridiculous attempts to fix a problem that has been proven to be unfixable (see also: perpetual motion).
One of my readers sent me this question (probably after stumbling upon a remark I made in the AWS Networking webinar):
You had mentioned that AWS is probably not using EVPN for their overlay control-plane because it doesn’t work for their scale. Can you elaborate please? I’m going through an EVPN PoC and curious to learn more.
It’s safe to assume AWS uses some sort of overlay virtual networking (like every other sane large-scale cloud provider). We don’t know any details; AWS never felt the need to use conferences as recruitment drives, and what little they told us at re:Invent described the system mostly from the customer perspective.
In mid-September, Carl Buchmann, Fred Hsu, and Thomas Grimonet had an excellent presentation describing Arista’s Ansible roles and collections. They focused on two collections: CloudVision integration, and Arista Validated Designs. All the videos from that presentation are available with free ipSpace.net subscription.
Want to know even more about Ansible and network automation? Join our 2-day automation event featuring network automation experts from around the globe talking about their production-grade automation solutions or tools they created, and get immediate access to automation course materials and reviewed hands-on exercises.
While I keep telling you that Google-sized solutions aren’t necessarily the best fit for your environment, some of the hyperscaler presentations contain nuggets that apply to any environment no matter how small it is.
One of those must-watch presentations is Fault Tolerance through Optimal Workload Placement together with a wonderful TL&DR summary by the one-and-only Todd Hoff of the High Scalability fame.
Justin Pietsch is back with another must-read article, this time focused on high-speed Ethernet switching ASICs. I’ve rarely seen so many adjacent topics covered in a single easy-to-read article.
James Miles got tons of really interesting questions while watching the Network Operating System Models webinar by Dinesh Dutt, and the only reasonable thing to do when he sent them over was to schedule a Q&A session with Dinesh to discuss them.
We got together last week and planned to spend an hour or two discussing the questions, but (not exactly unexpectedly) we got only halfway through the list in the time we had, so we’re continuing next week.
Contrary to what some evangelists would love you to believe, getting fluent in network automation is a bit harder than watching 3-minute videos and cobbling playbooks together with google-and-paste… but then nothing really worth doing is ever easy, or everyone else would be doing it already.
Here’s a typical comment from a Building Network Automation Solutions attendee:
I’m loving the class. I feel more confused than I ever have in my 23 year career… but I can already see the difference in my perspective shift in all aspects of my work.