Crappy Configuration Errors: Cisco Nexus OS Edition

TL&DR: Programmers writing Cisco Nexus OS code should have known better. They didn’t (care?).

One can understand that an operating system written in the late 1980s generates crappy suboptimal error messages, even though it was known even in those days how to do things right.

Is it too much to hope that the programmers writing newer operating systems (like Cisco Nexus OS) would be better? Obviously, it is 🤦‍♂️

I decided to implement Netmiko-based configuration for Cisco Nexus OS and tried to figure out the minimum regular expression needed to detect configuration errors. It looked like ‘anything starting with a percentage sign’ was a good starting point:

Enter configuration commands, one per line. End with CNTL/Z.
sw(config)# wrong
              ^
% Invalid command at '^' marker.

I did a few other spot checks, and the error messages all followed the same pattern. Cool, we’re done (I thought). I ran the integration tests and all IS-IS tests failed with a configuration error. Here’s the culprit (which appears only the first time you mention IS-IS on a loopback interface):

sw# conf t
Enter configuration commands, one per line. End with CNTL/Z.
sw(config)# feature isis
sw(config)# router isis Gandalf
sw(config-router)# !
sw(config-router)# interface loopback0
sw(config-if)# ip router isis Gandalf
%INFO: Command "no isis mtu-check level-2" has no effect on loopback0

OK, so I need to match a percentage sign that is not followed by INFO. Sure, I can do that. The next thing I know, @snuffy22 reported an exception. It turns out that some error messages do not start with the percentage sign:

sw(config)# interface ethernet 1/6
sw(config-if)# mtu 5000
ERROR: MTU can only be default or system jumbo MTU

The crazy part: that error message appears only if the Nexus OS port has no underlying VM NIC. You can set the MTU on “connected” interfaces, which is why the integration tests never caught this exception:

sw(config-if)# interface ethernet 1/1
sw(config-if)# mtu 5000

No big deal, I extended the regular expression to match on ’error’ regardless of where it appears, and then I stumbled upon a truly disgusting exception:

sw(config-if)# interface etherhet 0/0
                         ^
Invalid interface format at '^' marker.

The error message looks like any other “invalid something” error message, but whoever wrote it obviously forgot the percentage sign, and whatever tests Cisco runs on Nexus OS never caught this particular blunder.

Based on that, here’s the final error-detecting regular expression netlab uses when configuring Nexus OS switches with Netmiko, and I’m even more disappointed in the consistency of Cisco’s network operating systems than I was a week ago.

error_regexp: [ "^%(?!INFO).*", "(?i:invalid|error).*" ]

2 comments:

  1. That's just screen scraping. What about REST API, NETCONF and gRPC? It's year 2026.

    Replies
    1. "That's just screen scraping" << I agree, but is there a reason for screen scraping to be broken in 2026? Also: https://blog.ipspace.net/2026/03/network-device-telemetry-protocols/

  2. One of the most annoying things about working at a vendor is trying to get stuff like this fixed. The likely response from leadership is: "Why does it matter, everyone's just going to use AI to configure networks, AI will figure it out..."

    Replies
    1. Even worse, the fix brings no new money in, but might break an ancient script at a big customer that relies on the broken error message.

      As for "AI will figure things out", that makes as much sense (performance/resource-wise) as running Unix on a PDP-11 emulator written in JavaScript in your browser, but that never stopped anyone, more so as the vendor would get the hardware PO, but the AI tokens would be paid by customers 🤢

    2. The thing I am always told is: you will break the test scripts of our test engineers. You will break the test scripts of the platform testers. You will break the scripts of our customers.

      So what? If I change something, fix your scripts. No big deal. I always say: if you can't change anything, there will be zero improvement.

      The problem is: most managers say they understand that. But they rather see no change at all. They rather have me sitting on my arse, and not do anything. Than do something that might break a test script somewhere. Because doing anything like that will get you noticed. And getting noticed is what you don't want in a big company. Everyone wants to be invisible, sit still, and collect their paychecks. Sitting in meetings is way more safe than actually improving your software.

      I committed a 25K line diff 3 weeks ago. It was a cleanup of some technical debt we accrued over the last 25 years. Test managers almost got a heart attack. "It's low risk, trust me. We just need to run existing regressions, and we'll be fine". Nope, stuff needed to be postponed 4 months, so they could make a test plan. Like any tester would actually find any bug in that cleanup.

      I also made the mistake of cleaning up/changing some output formatting of some existing show commands. I like to do that. Some of our show commands look like they were done by imbeciles or toddlers. Cleaning things up, making it more readable, more consistent. I know I should not do that. The drama changing even the littlest things is unbelievable. "You added a space, and now our shitty parser broke! Revert your change!".

      I could fill this entire blog when horror stories, if Ivan let me. :)

      I remember the days when one of our developers was working all day on the new FIB-switching code (that was later renamed to CEF). Every night he uploaded a new IOS image to an ftp server. And all operators of the largest ISPs would upload that new untested image to a few of their core routers and peering routers. Those were the days. I miss those.

      (Today I got asked about a bug in software that was released 6 years ago. STFU and just upgrade).

Add comment
Sidebar