Watch Out: ISR Performance License
Bill Dagy sent me an annoying ISR gotcha. In his own words:
Since you have a large audience I thought I would throw this out here. Maybe it will help someone avoid spending 80 man hours troubleshooting network slowdowns.
Here’s the root cause of that behavior:
Cisco is now shipping routers that have some specified maximum throughput, but you have to buy a “boost license” to run them unthrottled. Maybe everyone already knew this but it sure took us by surprise.
Don’t believe it? Here’s a snapshot from Cisco 4000 Family Integrated Services Router Data Sheet:
Cisco 4000 ISR performance licenses
It’s also worth noting that the boost license throughput applies only to onboard Gigabit Ethernet interfaces, and that the ISR 4461 has two onboard 10GE ports, and a maximum throughput of over 7 Gbps (one has to wonder whether that’s with IMIX or maximum-MTU packets).
Have you encountered any other gotchas? Please write a comment!
ASR 1000 series have also the same performance limitation schema, you have to purchase the throughput level license.
https://www.cisco.com/c/en_in/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731632.html
Also note that performance is the aggregatted inbound traffic of all interfaces, so for a bidirectional 1 Gbps traffic performance you need at least a 2Gbps license.
I think this article is missing context. Without the boost license, the scheduler has the capability to guarantee to a high degree proper performance to all tasks (packet forwarding, QOS, Filtering, etc). When you enable the boost license, the scheduler will schedule all cores to 100% with many run-to-complete tasks. So that means that the predictability of your performance goes out of the window.
So its (as always) a trade-off.
@Rens: While you're technically correct in your description of the impact of CPU overload, that's how routers behaved since 1980s and charging more (boost license) to get worse overall experience makes absolutely no sense.
Also, one would expect a router to generate an error message saying "I'm overloaded, please do something" instead of "pay more to squeeze the last bits out of the CPU".
If you’re an MSP that is offering a guaranteed service, you benefit for from predictable performance than squeezing every last drop out of the CPU.
They’re the target audience for these devices.
@Rens: In the "guaranteed service" case, it would make perfect sense to rate-limit the total throughput and make that a configurable option. I would love to see a vendor doing that for the quality-of-experience reasons.
What we're discussing is a clearly-less-than-optimal marketing decision that has nothing to do with technology (see other comments, there's nobody else saying "it makes sense"), but I guess one's perception of it depends on which side of the fence one's sitting.
That's true, but the point here it's nobody would expect a performance limit so severe and obscure, some of them have a price tag over 1k and do not offer enough performance to even use a 100mbps link
Boost license it's also quite expensive, as much as the router itself.
These routers are current substitutes for 1900/2900/3900 but do not perform even as much as an 88x for simple use cases.
Cisco added software based throughput rate limit licensing when the ISR 4Ks came out in 2014. This went over like a lead balloon with customers, so they removed the throughput licensing for the newer Catalyst 8200/8300/8500 series routers intended to replace the ISR 4Ks which came out in 2020. This is only for the Catalyst 8000s running "autonomous mode" (i.e. traditional standalone IOS-XE), throughput licensing still exists if you want to run them in SD-WAN mode.
There is technical document about boost in ISR 4k 'Performance License on Cisco ISR4000' https://www.cisco.com/c/en/us/support/docs/routers/4000-series-integrated-services-routers/217135-performance-license-on-cisco-isr4000.html
This is my biggest frustration with Cisco. Their engineering and product teams build great tech and then someone on the business side always ruins it with licensing. They charge what they can get away with and seem like they are on a never ending quest to squeeze out more from their current customer base. I love Cisco so much but hope they stop the madness. They seem to overcharge until competition starts to pull business away, and then they just lower their prices once they risk losing marketshare and it's "forgive and forget" about all the time they took advantage of their dominant position. Hope they change this.
Of course I could go on and on on QA that feels like beta testing in prod and poor user experience of their user interfaces but we'll save that for a second "I love you but need you to change" rant about Cisco :)
We ran into this in a recent deployment when IPSec traffic hit the ceiling. Figuring out the issue, then purchasing the "fix" (which required an IOS update) is not great. And even the process of adding the license to the device is off-putting.
It's just not worth the extra cost or the bruises. Between performance licensing and the feature licensing (Sec vs AXV vs V vs AX vs VSec), I am looking and using other things like Vyos and pfSense/TNSR.
Hi @Rik, we're about to go through the same here with a 4351 and a 4431, except we haven't yet bought the licenses. Would you be able to share which routers, licenses and IPSec throughput limit you saw?
The below was my experience with the 4000series back in 2015 when we were deploying them for IPSec tunnels on 100mbps full duplex circuits;
Throughput figures provided are total throughput so 100mbps performance on the platform would have only been sufficient for 50mbps full duplex circuit and as such insufficient so we had to make sure every router was capable of 200mbps for full duplex, and then we ran IPsec over the tunnels expecting uncapped throughput.
This was probably my fault at the time for not reading the documentation in full depth but without the additional Hseck license I was limited to 85mbps! I do believe this was patched out in later revisions of IOS, but at the time it felt like Cisco were being underhanded with this and weren't upfront about this initially.
Cisco licensing used to be relatively easy to understand but in recent years it feels like its just disjointed and confusing.
Yes the licensing has become ridiculously complex. I’ve worked for many Cisco partners over the past twenty five years, and it was only in the last few that a three day licensing class or per product day long how to sell class became mandatory. This is not why I did a CCIE.
The biggest “gotcha” that’s not immediately apparent is that the limit is “one way”.
So say you have a 5G licence for an ASR. That means you can push 2.5 Gig through the box after which it’ll start dropping packets. Cos like “2.5 in” + “2.5 out” = 5. Rather than 2.5 forwarded = 2.5.
I’m glad I’m not dealing with Cisco any more, or at least that line of products.
That's the sleaziest example of marketing math I've seen so far. To make it worse, the marketing materials use the word "throughput" not "bandwidth".
Hope it was just a SNAFU and got fixed in the meantime -- highway robbery seems fair compared to this.
I believe it's rather telling when you look at the ASR1001-X licensing docs and read "You can upgrade the throughput of the ESP from 2.5 Gbps (default) to 5 Gbps, 10 Gbps, or 20 Gbps " - the 20G value kind of gives it away.
My time in the ISP world showed licensing issues with ISR/ASR are always a theme. Someone forgets to apply the 20G perf license on the main DC WAN router or doesn't do a restart or doesn't apply the RTU license + restart during the initial installation. And then you're the lucky one to discover the main DC is running on the default 2.5 Gbps simplex license and you need to do a reboot in the quarterly downtime window on some Saturday night.
Bonus points if the end customer is the one pointing out this licensing issue and you're trying not to facedesk.
Don't even get me started on the 10G interface licenses for the same ASR1K-X.
This licensing is the price you have to pay for a non-ASIC, generic CPU router. You cannot make wonders, if you add more features at some point there will be a break down in the effective throughput. If you want to have a deterministic performance than you should not buy the boost license, but rather a bigger router.
Who cares? We have FRR and it doesn't requires anything, though feature set is much reacher. Let Cisco die silently among their corporate adepts.