This is why I don’t trust “independent experts”

The Network World recently published a story describing the results of an independent security product testing lab, where they’ve discovered (surprise, surprise) that adding security features to Cisco routers “presents a tremendous bottleneck” and “can turn a 60G router into a 5G one or even a 100M bit/sec device”.

The test results haven’t been published yet; I’ve got all the quotes from the NW story, so they might be the result of an ambitious middleware.

We don’t need “independent experts” for that. Anyone who has ever configured VPNs in a high-speed environment can tell you how to kill the performance. The basics are always the same: make sure the dedicated silicon can’t handle the job, so the packets have to be passed to the CPU. Here are a few ideas:

  • Configure GRE over IPSec and make sure you don’t tweak the MTU on the GRE tunnel. This will result in IP fragmentation and the receiving router will have to process every fragment in process switching path. A sure killer for any box, not just the 6500/7600.
  • Make sure you configure features for which you have no hardware accelerator installed in the high-end boxes and watch the performance fall (at least) 100x.
  • Even if you’ve managed to install an accelerator, configure the network in a way that effectively disables the hardware. For example, configure multiple GRE tunnels terminating on the same loopback interface
  • Design your test so that all the traffic has to pass through a bottleneck. FWSM with its 3-5GBps throughput is an ideal candidate.

What these tests prove to me is that someone who doesn’t understand what he’s doing can destroy the performance of almost any device … but we don’t need independent tests to prove that. Am I missing something? Please let me know.

10 comments:

  1. you missed to mention that not only shouldn't you trust "independent experts" neither schouldn't you trust "journalists" in general and especially not technology "journalists".

    by the way the nss press release which the networkworldstory refers to is from Nov. 2007!?

    "Using IPS in your router can turn a 60G router into a 5G one or even a 100M bit/sec device" - the only cisco routers i know about which support IOS-IPS are 800 to 73xx, none of them is not even near "2G". all other cisco IPS solutions are either appliance or module based. And if you put a 500Mbps labeled IPS module in your 60G router, not a good idea anyway.

    And by the way if you start to use gravity in your routers then all routers turn into a "1G".

    ReplyDelete
  2. You refer "[configuring] the network in a way that effectively disables the hardware." What are some non-obvious examples of this besides the one you named? as I didn't know that multiple GRE tunnels terminating at one loopback was bad design.

    ReplyDelete
  3. @Anonymous: terminating multiple GRE tunnels on one loopback interface causes problems if the ASICs cannot do a lookup on the source IP address, which is the case with some hardware.

    To figure out the exact limitations of your particular combination of hardware+software, it's best to talk to your Cisco SE or (if you're an end user) your Professional Services partner.

    @Michael: You forgot to mention that you can temporarily turn 1G gravity routers into 10G boxes if you drop them.

    ReplyDelete
  4. To Ivan Pepelnjak,

    "terminating multiple GRE tunnels on one loopback interface causes problems if the ASICs cannot do a lookup on the source IP address, which is the case with some hardware."

    Could you please give me the idea which model of cisco router got the problem about ASICs?

    Thank you very much!

    ReplyDelete
  5. The 7600 in particular will warn you if you attempt to terminate multiple tunnels on a single loopback or interface.

    It provides a warning message about traffic will now be software switched.

    ReplyDelete
  6. Ivan,
    Can you point me in the direction where you got your stats for the FWSM being a bottle neck?
    Thanks.

    ReplyDelete
  7. @anonymous: Very simple - the first bottleneck is the bus bandwidth between FWSM and the backplane. I've probably got the figures from the FWSM book published by Cisco Press (I was reading it at approximately that time).

    ReplyDelete
  8. Thanks. Are you referring to the 6-Gbps dot1q EtherChannel connection to the backplane? If so I am wondering how this is a bottleneck when the specs for code <3.2 all state 5Gbps max throughput. Starting with 4.0 code you can send some flows up to the SUP for a max throughput of 20Gbps.

    ReplyDelete
  9. The FWSM-6500/7600 EtherChannel interconnect has an obvious limitation because a single IP flow will be glued to one EC 1Gbps member due to src-dst XOR balancing, maxing out your VPN tunnel at 1 Gbps each.

    The 4.0 code can indeed alleviate that problem by pushing forwarding to the PFC, but you should only do this for stateless flows (UDP, ESP,...), as you lose all stateful checks after doing so.

    ReplyDelete
  10. The loopback / single source restriction for hardware acceleration is a restriction of the supervisor. If using the ipsec spa it works perfectly fine. Originally the documentation had you create a dummy tunnel against the loopback first in order to ensure that the rest of the tunnels would get passed on to the spa.

    That restriction is also annoying. When setting up a head-end for N remote sites who is going to create N different loopbacks, each with a unique address, simply to create the tunnel? It's not something that is expected and goes against how it would normally be deployed. A single tunnel to manage the device, fine, but not if it's going to be some sort of tunnel aggregator. And that restriction is very, very poorly documented.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.