Build the Next-Generation Data Center
6 week online course starting in spring 2017

Trident 2 Chipset and Nexus 9500

Most recently launched data center switches use the Trident 2 chipset, and yet we know almost nothing about its capabilities and limitations. It might not work at linerate, it might have L3 lookup challenges when faced with L2 tunnels, there might be other unpleasant surprises… but we don’t know what they are, because you cannot get Broadcom’s documentation unless you work for a vendor who signed an NDA.

Interestingly, the best source of Trident 2 technical information I found so far happens to be the Cisco Live Nexus 9000 Series Switch Architecture presentation (BRKARC-2222). Here are a few tidbits I got from that presentation and Broadcom’s so-called datasheet.

Number of GE ports. The Trident-2 chipset supports 32 40GE ports with 128 10GE SerDes circuits, so you can split each 40GE port into four independent 10GE ports. The 10GE ports support 1GE interfaces, but it seems cannot have more than 104 GE interfaces.

Forwarding of small packets is not done at line rate (BRKARC-2222 slide 25). Vendors using Trident 2 chipset usually claim to have line rate performance. In reality, you get full line rate forwarding of small packets on 24 out of 32 40GE ports – the limiting factor is the performance of Trident 2 forwarding engine.

Some vendors (including Arista and Juniper) specify maximum throughput and packet forwarding capacity of their switches. Fair enough – do the math and decide whether the numbers are good enough for your workload. Cisco decided to use 24 out of 32 ports – that explains the “weird” number of ports (24, 36, 48 or 96 instead of 32, 64 or 128) on some switches and linecards.

Unified forwarding tables (BRKARC-2222 slide 38). Trident 2 chipset has 16K TCAM table (used for traditional longest prefix matching – LPM) and large Unified Forwarding Table (UFT) that can be used for exact matches (MAC addresses, IP host routes and ARP/ND entries, IP multicast entries) as well as prefix matches in Algorithm LPM mode.

Not surprisingly, the numbers given in the BRKARC-2222 presentation match the numbers found in Arista data sheets (32K to 288K MAC entries, 32K to 288K IPv4 host routes…) – just keep in mind that:

  • You cannot reach all maximums at the same time (you cannot have 128K IPv4 routes with 288K IPv4 ARP entries);
  • We don’t know what the granularity of UFT allocations is. You probably cannot trade one ARP entry for one MAC address;
  • You might have to specify how you want the UFT sliced up (sdm prefer anyone?)

The multistage forwarding (linecard+fabric) in Nexus 9500 uses an interesting UFT optimization (BRKARC-2222 slide 47):

  • UFTs on linecard chipsets contain MAC and ARP/ND (IP host route) information;
  • UFTs on the fabric modules contain longest-prefix match entries (IP FIB).

End result: 160K MAC entries, 88K ARP entries and 128K IP routes at the same time.

The Nexus 9500 non-hierarchical forwarding mode (BRKARC-2222 slide 49) moves the IP LPM table into linecard UFT. This forwarding mode reduces inter-subnet forwarding latency between ports on the same linecard at the expense of reduced number of forwarding entries (LPM entries share the UFT space with MAC and IP host route entries).

Finally, the Max Host mode uses fabric UFT for IPv4 and linecard UFT for IPv6, maximizing the number of ARP/ND entries at expense of the FIB table size.

No routing with overlays (BRKARC-2222 slide 81). Trident 2 chipset doesn’t support routing of VXLAN-encapsulated packets, and based on some other vendors’ limitations it seems it has the same challenges with any overlay technology (including TRILL and potentially SPB).

It’s my understanding (based on scarce information available) that the problem might lie in the structure of the forwarding pipeline – by the time the chipset figures out it’s the overlay tunnel endpoint for the incoming packet, and performs the L2 lookup of the destination MAC address, it’s too late for another L3 lookup.

The workaround is hinted at in the BRKARC-2222 presentation: the packet has to be recirculated through the forwarding pipeline.

Remember the front-panel cables between F2 and M1 linecards on Nexus 7000? Same idea, implemented in silicon, probably resulting in similar performance.

Cisco solved the problem with its ACI Leaf Engine (ALE) chipset. One could also implement L3 forwarding on fabric modules in modular switches, or use a second Trident 2 chipset (building a leaf-and-spine architecture within the ToR switch).

Takeaway: Trident 2 has challenges performing L3 forwarding in combination with L2 tunnels. Have a long discussion with your vendor before implementing a design that uses the two features together, even when the datasheets imply everything works just fine.

Finally, looking at the Nexus 9300 architecture (BRKARC-2222 slide 59), there are only 8 40GE lanes between Trident 2 chipset and ALE chipset on Nexus 93128TX, which means that you won’t get line rate VXLAN routing on Nexus 93128TX.

More information

Last week we recorded the second part of the 2014 Data Center Fabrics Update webinar. The videos and slide deck are already available to all participant of past webinars, anyone who bought the webinar recording at any time in the past, and webinar subscribers.

8 comments:

  1. Wonder if it can do encap / decap at line rate so that if a VM is on an SR-IOV the nic can simply tag with a VLAN and the trident can map the VLAN tag to a GRE tunnel or VxLAN VNI

    ReplyDelete
  2. Hi Ivan

    have you seen or review cisco's ACI solution and if so what are you thoughts?

    Ben

    ReplyDelete
  3. Thanks for the information. I´m actually interested in 10GbE switches. Therefore I´m looking in information about "Trident 1". Does a similar "FBRKARC-2222"file exist for Broadcom´s first Trident ASIC (and especially its limitations, comparison to Trident 2,...)?

    ReplyDelete
  4. "The Nexus 9500 non-hierarchical forwarding mode (BRKARC-2222 slide 49) moves the IP LPM table into linecard UFT"...But how then they manage to place LPM entries which require TCAM by their nature into UFT (which is probably a CAM table)?

    ReplyDelete
    Replies
    1. LPM matching doesn't need full TCAM functionality. You might find more information in Intel's FM6000 documentation.

      Delete
  5. "Trident 2 chipset has 16K TCAM table and large Unified Forwarding Table (UFT)"

    Any idea about the impact of using the UFT? What happens when you start using more then the 16k TCAM? Is the lookup as fast or will the latency start to increase? Any other side effects?

    ReplyDelete
    Replies
    1. I guess you'd have to sign an NDA with Broadcom to get detailed answers to these questions ;)

      Delete
  6. I know this post is old but for someone landing on this - Trident2 Broadcom chipset also supports a feature called ALPM (Algorithmic Longest Prefix Match). This is enabled on QFX5100 which allows you to store upto 384K IPv4 entries and upto 128K IPv6 entries by using larger SRAM along with TCAM. I am not certain if ALPM allows both IPv4 and IPv6 entries to go to their maximums.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.