Port or Fabric Extenders?

Among other topics discussed during the Big Hot and Heavy Switches (Part 1) podcast (if you haven’t listened to it yet, it’s high time you do), we’ve mentioned port extenders. As our virtual whiteboard is not always clearly visible during the podcast (although we scribble heavily on it), here’s the big-picture architecture:

After the podcast I wanted to dig into a few minor technical details (just in case someone throws me a curveball during the Data Center webinar) and stumbled into a veritable confusopoly.

Port or Fabric? Cisco uses the term fabric extender; port extender is used in the IEEE 802.1Qbh standard. Obviously one is a technology term, the other one a marketing term (or product name). It looks like everything in the Data Center is *Fabric*, everything in WAN is *Borderless*. Slowly I might get used to it.

Another explanation I found says “Fabric Extender is a standalone implementation of a Port Extender”. I’m not buying it; see the 802.1Qbh introduction (from Cisco) for details; it's very clear they are not talking about a virtualized environment.

Standardized or not? Cisco brought the idea of port extenders to IEEE 802.1 and effectively started the 802.1Qbh working group. However, 802.1Qbh specifies only the control protocol between the controlling bridge and the port extender. It refers to 802.1Qbg for the specification of the tagging on the link between them. 802.1Qbg (which is close to VNTag/VNLink idea from Cisco) refers to 802.1Qbc standard (well hidden behind the paywall) which specifies the actual encapsulation mechanism. The claim that “Cisco has pioneered the VNTag scheme and IEEE 802.1Qbh will use it as a base for the standard” (Cisco Unified Computing System, Cisco Press 2010, page 85) is thus somewhat dubious.

VNTag or not? I was not able to find any authoritative information on the encapsulation format used between fabric extender and controlling bridge. There are plenty of allusions floating around, but nothing very explicit.

There’s also a slight nesting problem: if a fabric extender (Cisco UCS 2100 Fabric Extender) uses VNtag toward the controlling bridge and the UCS blade NIC (Cisco Palo) uses VNtag toward the controlling bridge, we might end with two stacked VNtags.

Fabric Extender limitations. Compared to IEEE 802.1Qbh draft that Cisco started, fabric extenders are extremely limited. For example, the standard aims to provider:

  • Autodiscovery of port extenders. Not available in the Nexus software; you have to manually configure the switchport type and FEX ID on the interface to which the fabric extender is connected.
  • Chaining of port extenders. A port extender can be connected to another port extender, creating a hierarchy. Not available with Nexus 2000 series.
  • Port extender transparency. You can connect anything (host or another bridge) to a port extender physical port. Nexus 2000 products allow only hosts to be connected to the downstream ports; BPDU guard is automatically configured on those ports.

Fabric Extender enhancements. As always, Cisco offers more than the standard it has launched. For example, you can use Virtual Port Channel to have dual-homed fabric extenders.

More Data Center-related technologies

Data Center 3.0 for Networking Engineers webinar (buy a recording or yearly subscription) covers a wide range of Data Center technologies, including port extenders, VNTags, FabricPath and TRILL.

Further reading

6 comments:

  1. It's likely that those features you mentioned that are in 802.1Qbh but not in the current Fabric Extenders will be available in future Fabric Extender products.

    ReplyDelete
  2. Ivan Pepelnjak18 August, 2010 09:23

    No doubt Cisco will have something similar and/or fully support 802.1Qbh. I just wanted to point out the differences that exist today.

    ReplyDelete
  3. Pavel Skovajsa18 August, 2010 19:24

    one of beautiful limitations currently on the Nexus 2k or 5k is the inability to configure SPAN session that has source and destination on the same 5k FEX interface.

    So say you want to span the traffic between one server in the rack, into another server in the same rack - bad luck with your "top of the rack" design, you need to pull another cable from N5k directly into the server.

    Also, as a rant, the older generation of N2k had 1000Mbps port only, so connecting servers with iLO/remote management card that is only capable of 100Mbps was not possible, you had had to pull YET ANOTHER cable from N5k....

    ReplyDelete
  4. Not to mention, the 2148s have the "no port-channel" limitation as well.

    ReplyDelete
  5. features you mentioned that are in 802.1Qbh but not in the current Fabric Extenders will be available in future Fabric Extender products.

    ReplyDelete
  6. Morris Gomes07 May, 2013 07:07

    Fabric Extender Tranceiver(FET) have limitations with the 5k FEX interface too.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.