Plexxi PSI: MAU at Gigabit Speed

Regardless of the advantages of photonic switching (David Husak claims it’s 20.000 times more effective than electronic switching), the programmable optical components remain ludicrously expensive, prompting Plexxi to launch a cost-optimized fixed-topology version of their data center products.

The next generation of Plexxi switches has two product lines:

  • Plexxi Switch 2 and 2S use the same WDM LightRail backbone as the existing Plexxi Switch 1;
  • Plexxi Switch 2P and 2SP (targeting fixed-topology pods) use the same LightRail connectors, but simpler FP 1310 optics – each fiber pair in the LightRail cable carries a 10 GE point-to-point link.

The beauty of the Plexxi solution is the Chordal ring between the Plexxi switches that obviates the need for the data center fabric spine layer. WDM-based Plexxi switches implement the ring with programmable optics (note: although the optical components are programmable, the current ring topology is fixed); in fixed-topology deployments the switches connect to a Pod Switch Interconnect (PSI) that implements the Chordal ring with prewired fiber cables.

Here’s how a 2-pod deployment would look like:

Plexxi claims PSI has these capabilities:

  • 100% passive optical chassis solution;
  • Passive bypass;
  • Online insertion-and-removal of nodes;
  • Inter-switch paths not disrupted by leaf node maintenance.

Sounds exactly like a description of a Token Ring MAU to me ;)

Joking aside, you might be left wondering (like we were): how exactly does a PSI work?

PSI Internal Wiring

Let’s start with a 30-node Plexxi ring implemented with WDM switches. Each node establishes four 10GE connections (lambdas) with adjacent (east and west) nodes and two 10GE connection with subsequent four east and west nodes. The Chordal ring between the 30 switches looks like this (width of the line between two nodes represents bandwidth; individual lambdas are not shown).

The “bypass” links that don’t touch individual nodes still flow through the nodes on the WDM optical layer.


Plexxi Chordal ring with 30 nodes (5 pods)

Focus on a single 6-node section of the ring. These are the optical connections (lambdas) between the six nodes that we’ll turn into a pod and the rest of the switches in the ring


Connectivity of a single 6-node pod

Replace the lambdas in the Chordal ring with fixed fibers. Optical connections (now fibers) that would terminate at a node are hard-wired to the node’s access port on the PSI. Two 24-strand cables connect the node (switch) and its associated PSI ports, implementing 12 x 10GE connectivity in east and west directions.


Pod wiring implemented with PSI

The “bypass” links cannot flow through the switch nodes anymore; they have to be implemented within the PSI. The internal PSI wiring that implements intra-pod connectivity looks like this:


Intra-pod PSI connections

Connections in our Chordal ring extend beyond a single pod. PSI has to provide those as well – fibers link individual node ports with East and West extender ports:


External PSI connections

Now, what happens if you deploy a single Plexxi pod? All the red links from the previous figure become disconnected (and bits might drop on the floor), significantly reducing the overall bandwidth available in the ring backbone. To use all the available backbone bandwidth in a single pod deployment, you must connect East extender interfaces with West extender interfaces of the same PSI, as shown in the wiring diagrams from the Plexxi NDF6 presentation:

More information

To learn more about the new Plexxi products, watch the NFD6 video. To learn more about the underlying principles and their controller-based networking, watch Dan Backman’s presentation from Data Center Fabric Architectures webinar.

Disclosure

Plexxi was a sponsor of Networking Tech Field Day 6.

2 comments:

  1. Hi Ivan,

    Excellent write-up as usual and your chordal graphing is impeccable! Of course, we could do without the TR comments :)

    Couple of points I'd like to make so there is no confusion on why we did this. The statement "programmable optical components remain ludicrously expensive" is not quite accurate. The only difference in the optics from our WDM to our PSI-based solution is the WDM solution uses CWDM transceivers and the the PSI uses non-WDM (FP) single wavelength transceivers. There is a cost differential in these transceivers, but there is no difference in "programmability" (I'll get to that later).

    On the notion of cost, while it is true that the PSI solution offers an extremely low cost solution, the WDM solution is still very cost competitive against most traditional networks, especially when compared at like for like over subscription models. The Plexxi WDM solution gives customers a way to affordably build low OSR solutions by optimizing around the applications that require low OSR, rather than having to build the entire network for the worst case, and offers a linear growth model that only ever requires one switch at a time.

    The PSI version of our solution was driven by customers that loved the concept of Affinities and dynamic photonic networks, but were lured by the *initial* low cost of high OSR leaf/spines, even if they knew that changes in east-west demands would be much more costly down the road. Since they were also buliding uniform pod structures,they did not need the linear growth capability of the WDM solution. So we were able to optimize the solution, with the only tradeoff being that it needs to be grown in fixed increments of a single PSI worth of switches (6 Switches, or about 400 10 GbE ports instead of 1 Switch at a time).

    On the topic of programmability - just so it is clear - there is no difference in the 2 solutions in this respect. Both WDM and PSI use a fixed pattern of wiring of fibers between switches; the WDM solutions implement the wiring pattern in each Switch, while the PSI solution implements the wiring in the PSI itself, but the pattern is is exactly the same and both are fixed fibers. The programmability comes from the L1 crossbar that is physically on the Switch, in both models, and allows us to create dynamic L1 topologies under SW control.

    Cheers,
    Mat
    VP, Product Mgmt, Plexxi
  2. Hi Mat an Ivan,

    One of you know me a bit, the other I'm hoping to meet. I am a Plexxi customer and would say I really liked their earliest product. As Mat points out, this particular product (you seem to understand quite well) is actually a niche design. When I heard about the PSI, I wasn't real keen on having that over what Plexxis I already had, for some of the reasons that you pointed out. You've also pegged something their sales engineer have somewhat said, too, in that service mode for less than 5 boxes isn't ideal. Four nodes is in my personal book the lowest count I'd run, so that during servicing a switch I wouldn't drop below 3 nodes. But, the beauty of Plexxi is how many boxes you can run off of that Chordal ring, with such ease of cabling and excellent visibility of the system thru the latest controllers. Fairly stated, any mesh or hub/spoke design will have some of the bandwidth issues during service, the nice thing here is cables can run quite far and are extremely simple to connect/manage, unlike a bulk of copper qsfp or dozens of zip cords. I understand Plexxi has a pan-asian provider with the optics broken out across different localities...

    Thanks for your excellent posts!
    Guile
Add comment
Sidebar