Anyone Using Intel Omni-Path?
One of my subscribers sent me this question after watching the latest batch of Data Center Fabrics videos:
You haven’t mentioned Intel's Omni-Path at all. Should I be surprised?
While Omni-Path looks like a cool technology (at least at the whitepaper level), nobody ever mentioned it (or Intel) in any data center switching discussion I was involved in.
Intel’s solution never came up in my consulting engagements, and it’s not even mentioned in the 2018 Gartner magic quadrant (which means it doesn’t exist in their customer base).
Also, I keep wondering why nobody is using Intel silicon. Arista did something years ago with FM6000, but that was the only time I’ve ever seen Intel ASIC used in a data center switch.
The only time I heard a similar idea was years ago when Intel was talking about switching silicon in NICs (HT: Jon Hudson during an Interop party). At that time, the architecture they promoted was a hypercube built from servers with switching NICs.
While that idea might make sense for very particular workloads (= Finite Elements Method) it’s basically NUMA writ large… and it looks like Intel abandoned that idea in favor of a more traditional approach.
It seems Omni-Path is heavily used in High-Performance Computing (HPC) environments as an Infiniband replacement. No surprise there, Intel always had very-low-latency chipset (that was the reason Arista used FM6000), and combined with all other features they claim they implemented in their Fabric Manager (think proprietary SDN controller that actually works) that would make perfect sense.
However, it looks like even High-Frequency Trading (HFT) doesn’t need that kind of speed. Arista was traditionally very strong in HFT, but after launching one product each Cisco and Arista effectively stopped competing on very-low-latency switches… or maybe the mainstream merchant silicon became fast enough.
Are you seeing something different? Is anyone using Omni-Path outside of HPC world?
Red Rock Canyon was, best I can tell, repositioned as a value add NIC (after all what NIC has a full switch packet processing pipeline available).
And do not forget that when optics directly from the big CMOS ASICs (manufacturable at scale) happens in the 2020s, the economics of those torus-etc designs will be better than leaf/spine for configurations like hyperscale (bought and installed rows at a time and not upgraded or reconfigured).
I looked into a Red Rock Canyon torus design in 2015-2016 and declined to implement it due to cost of optics and poor scalability of bisection bandwidth. Ultimately I don't think a switch-per-server design will ever make sense, especially considering the increasing throughput and radius of merchant switch ASICs. Rack-level direct switching topologies (like Jellyfish and SlimFly above) make more sense but there's a question of whether the routing complexity is worth the cost savings.
I don't think the Silicon Photonics Rapture is coming for the same reason that flash won't replace hard disks: if SiPh becomes briefly cheaper than VCSELs the surge in demand would push the price back up to parity. At best SiPh will reach parity and then both technologies will reduce costs over time, maybe with the mix slowly shifting from one to another.
Intel was a leader in 10G NICs and they threw that away by not releasing mainstream 25/50/100G NICs; this may be tied up in their 10 nm problem. Now Mellanox has 75% NIC market share and there are rumors about Intel buying them.
Omni-Path appears to be derived from Cray and QLogic Infiniband technology and probably has nothing to do with Fulcrum. I would not recommend using Infiniband/Omni-Path outside of HPC now that Ethernet is the same speed.