Category: design

An attendee of my Building Next-Generation Data Center online course asked me what the best learning path might be for a total (data center) beginner that has to design and install a small leaf-and-spine fabric in a near future.

This blog post was written for ipSpace.net subscribers who want to get the most out of ipSpace.net content. If you’re only interested in free stuff, you might feel it’s a waste of your time. You’ve been warned ;)

Our good friend mr. Anonymous has too many buzzwords and opinions in his repertoire, at least based on this comment he left on my Using 4-byte AS Numbers with EVPN blog post:

But IGPs don't scale well (as you might have heard) except for RIFT and Openfabric. The others are trying to do ECMP based on BGP.

Should you be worried about OSPF or IS-IS scalability when building your data center fabric? Short answer: most probably not. Before diving into a lengthy explanation let's give our dear friend some homework.

I always love to read the practical advice by Andrew Lerner. Here’s another gem that matches what Brad Hedlund, Dinesh Dutt and myself (plus numerous others) have been saying for ages:

One specific recommendation we make in the research is to “Build a rightsized physical infrastructure by using a leaf/spine design with fixed-form factor switches and 25/100G capable interfaces (that are reverse-compatible with 10G).”

There’s a slight gotcha in that advice: it trades implicit complexity of chassis switches with explicit complexity of fixed-form switches.

The proponents of the “let’s run EVPN over EBGP underlay” idea often ignore an interesting challenge: EVPN advocates the use of automatically-generated Route Targets, which might not work when every leaf switch uses a different AS number.

I explored this particular can of worms in the EVPN Route Target Considerations section of the Using BGP in a Data Center Leaf-and-Spine Fabric saga.

My BGP in EVPN-Based Data Center Fabrics blog post generated numerous comments from engineers disagreeing with my views on using IBGP-over-EBGP.

As usual, there were three kinds of comments:

We managed to get another awesome lineup of speakers for the Spring 2018 Building Next-Generation Data Center online course.

Russ White, one of the authors of CCDE and CCAr programs and highly respected book author will start the course with a topic everyone should always consider when designing new infrastructure: how do you identify tradeoffs and manage complexity, making sure you meet the customer requirements while at the same time having an easy-to-operate infrastructure.

Russ White wrote a great article along the lines of what we discussed a while ago. My favorite part:

There are companies who consider the network an asset, and companies that consider the network a necessary evil.

Enjoy!

On a tangential topic: Russ will talk about network complexity in the Building Next-Generation Data Center online course starting on April 25^th.

Years ago, I decided to try out another idea: solving real-life challenges with the help of an easy-to-consume online consulting service. When I discussed the idea with my friends during one of the early Networking Field Day events the opinion was pretty unanimous: “this will never work”

Fortunately, they were wrong. Not only did ~100 customers decided to use it in the meantime, the simple idea grew to a point where I couldn’t do it all on my own.

I had an interesting “how do you build a small fabric without throwing every technology in the mix” discussion with Nicola Modena and mentioned that I don’t see a reason to use EVPN in fabrics with just a few switches. He disagreed and gave me a few good scenarios where EVPN might be handy. Before discussing them let’s establish a baseline.

The Setup

Assume you’re building two small data center fabrics (small because you have only a few hundred VMs and two because of redundancy and IT auditors).

Although it’s almost three months till the start of the Building Next-Generation Data Center online course, we already have most of the guest speakers. Today I’d like to introduce the first two (although they need no introduction).

You might have heard about Russ White. He was known as Mr. CCDE when that program started and recently focused more on data centers, open networking and whitebox switching. He’s also an authority on good network design and architecture, network complexity, and tradeoffs you have to make when designing a network.

Regardless of how much I write about (the ridiculousness of using) stretched VLANs, I keep getting questions along the same lines. This time it’s:

What type of applications require L2 Extension and L3 extension?

I don’t think I’ve seen anyone use L3 extension (after all, isn’t that what Internet is all about), so let’s focus on the first one.

Stretched VLANs (or L2 extensions) are used to solve a number of unrelated problems, because once a vendor sold you a hammer everything starts looking like a nail, and once you get used to replacing everything with nails, you want to use them in all possible environments, including public and hybrid clouds.

Two weeks ago we discussed whether it makes sense to use BGP as the routing protocol in a data center fabric. Today we’ll tackle three additional design challenges:

• Should you use IBGP or EBGP?
• When should you run BGP on the spine switches?
• Should every leaf switch have a different AS number or should they share the same AS number?

A while ago I helped a large enterprise redesign their data center fabric. They did a wonderful job optimizing their infrastructure, so all they really needed were two switches in each location.

Some vendors couldn’t fathom that. One of them proposed to build a “future-proof” (and twice as expensive) leaf-and-spine fabric with two leaves and two spines. On top of that they proposed to use EBGP as the only routing protocol because draft-lapukhov-bgp-routing-large-dc – a clear case of missing the customer needs.

One of my friends wanted to design a nice-and-easy layer-3 leaf-and-spine fabric for a new data center, and got blindsided by a hyperconverged vendor. Here’s what he wrote:

We wanted to have a spine/leaf L3 topology for an NSX deployment but can’t do that because the Nutanix servers require L2 between their nodes so they can be in the same cluster.

I wanted to check his claims, but Nutanix doesn’t publish their documentation (I would consider that a red flag), so I’m assuming he’s right until someone proves otherwise (note: whitepaper is not a proof of anything ;).

I published the last part of my Optimize Data Center Infrastructure series: build an optimized data center fabric.

To learn more about data center fabric designs, check the new online course or enroll into the Spring 2018 session of Building Next-Generation Data Center course.