BGP in Data Center Fabrics

You must use BGP as the endpoint reachability routing protocol in your data center fabric if you decide to run VXLAN with EVPN control plane… but should you also run it as the transport (underlay) routing protocol instead of OSPF or IS-IS? The resources collected on this page might help you make that decision.

We covered this topic in these webinars, articles, and podcasts:

• Leaf-and-Spine Fabric Architectures
• EVPN Technical Deep Dive
• Using BGP in a Data Center Leaf-and-Spine Fabric
• BGP in EVPN-Based Data Center Fabrics
• BGP Navel Gazing (Software Gone Wild episode 112)
• Can We Trust Routing Protocols (The Hedge episode 43)
• Cool or Hot? Lapukhov + Nkposong’s BGP SDN (Packet Pushers episode 164)

Petr Lapukhov (the author of BGP-as-better-IGP idea) initially proposed to use BGP as a data center SDN mechanism. I covered that use case in the BGP-Based SDN Solutions webinar and in these blog posts:

• Exception Routing with BGP: SDN Done Right (2013)
• The Four Paths to SDN (2014)

We also tried to answer the question “Do we need a new routing protocol for data center fabrics?” in these Software Gone Wild episodes and related blog posts (TL&DL: No):

• Data Center Routing with RIFT with Dr. Tony Przygienda (2018)
• OpenFabric with Russ White (2018)
• Is BGP Good Enough with Dinesh Dutt on Software Gone Wild (2018)
• BGP or OSPF? Does Topology Visibility Matter? (2016)
• BGP as a Better IGP? When and Where? (2017)
• Is OSPF or IS-IS Good Enough for My Data Center? (2018)
• Is EBGP Really Better than OSPF in Leaf-and-Spine Fabrics? (2018)
• Questions about BGP in the Data Center (with a Whiff of SRv6) (2021)

I covered the design aspects of using BGP in data centers (in particular in combination with EVPN) in these blog posts:

• Using BGP for Firewall High Availability: Design and Software Upgrades (2019)
• Tuning BGP Convergence in High-Availability Firewall Cluster Design (2019)
• BGP in EVPN-Based Data Center Fabrics (Updated) (2018)
• Scaling EVPN BGP Routing Designs (2018)
• BGP AS Numbers on MLAG Members (2020)
• Improving BGP Convergence without Tweaking BGP Timers (2017)
• To BFD or Not to BFD? (2017)
• Avoid Summarization in Leaf-and-Spine Fabrics (2018)
• Leaf-and-Spine Fabric Myths (Part 2) (2018)
• When EVPN EBGP Session between Loopbacks Makes Sense (2020)
• EVPN: The Great Unifying Theory of VPN Control Planes? (2020)
• EVPN Control Plane in Infrastructure Cloud Networking (2020)

It’s also popular to run BGP on redundantly connected servers, or on edge appliances connecting overlay virtual networks with physical world. More details in these blog posts and in the Routing on Servers part of Leaf-and-Spine Fabric Architectures webinar.

• Running BGP on Servers (2016)
• Running BGP between Virtual Machine and ToR Switch (2016)
• Why Would I Use BGP and not OSPF between Servers and the Network? (2016)
• Don’t Run OSPF with Your Customers (2016)
• Sysadmins Shouldn’t Be Involved with Routing (2016)
• Host-to-Network Multihoming Kludges (2016)
• Running BGP between Virtual Machines and Data Center Fabric (2022)

The designers of Cumulus Linux preferred the EBGP-only data center design, and added numerous features to their BGP routing daemon (now FRR). We covered those features in the FRRouting Architecture and Features webinar, in the Cumulus Linux part of the Data Center Fabric Architectures webinar, and in these blog posts:

• BGP Configuration Made Simple with Cumulus Linux (2015)
• Layer-3-Only Data Center Networks with Cumulus Linux on Software Gone Wild (2015)
• Auto-MLAG and Auto-BGP in Cumulus Linux (2019)
• BGP Unnumbered Duct Tape (2022)

Some vendors love making your life overly complex, inventing ridiculous architectures like running IBGP (EVPN) over EBGP (IP routing). Here are a few choice rants blog posts on that topic:

• Is BGP Really that Complex? (2016)
• Dissecting IBGP+EBGP Junos Configuration (2018)
• Don’t Sugarcoat the Challenges You Have (2019)
• The EVPN Dilemma (2019)
• The EVPN/EBGP Saga Continues (2020)
• Networking, Engineering and Safety (2020)

Two other interesting topics often pop up in BGP-related discussions: anycast and multipathing:

• Local TCP Anycast Is Really Hard (2021)
• Using Unequal-Cost Multipath to Cope with Leaf-and-Spine Fabric Failures (2021)
• Unequal-Cost Multipath with BGP DMZ Link Bandwidth (2021)
• Optimal BGP Path Selection with BGP Additional Paths (2021)
• Building a BGP Anycast Lab (2021)

You might also find these blog posts somewhat relevant to your data center BGP designs:

• BGP-Free MPLS Core with Segment Routing (2021)
• Packet Forwarding and Routing over Unnumbered Interfaces (2021)
• BGP Optimal Route Reflection 101 (2021)
• Circular Dependencies Considered Harmful (2021)
• BGP Route Reflector Myths (2022)
• Mixed Feelings about BGP Route Reflector Cluster ID (2022)

These BGP details might help you when designing or deploying your next BGP-based network:

• Dynamic Negotiation of BGP Capabilities (2021)
• Highlights: Dynamic Negotiation of BGP Capabilities (2021)
• Feedback: Recursive BGP Next Hop Resolution (2022)
• Three Dimensions of BGP Address Family Nerd Knobs (2022)
• Modern IBGP Design with AddPath and ORR (2022)

Finally a few more abstract blog posts to tickle your gray cells:

• Routing Protocols: Perfect Example of RFC 1925 Rule 5 (2017)
• BGP: the Tragedy of the Commons (2017)
• Valley-Free Routing (2018)
• Valley-Free Routing in Data Center Fabrics (2018)
• Implications of Valley-Free Routing in Data Center Fabrics (2018)
• Can We Trust BGP Next Hops (Part 1)? (2020)
• Next-Hop and VTEP Reachability in EVPN Networks (2020)
• Can We Trust BGP Next Hops (Part 2)? (2020)