Reinventing Your Own STP Wheel...

One of my readers sent me a link to an interesting L2-over-IP "design". Someone tried to connect two data centers with redundant etherip links using home-brewed redundancy mechanism and (surprise, surprise) managed to bring both of them down. The obvious fix: patch the etherip device driver.

EtherIP is pre-VXLAN Ethernet-over-IP technology yet again proving RFC1925 Rule 11.

I don't know enough about OpenBSD to figure out whether (A) it doesn't have STP at all, (B) STP doesn't work over EtherIP, (C) host routing based on ARP entries would be too much of a hassle, (D) some people don't understand the networking fundamentals, (E) everything looks like a nail once you found a hammer, or (F) all of the above. Insightful comments would be highly appreciated.


  1. OpenBSD has support for STP. It's part of the bridge implementation. The manual for OpenBSDs bridge explains why STP does not work with etherip pseudo interfaces: they lack a hardware MAC address (

    There is a continuation to that story here:
    1. Apparently this limitation of the etherip interface does not exist anymore:
    2. I always thought you don't need a MAC address per interface (just one per box) in a bridge.
    3. The impression I get from the spec is that each individual port has a MAC address. The bridge has one too, but may reuse the MAC address from one of the ports if it wants.
    4. The way I read section 8.13.2, there MUST be a MAC address per MAC Service (= bridge, see section 6.2) and there COULD be a MAC address per bridge port which COULD be used for outgoing frames UNLESS specified otherwise.

      However, I think we should move this discussion to email. Remi knows how to get in touch with me (and I'm guessing you know how to get in touch with Remi ;).
  2. (A) is not true, (B) is not true, (C) is true, (D) is quite likely, and (E) is definitely true.

    FWIW, one side of this link was going into FEXes on Cisco Nexus kit, and they don't appear to like doing spanning tree out the front of those ports. Also, the first attempt at this had pairs of boxes at each site driving a single etherip link between them, which made it hard for STP to do something useful.

    The current setup has been very robust. I'm just sorry to have upset you by showing an intermediate step while I was learning some hard lessons.
    1. Thanks for the feedback... and yes, FEXes (and rbridges and LAN extenders and a gazillion similar kludges) should never have been invented.

      Also, I always appreciate people documenting the failures - that's the only way for everyone to learn and move forward without repeating the same mistakes.
    2. FEX's always block spanning-tree packets (BPDUs).

      Too many network designers think LAN extension is the only way to do IP mobility. That's a shame because IP mobility is delightfully easy and works for almost all network traffic. (the most common exception is multicast & broadcast traffic traffic like heartbeats and discovery protocols -- those still require full LAN extension)
Add comment