STP loops strike again
Vasilis sent me an interesting campfire story. It started with a common mistake:
An external partner of my company used an Ethernet cable and connected two switchport interfaces of one of our access switches .
Being a conscientious networking engineer, he had the usual safeguards in place ...
I have to mention that these access ports are configured with Root, BDPU guard as well as Portfast.
... but he nonetheless experienced a total LAN meltdown:
This had a severe effect (unavailability) to our LAN. L2 loops caused our L2 switches, Core Switches and Routers to reach 100% CPU!
Root cause analysis turned out a surprising fact:
I thought that BPDU and Root guard were enough to protect the access ports of our LAN but found out it was not true.
Fortunately, the Cisco documentation explaining the problem is quite explicit:
It should be noted that short-living loops are not prevented by Root or BPDU Guards, if two portfast-enabled ports are connected directly or through the hub
The problem? Looped ports won’t shut down until a BPDU packet is sent through one of them, and a single broadcast (for example, ARP packet) sent in that interval can cause a network meltdown.
The solution? Vasilis found a solution similar to those proposed in comments to my Preventing Bridging Loops Without STP post: use switchport port-security and limit the number of MAC addresses accepted on the switch port.
Unfortunately, this solution works primarily in campus environments; you cannot use it in virtualized data centers with moving VMs as you can never predict how many VMs (and MAC addresses) will reside within a physical server.
Better ideas? Please share them in the comments!
Short summary: Bridging (also known as layer-2 switching) sucks (and some implementations suck more than others). We need to move to L3-based solutions.
http://www.cisco.com/en/US/docs/switches/lan/catalyst2950/software/release/12.1_22ea/SCG/swtrafc.html#wp1229854
We should ask Vasilis what is the size of his VLANs (how many hosts/ports) because this surprise me that the storm is so brutal (too many broadcast? Multicast?)!
https://supportforums.cisco.com/message/3422955
- Explicitly shut down ports that are not supposed to be connected to anything
There's another interesting case when Rx and Tx on the same optical port are plugged together. There's a command that detects looped ports and shuts them down (don't ask me what it is though).
Wrong. Well, it depends I guess if he configured portfast on port or globally. If configured on port then the port will always be in portfast mode but if configured globally then the port would revert out of its portfast mode if a BPDU is received. Another common mistake is to use both bpdu filter and root/bpdu guard on a port. The filter will take precedence and the guards will never be used.
Brads solution is of course the best but maybe not always an option.
Storm control and port-security should also be configured if possible.
The ARP situation you mention is very interesting, I've seen this before, one single packet can wreak havoc.
Another interesting situation can occur when customer creates loop. Say that you have customers connecting to a l2 network via some triple play box and they accidentally create a loop locally. Even if your backbone is protected from the backbones perspective this is not really a loop. It's just a lot of traffic coming from the customer which could be ARP or other traffic and this is very difficult to protect against unless you own the CPE or at least configure it.
Probably this happens because the affected switch is in 100% cpu and cannot send BPDUs, udld packets
STP is a great protocol ... for the 1990s.
As for bpdu filter, it always helps to remember the different behavior if enabled globally vs. on the switchport directly; If you enable bpdu filter globally and portfast on a switchport, it will prevent that switchport from sending BPDUs. However, if that port receives a BPDU, the port loses it's portfast state, disables BPDUfilter and becomes a normal STP port. If you enable bpdu filter on a switchport directly, then that disables STP on that port -- ie. wont send BPDUs and it will ignore inbound BPDUs.
P.S. Also its better if you have a MRTG/RRD graphs, drawing the non-unicast pps per port.
What are the chances that a cross-over cable gets used in this or any scenario?
We can't really set that to 1, as anyone experimenting with virtual machines will run into issues. Is setting the mac address limit to something like 20 still helpful? I would hope that setting it to 20 would be high enough to prevent causing issues with any legitimate activity, but still low enough to prevent loops.
One MAC address is a bit tough, but more than a few doesn't make sense. Also, you can age secure MAC addresses if you wish.
The switch should send several BPDUs on a portfast port after it transitions to enabled and delay forwarding for a second or two (to check for potential loops).
Whatever alternate protocol (LLDP would be the best bet) could be used to detect loops in case someone thinks using BPDUs is not a good idea, and the port should not go into forwarding state until the basic are-we-looping checks have been performed.
Maybe smth else is not mensioned, e.g. bpdufilter?