Nexus vPC and Consistency Checker

Michel sent me a detailed e-mail describing both his enthusiasm with vPC and the headaches consistency checker is causing him. Here’s the good part:

Nexus vPC seems like a perfect solution for real multi-chassis etherchannel. At work we're using it extensively on a few pairs of Nexus 7000's.

... and then it turns sour:

However, there is one MAJOR drawback with vPC at this time, it's the way the consistency checker works (or rather, does not work). We've come across two specific situations where consistency checker will bring down your beautiful and redundant vPC link, and we've found no way around.

Here are his problems:


Differing link speeds

Suppose two copper gigabit links in a vPC are set to autonegotiate and are up at 1Gb/s. If for any reason (from cosmic rays to bad cabling) the link on one N7K decides to negotiate down to 100 Mb/s, consistency checker will bring down BOTH links as inconsistent. There goes your network.... brought down by a single link failure.

If you would be able to fix duplex and speed in all cases, it would only be half as bad (but you still need to do the extra config...), but we've also come across (firewall-)vendor that insists some gigabit standard forbids disabling autonegotiation (It might actually be a case of misinterpreting standards in incompatible ways: forcing speed&duplex does not necessarily mean disabling autonegotiation, it might just restrain options).

Configuration changes on vPC members

Some configuration changes (many, in fact) are caught as inconsistencies. There seems to be no non-disruptive way to apply such config changes to a vPC. For instance, we would like to change port MTU. It's a change that fires inconsistency, and even shutting down a member doesn't permit to apply the change to the shutdown member.

Why is there no way to apply config changes to the Po interface and have them propagate to the member ports? OK, there is not a single Po (but two: one on each vPC peer), but what about using Cisco Fabric Services (CFS) to distribute the change and apply in sync on both peers?


I’m no NX-OS expert (as people constantly point out, I’m no expert in many things, but you guys always help me get the story straight), so I decided to go to the source and forwarded the question to Andy Sholomon. He provided a perfect answer and even agreed that I could publish it on my blog. Here it is.


Virtual Port Channels are actually a big part of the Networkers presentation I have done for the last 2 years, so I am familiar with a lot of the issues people have with them.

I have heard issues you raise discussed before, and we are working on making it better. I think some of the newer features would help your reader.

The consistency checker exists to keep bad things from happening on your vPC port-channel (like "unexplained" packet drops). There are two types of consistency check failures. The ones that will bring down the entire port-channel (these are type 1s) and the ones that will only cause an error, or keep a single or group of VLANs from becoming active on the port-channel (type 2).

Decisions for what a type 1 and 2 are were made by the engineering team.

For example the MTU mismatch will bring your vPC port-channel down, and it does so in a non vPC port-channel as well. I am sure that you can see the types of issues that having a link with a MTU of 9000 and one of 1500 can cause for a port-channel.

Some of the other issues are caused by inconsistent configurations. In the N5K we already have the config-sync feature specifically to deal with config mismatch issues. This feature is also coming for the N7K.

Also, starting from NXOS 5.0(2)N2(1) the N5K has "graceful" consistency checks which are on by default. With this feature on, in the presence of a Type 1 mismatch, vPC links stay up on the primary switch. The secondary switch brings down its vPCs until the inconsistency is cleared. This is also being ported into the N7K.

We also have the "peer-config-check-bypass" feature to deal with configuration changes done when one side of the port-channel is down. I think this would help with one of the issues your reader is finding challenging. Bring down one side, make the change on both sides, bring the side back up...at least that is what I understand he wants to do.

We also have the "reload restore" and "auto-recovery" features.

There are some things that you have to be aware of with vPC connections to FEXs (N2Ks) and REALLY painful issues to deal with with "orphan ports" if you don't design accordingly and use the right features. Again, in my opinion, we give you the power and most often the information about issues, you have to design and implement accordingly.

Overall I think the feature is very well designed and stable. I personally like this implementation better than things like VSS, because (again personally) I prefer to have the dual control planes, even when dealing with a "single" (note the quotes) data plane.

As far as speed and duplex issues they fall in the same category as the MTU mismatch, I am sure you can understand why we would want to fail the vPC port-channel. Frankly I hate the "you did the implementation wrong" thing but there are vendors out there that can't get their speed and duplex negotiation done right. I can connect a server to the same port and never have an issue, and then I connect the firewall, based on the same exact server hardware and have renegotiation happen every 5 minutes. I don't really have a good answer for it, beside the fact that we just can't test speed and duplex with every vendors hw and sw combination (again this is my opinion and not Cisco's).


Thanks for a perfect answer, Andy! My beer debt (and I think the beer debt of quite a few readers) has increased significantly.

8 comments:

  1. Hi,

    We've suffered the mutual exclusion check on a pair of N5K's because we didn't use the config sync feature. So, please, keep that in mind when dealing with vPC's!

    Cheers,
    Carlos.

    ReplyDelete
  2. I think Cisco Fabric Services are an excellent suggestion. CFS keeps SAN fabrics consistent. As data center networks become more tightly coupled, these tools are needed.

    In the mean time, while by no means perfect, Putty Connection Manager or 'screen' helps with issuing commands to multiple hosts at the same time.

    ReplyDelete
  3. I think Cisco Fabric Services are an excellent suggestion. CFS keeps SAN fabrics consistent. As data center networks become more tightly coupled, these tools are needed.

    In the mean time, while by no means perfect, Putty Connection Manager or 'screen' helps with issuing commands to multiple hosts at the same time.

    ReplyDelete
  4. I think Cisco Fabric Services are an excellent suggestion. CFS keeps SAN fabrics consistent. As data center networks become more tightly coupled, these tools are needed.

    In the mean time, while by no means perfect, Putty Connection Manager or 'screen' helps with issuing commands to multiple hosts at the same time.

    ReplyDelete
  5. Ivan,

    I agree with the original complaint - that consistency checker takes out VPCs. My gripe is how quick the checker works. If I want to modify the MTU on a 5K pair with back to back VPC then I need out of band access to the switch, which most do not have because it seems to make more sense to uplink 5K pairs direct instead of through an out of band management network considering they are copper. It's so quick that even the multi-send technique of secureCRT does not work. At least give me 10 seconds!!!!

    The consistency checker makes anyone fear god to make even the smallest changes without standing right in front of the 7K or the 5K. Who, still, doesnt get a little scared every time <vlan> scrolls the buffer every time you create one?

    I went ahead and spent quite a bit of money on console servers, which i really never felt i needed in the past for access switches, because of consistency checker.</vlan>

    ReplyDelete
  6. OH and config sync? Yeah i love trusting a switch to make configuration changes for me. How'd that work out VTP?

    ReplyDelete
  7. Having spent the last week or so playing with our new Nexii (?) before they go live we had the same issues with inconsistencies when applying new config to vPC'd ports.

    One thing that we have used is the CFS service to keep both sides in sync at all times, the main reason that we were using it was to keep the FEX configs all the same (we dual-home them) but I suppose it could be used for other uses such as Po's.

    ReplyDelete
  8. >> We've suffered the mutual exclusion check

    Sorry to be dense, but which one is that?

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.