Michel sent me a detailed e-mail describing both his enthusiasm with vPC and the headaches consistency checker is causing him. Here’s the good part:

Nexus vPC seems like a perfect solution for real multi-chassis etherchannel. At work we're using it extensively on a few pairs of Nexus 7000's.

... and then it turns sour:

However, there is one MAJOR drawback with vPC at this time, it's the way the consistency checker works (or rather, does not work). We've come across two specific situations where consistency checker will bring down your beautiful and redundant vPC link, and we've found no way around.

Here are his problems:

---

Differing link speeds

Suppose two copper gigabit links in a vPC are set to autonegotiate and are up at 1Gb/s. If for any reason (from cosmic rays to bad cabling) the link on one N7K decides to negotiate down to 100 Mb/s, consistency checker will bring down BOTH links as inconsistent. There goes your network.... brought down by a single link failure.

If you would be able to fix duplex and speed in all cases, it would only be half as bad (but you still need to do the extra config...), but we've also come across (firewall-)vendor that insists some gigabit standard forbids disabling autonegotiation (It might actually be a case of misinterpreting standards in incompatible ways: forcing speed&duplex does not necessarily mean disabling autonegotiation, it might just restrain options).

Configuration changes on vPC members

Some configuration changes (many, in fact) are caught as inconsistencies. There seems to be no non-disruptive way to apply such config changes to a vPC. For instance, we would like to change port MTU. It's a change that fires inconsistency, and even shutting down a member doesn't permit to apply the change to the shutdown member.

Why is there no way to apply config changes to the Po interface and have them propagate to the member ports? OK, there is not a single Po (but two: one on each vPC peer), but what about using Cisco Fabric Services (CFS) to distribute the change and apply in sync on both peers?

---

I’m no NX-OS expert (as people constantly point out, I’m no expert in many things, but you guys always help me get the story straight), so I decided to go to the source and forwarded the question to Andy Sholomon. He provided a perfect answer and even agreed that I could publish it on my blog. Here it is.

---

Virtual Port Channels are actually a big part of the Networkers presentation I have done for the last 2 years, so I am familiar with a lot of the issues people have with them.

I have heard issues you raise discussed before, and we are working on making it better. I think some of the newer features would help your reader.

The consistency checker exists to keep bad things from happening on your vPC port-channel (like "unexplained" packet drops). There are two types of consistency check failures. The ones that will bring down the entire port-channel (these are type 1s) and the ones that will only cause an error, or keep a single or group of VLANs from becoming active on the port-channel (type 2).

Decisions for what a type 1 and 2 are were made by the engineering team.

For example the MTU mismatch will bring your vPC port-channel down, and it does so in a non vPC port-channel as well. I am sure that you can see the types of issues that having a link with a MTU of 9000 and one of 1500 can cause for a port-channel.

Some of the other issues are caused by inconsistent configurations. In the N5K we already have the config-sync feature specifically to deal with config mismatch issues. This feature is also coming for the N7K.

Also, starting from NXOS 5.0(2)N2(1) the N5K has "graceful" consistency checks which are on by default. With this feature on, in the presence of a Type 1 mismatch, vPC links stay up on the primary switch. The secondary switch brings down its vPCs until the inconsistency is cleared. This is also being ported into the N7K.

We also have the "peer-config-check-bypass" feature to deal with configuration changes done when one side of the port-channel is down. I think this would help with one of the issues your reader is finding challenging. Bring down one side, make the change on both sides, bring the side back up...at least that is what I understand he wants to do.

We also have the "reload restore" and "auto-recovery" features.

There are some things that you have to be aware of with vPC connections to FEXs (N2Ks) and REALLY painful issues to deal with with "orphan ports" if you don't design accordingly and use the right features. Again, in my opinion, we give you the power and most often the information about issues, you have to design and implement accordingly.

Overall I think the feature is very well designed and stable. I personally like this implementation better than things like VSS, because (again personally) I prefer to have the dual control planes, even when dealing with a "single" (note the quotes) data plane.

As far as speed and duplex issues they fall in the same category as the MTU mismatch, I am sure you can understand why we would want to fail the vPC port-channel. Frankly I hate the "you did the implementation wrong" thing but there are vendors out there that can't get their speed and duplex negotiation done right. I can connect a server to the same port and never have an issue, and then I connect the firewall, based on the same exact server hardware and have renegotiation happen every 5 minutes. I don't really have a good answer for it, beside the fact that we just can't test speed and duplex with every vendors hw and sw combination (again this is my opinion and not Cisco's).

---

Thanks for a perfect answer, Andy! My beer debt (and I think the beer debt of quite a few readers) has increased significantly.