A while ago, I published a blog post proudly describing the netlab integration test that should check for incorrect OSPF network types in netlab-generated device configurations. Almost immediately, Erik Auerswald pointed out that my test wouldn’t detect that error (it might detect other errors, though) as the OSPF network adjacency is always established even when the adjacent routers have mismatching OSPF network types.

I made one of the oldest testing mistakes: I checked whether my test would work under the correct conditions but not whether it would detect an incorrect condition.

Effectively, my test was a highly dramatized version of¹:

def check_ospf_network_type(node):
  return true

Lesson learned: Whenever you write a test, check what happens when you give it all sorts of incorrect inputs. Also, a false positive (claiming things work when they’re broken) is even more annoying than a false negative (claiming things are broken when they’re OK). You can oftentimes quickly fix a false negative, but you won’t know you’re dealing with false positives until something breaks badly enough to be noticed.