One of my readers sent me a question along these lines after reading the anti-automation blog post:

Your blog post has me worried as we're currently reviewing offers for NGFW solution... I understand the need to keep the lid on the details rather than name and shame, but is it possible to get the details off the record?

I always believed in giving my readers enough information to solve their challenges on their own (you know, the Teach a man to fish idea). While I could tell you “avoid vendor X”, that doesn’t mean that vendors Y and Z don’t have a similar problem, or won’t have it in the future… but if I tell you that some vendors can’t spell automation, you know what to look for.

In any case, you should have a more structured vendor selection process anyway. This is what I’d do:

Figure out what you need based on real business needs, not on “it would be so cool to have” wish-list compiled by IT engineers. We all know the impact of vendor whitepapers and conference attendance on such lists.

Build a high-level list of realistic requirements based on what you expect to be doing in the next ~3 years. Have everyone else sign off that list, so they know what they can expect to get.

Kitchen-sink lists I see in many RFPs are a clear sign of either laziness or CYA mentality, and make as much sense as asking for interchangeable wheels and tank treads on your car because you might do some mud racing in a few years. Oh, and while we’re at it, how about a machine gun stand on the roof because we might diversify into high-speed clay pigeon shooting.

Transform the high-level list into vendor selection requirements and a test plan. Don’t ask for something unless (A) you know why you need it and (B) you know how to check whether it works as expected.

Have no idea what to ask for? I put together a list of potential automation requirements a while ago, and we spent a lot of time on a list of IPv6 requirements that made it into a RIPE document.

Don’t ask me for a security-related list of requirements. I know nothing about security, but I’m positive there are tons of people out there (like my friends at ERNW) who could help you.

Need help with this step? Most probably I won’t have the time to help you, but I’m positive we can find someone in the ExpertExpress team who can.

Build a short list of vendors. Lacking better ideas, use the Gartner Magic Quadrant.

Do a quick sanity check. Don’t trust $vendor PowerPoint presentations. Do a quick scan of their documentation to see whether it describes how to do what you need. Can’t find any mention of what you’re looking for? That’s usually a red flag.

Walk away if you can’t find public online version of product documentation… after writing a polite email to the pesky $vendor account manager explaining why you can’t possibly consider their products. Also, public shaming sometimes does wonders (or not).

Get an evaluation version of the product. Can’t do it quickly because it’s still not available in VM format? Walk away – there are other vendors out there who understand which millennium we’re in.

Test whether the product meets your requirements. I know this sounds like a waste of time, and I agree the vendors shouldn’t make claims that their products don’t meet… but could we get back to the reality of planet Earth where so many marketectures work only in PowerPoint?

Oh, and if you’re serious about automation and want to have some fun, here’s a tip by David Barroso:

If you want to have fun tell vendors that during RFPs/demos any interaction with equipment can only be done with curl/requests/ncclient and that they have to be ready to use those to do small changes and troubleshoot while on site.