Anti-Automation from the Antimatter Universe

One of my readers sent me a vivid description of his interactions with one of the so-called next-generation firewall vendors. Enjoy!

We’re using their highly promoted Next Generation Firewall (NGFW) management solution. New cutting edge software, centralized manager… but no CLI for configuration (besides some initial bootstrap commands). "You don't need that because everything is managed from our centralized manager GUI", says $vendor sales managers.

After clicking dozens of settings via manager GUI, reference setup is prepared for NGFW appliance.

Having multiples sites with similar architecture, we would like to replicate setup for another NGFW appliances. A perfect automation case: change a few input parameters, produce rest of the typical configuration.

Ouch. CLI is not supported. Forget Python or Ansible.

Even worse: configuration templates or configuration cloning is not supported in centralized manager GUI. "Good idea, please create enhancements request", says $vendor sales managers. "And by the way we have REST API support in our centralized manager"

OK, sounds acceptable. After wasting hours and opening numerous TAC cases, it appears that:

  • There’s ~30% feature parity between REST API and GUI configuration options. You still need to tweak majority of the configuration manually in manager GUI.
  • Even for configurable parameters not all REST API methods are supported. For example, to update an Object List to add a new host, you need to download the whole Object List, parse and modify its JSON structure, and upload the whole list with another REST API call. A single mistake and your list is gone.

Here’s the result of a long discussion with $vendor engineers:

Currently our REST API implementations is very raw. We don't recommend to use it in production. We’re accepting all your observations and will work toward improving the API. Stay tuned and check the next SW releases

Let’s not comment how our team is feeling at this moment…

Going Further

So we manually (clicking madness) replicated configuration to other appliances. We’re making progress, everything is fine… until you have your first NGFW appliance hardware failure.

According to the RMA procedure, a new appliance is shipped and replaces the failed one. You set some initial settings to pair it with the centralized manager.

Configuration from the failed appliance is already in the centralized manager. You just need to sync/deploy it… but nothing happens. You get multiple error messages. Documentation is silent.

Then you get this explanation after lengthy investigation by $vendor TAC:

Currently it is not possible to Sync/Deploy config from centralized manager to replaced appliances due to some internal reasons, which we cannot disclose. As workaround please delete all configuration from manager and implement all from the scratch, as during new implementation. Please don't forget to make dozens of screen shots with existing configuration options

You could guess how our team is feeling after this revelation.

After long discussions with $vendor sales engineers they tell us: "We understand your frustration, we indeed registered some enhancements request. Based on priorities, they will be implemented"

I Couldn’t Resist Adding…

And this is what happens when you buy things based on glitzy PowerPoint without ever considering what you should ask for to make your operations easier.

Next step: CxO (who made the purchasing decision in the first place) will complain how expensive his internal security and networking teams are.

I also sent a draft of this blog post to my friends working in the security industry and one of them replied:

I don’t understand why some vendors actually believe that CLI’s will die. I think their vision of automation is a monkey trained to click certain sequences of GUI buttons to get to a banana…

As we all know it’s not just the vendors. So-called thought leaders with zero operational experience are no better.

But Wait, That’s Not All

Finally, the icing on the cake. The same reader sent me this gem a few days after the original email:

We asked the vendor: “Could you please officially share other customers’ experience with us?”

Their answer: “No, because that would affect our business”

Now you know how some players in this industry work.

But Wait, There’s Even More

Got more feedback from another unhappy customer (probably using the same gear). Here it is:

Allow me to add my experiences with $vendor and his $product (I had a look at all releases starting from 6.0 to the first 6.2 releases in the course of about 2015-2017):

  • Management access is utterly slow, needs an awful lot of resources to run just below an acceptable speed baseline,
  • The whole thing is a huge pile of bugs in runtime as well as management code,
  • Numerous TACs, often taking weeks to resolve one issue,
  • Essentially no help in $vendor forums (often discussions cease with “open a TAC”).

$vendor bought $product with the acquisition of $company some years ago and struggles since then to get the whole pile transformed to something I could recommend to customers with good faith.

On the other hand, I had contact with some techs at $vendor and they are competent and know how to diagnose problems.

The best part (for $vendor) is that $vendor charges customers money (via a so called "support contract") for allowing customers to help finding and (hopefully) fixing bugs the software shouldn't have in the first place. At least not in the given amount.

Looking at the base FW (without the shiny NG): it’s mature, has a lot less of features in comparison but is reasonably stable.

Seems to be a good lesson in how to upset customers.

Latest blog posts in CLI versus API series


  1. Now we would like to know what firewall vendor that is..
    Not doing so would be akin to not helping someone in danger :)
  2. I understand why you might be reluctant to disclose the vendor name but it would be great if there was a way for these sort of vendor issues to be reported and shared openly. It would also be good to see the positive experiences that people have too.

    Vendors constraining everyone with NDAs is not good for the industry (customers in particular).
    1. A pure guess, check out which security vendors is missing from napalm supported matrix.
  3. pretty sure it starts with a c and ends with an o
    1. Hah, judging from the end 6.0 and 6.2 comment and my familiarity with C --- O's NGFW product, I bet you're spot on. Although to be fair, CLI is supported from a troubleshooting/diagnosis perspective, just not config. I wish they'd take notes from Big Switch in their controller based architectures...
  4. The vendor you're suggesting isn't exactly unfamiliar with the power of a CLI - my guess in this case would be that it starts with a C and ends with a t.
  5. "My guess in this case would be that it starts with a C and ends with a t." That seems correct from our perspective, save for the last caveat described in the post: "delete all configuration from manager and implement all from the scratch". We actually could restore a configuration (didn't have to rebuild from stcatch).
  6. Sounds like Checkpoint, in which case, maybe they can refer to this for some info on automating Checkpoint using Ansible;
  7. and for the this reason we are still using the A*A firewalls for East-West data-center traffic as they are really mature and robust (with no performance penalty of DPI) and use the Fortigate for North-South traffic with too many security features like AVC , IPS , AV ....the FortiOS is really reach in CLI but lack some parity on the API side.
  8. Since it says an “acquisition” and the software release versions 6.x, I strongly believe it starts from C and ends with O :)
  9. Please someone helps.

    What vendor are you talking about?

    Cisco or Checkpoint or ...?
  10. C***o trying to evolve from the A*A: First separate 3rd party deep inspection modules, then separate own modules, then bought big IPS vendor and mess everything up. Still separate module, half of the box has one os and cli, the other half doesn't and is managed only from external system, no one understands anything anymore. Dumped them and switched to FG.
  11. FirePower is a nightmare.
    I just deleted like 5-6 lines of comment here pointing on all of the issues we had and we are having now with this "revolution".

    We have enormous amount of issues using this product (and we are not using single standalone setup but enterprise customer solution) and we have asked the same:
    - are we the only one?

    Noone will give you direct statement of course but i was pretty sure that blogs like this will come up eventually.

    Fun fact: we have provided 3 A4-size pages to vendor (bugs/feature requests) like half year ago.
    Guess how many of these will be fixed in 6.2.3? :)

    It's not even funny anymore.

    My personal list of "features"
    - there is no logical backup!
    - if you have to re-register device to management, you will loose zone-to-interface bond and all L3 related configuration (even routing!)
    - try to ping unreachable routed host with for example 1000 repeats (you will see magic happening)
    - if management will accept something that device is not capable of, this results in config wipeout

    And many more ...

    And amount of TAC cases is enormous!

  12. We have the same experience with this device.
    An absolute horrifying monster of a nightmare.

    Literally none of the features is working as designed and every bug you file is either converted to a feature request (which will never actually be implemented) or as a low-priority bug. In the last 2 years, out of 40+ bugs, one was "fixed" but it was only fixed cosmetically, so the problem still existed.

    I still cannot wrap my brain around the fact that this is literally the most expensive NGFW that you can buy and nothing of it is really working.
    I have next to me a cheap fortinet firewall that, although it doesn't have the same throughput, can literally do more than the FP.

    It is literally a sourcefire box, taped on top of an ASA which has been glued into a UCS server but they forgot all of it needs to be able to interface with each other.

    Also, the vendor has commented that they currently do not have the resources to fix all the bugs, let alone look at any feature requests.

    Also, if you imagine you will do things yourself over the API, API is mostly broken or does not accept commands that are in the vendor's documentation (like enabling an interface after creation - you have to still do this manually through GUI).
    Furthermore, there is finally a backup option, but restore is not possible...

    Stay away from this "next generation" firewall as far as you can.

  13. This sums it up pretty well also:

    And the numerous, *numerous* other cries from the depths of the FP rabbit hole
Add comment