I was running fantastic Network Security in a Private Cloud workshops in early 2010s and a lot of the discussions centered on the mission-impossible task of securing existing underdocumented applications, rigidity of networking team and their firewall rules and similar well-known topics.
The make all firewalls virtual and owned by the application team idea also encountered the expected resistance, but enabled us to start thinking in more generic terms.
Keep the Physical Firewalls
Nobody was willing to get rid of the physical firewalls separating the private cloud from the Internet. The idea of connecting the outside network straight to a set of servers (even though they might be in a separate cluster and running solely VM-based firewall appliances) was simply too radical.
However, we also quickly agreed that it doesn’t make sense to maintain thousands of firewall rules on the Internet-facing physical firewalls. Those firewalls should do the basic traffic scrubbing and potentially some content inspection. They could also be combined with IDS/IPS systems or traffic monitoring systems. Most importantly, these devices wouldn’t do any application-specific filtering. Their ruleset would be kept simple and thus easy to monitor and audit.
The outside perimeter built with trusted physical appliances would thus separate the Internet Wild West from a scrubbed (but still not very trusted) DMZ segment. Proxy servers, shared caching servers and some load balancers could connect straight to that segment.
Add the Applications
Individual application stacks would connect to the scrubbed DMZ segment through application-specific firewalls implemented in virtual machines (an approach known in Roman times as divide et impera).
The application teams could use NSX-T edge firewalls and load balancers, F5 BIG-IP VTM or any other VM appliance that offers firewalling and load balancing in the same VM instance, and the simpler single-VM applications might use VM NIC firewalls (available in one form or another in most private cloud ecosystems). Each VM appliance would contain the rules specific to the protected application, making configuration, monitoring and auditing relatively simple.
Most importantly, the security team would generate sample VM images that the application teams could deploy from a central catalog, making sure the application teams start with a well-configured initial state that they could use as-is or modify it for their own needs (obviously taking full responsibility for any deviation from the baseline).