Public Cloud Networking Security is Different
If you’re running a typical (somewhat outdated) enterprise data center, you’re using tons of VLANs and firewalls, use VLANs as security zones, and push inter-VLAN traffic through firewalls for inspection. Security vendors love that approach - when inspecting traffic they can add no value to (like database- or backup sessions), the firewalls quickly become choke points that have to be upgraded.
Networking security in public clouds is totally different - you’re supposed to use stateful packet filters (usually called security groups) in front of every VM and use dynamically-updated VM groups instead of IP addresses or subnets to specify traffic source and destination. Inserting your own VM-based firewall into the forwarding path is pretty hard in AWS and Azure… and might become expensive as public cloud providers happily charge you for every VM resource you use.
AWS recently launched Ingress VPC Routing feature that makes it easier to inspect traffic coming from the outside of your cloud environment, but the setup is still interestingly complex.
Is it safe to replace traditional firewalls with packet filters? Should you bring your own firewall appliance or use edge firewalls provided by major public cloud providers? Should you combine edge firewall with intra-cloud security groups? How hard would it be to insert your own firewall VM into the forwarding path, and is it worth it? Can you do it for intra-cloud traffic, or does it work only for external traffic? How could you implement a firewall instance failover? We’ll address all these questions in our Networking in Public Cloud Deployments online course. All you have to do is to register.
Also, there's a huge difference between intra-VPC service insertion, inter-VPC service insertion and north-south (Internet-to-VPC) service insertion.
Finally, FWIW: BGP on VM appliance is useful only when you're running IPsec tunnels with spoke VPCs. Slow and expensive.
For HA use BGP on HUB. In the spoke use cloud VPN with BGP IPSEC.
This topic is well covered by the networkdesign arena:
AWS Transit Gateway and Multi-VPC Design Options, for Hybrid Cloud Architecture
Also FYI the blog page is not allowing me to publish with IPSPACE account
As for using ipSpace account for comments - that's been on my to-do list for years. Unfortunately there's always something more urgent or important to do, like creating new content... but eventually I'll get there, I promise!