Could You Replace MPLS/VPN with IPSec-over-Internet?

Someone recently sent me this scenario:

Our CIO has recently told us that he wants to get rid of MPLS because it is too costly and is leaning towards big Internet lines running IPSEC VPNs to connect the whole of Africa.

He was obviously shopping around for free advice (my friend Jeremy Stretch posted his answers to exactly the same set of questions not so long ago); here are the responses I wrote to his questions:

As you can imagine, this has caused a huge debate between the networks team and management, we run high priority services such as Lync enterprise, SAP, Video conferencing etc. and networks feel we need MPLS for guaranteed quality on these service and management feels the internet is today stable enough to run just as good as MPLS.

Plenty of my customers use Internet-only approach for their international connectivity. Obviously there’s no QoS or end-to-end SLA on the Internet - if it works, it’s great, if not, there’s nothing you can do. That’s why it’s cheaper than MPLS.

Networks with more stringent QoS requirements might use IPSec-over-Internet for non-real-time traffic and MPLS for real-time and mission-critical traffic (example: Lync and SAP) – I’ve seen a large global organization doing exactly that.

If nothing else, this approach reduces the bandwidth requirements of your MPLS network, and thus its costs.

What is your take on the MPLS vs Internet debate from a network engineer point of view? And more so, would running those services over Internet provide the same user experience as running them on MPLS WAN links?

Obviously not. If there’s no difference, you have a bad MPLS service provider that’s overcharging you.

However, you’ll never know unless you try it. As always, start with a small pilot, gather experience, and work from there.

Finally, if you decide to use two parallel networks, you’ll find a few design tips in my Data Center Design Case Studies book, and I’m always available for short online consulting engagements.


  1. The MPLS VPN vs IPSec over the internet debate is good one to have. I think besides performance and reliability it's important to consider management and support of each option. MPLS VPN is much simpler to manage and troubleshoot; the provider handles the transport and routing and the customer just runs a default static route (or BGP if multi-homed) on the CE for most remote sites.

    With IPSec, the customer owns the solution. And that includes configuring, managing, and troubleshooting those IPSec tunnels and that could be a challenging job sometimes especially if customer does not have the right resource in house.
  2. The business I work for made the switch from MPLS to DMVPN (Internet) a few years back in the US/Canada. We noticed no difference in performance. The only difference was a 72% reduction is monthly bandwidth charges. Yeah baby! If your sites are in major metro regions that are in reach of Metro E connections then there shouldn't be a big drop off.
  3. We've had this discussion at where I work, although it's 3 sites.

    IPSEC can be suicide for traditional SIP\RSTP Voice because your gateway devices need to either have extra capacity so they never run into proc\memory congestion, or be able to prioritize proc\memory and throughput for the voice and video traffic. The only devices I have found thus far are the Juniper SRX, which can be setup in a very paranoid (and incredibly complicated) fashion. Everyone else, except the ASA which isn't as nice, seems to expect a router to do that duty.

    Once you run into congestion from an expanding network, or management forces a refresh at a lower-powered hardware, you end up with a completely broken system that requires baby-sitting.

    Sure you could do exec meetings over a paid Skype account, but you need them to agree on a reduced SLA.

    Good news is properly configured IPSEC VPN is 100% rock solid stable.

    There's a very dangerous movement towards putting everything on cloud and fundamentally on consumer grade hardware and services by management in order to save a buck. This is driven by ignorance by both management and staff. Not just from the security angle; we had a discussion about implementing an industry-specific ERP System and I asked a manager who was gung-ho about it "So who owns their stock? Our competitors? What if the competitors buy them out later?". He wasn't happy with that question considering they're a publicly traded company.

    You almost have to be smarter than your manager is these days to justify your job, they are like lemmings, ready to jump the cliff at a moments notice.

    Right now I'm dealing with an IT admin that has some knowledge, but keeps telling me "You go into too many details". E.G. We are building a new server room, and this admin wants to have capacity for X racks from one. "So how do we scale the aircon and generators to get to X racks from one?" What kind of cost and how does that compare to Cloud offerings?". I haven't asked him that question directly, but I've shown him hey, BTU's are units of energy!

    He's going to get steamrolled soon. Same thing.

    MPLS is a very cheap config change for the ISP to make in my area, so we've opted for MPLS.
  4. This is something I struggled with pretty frequently in a prior job working for a managed services provider. MPLS WANs are great because they provide flexible, private connectivity with guaranteed throughput. Most MPLS providers also allow you to choose from a menu of QoS schemes and classify your traffic so that real-time voice and video services are treated higher preference during periods of congestion.My advice would be to stick with the MPLS WAN if you can afford it. A VPN overlaid on top of Internet circuits might work most of the time, but when it doesn't perform adequately, you'll have little immediate recourse. Should you decide on moving to a VPN overlay, do so in phases: Keep the MPLS WAN around for a few months in case the overlay strategy doesn't work out. But if you find that your Internet circuits provide sufficient throughput so that congestion of real-time services never becomes a problem, maybe that's an acceptable solution.
    And thanks to the great support from iron socket vpn nerds,
  5. Does anyone have any experience with services that bond multiple types of broadband Internet connections together simultaneously? Some are beginning to claim "voice quality" service due to "smart multiplexing" over diverse paths, including wireless.

    They usually provide appliances that do the encryption, etc.

    (Fyi: I posted a version of this on Jeremy's blog too)
Add comment