Rant: Some Internet Service Providers Should Really Know Better...

I was listening to a nice podcast with Nick Buraglio discussing the recent BGP hijack SNAFU impacting Cloudflare (and their reaction) and while I usually totally agree with Nick, I think that he tried to be way too nice when saying (paraphrasing) “I think Cloudflare was a bit harsh - I would prefer a more community-oriented approach along the lines of how could we help you do your job better

That response would be totally appropriate for a small customer who cobbled their BGP configuration from google-and-paste results (don’t get me started), or for a small ISP that can barely keep afloat, but the SNAFU generator in this particular instance was Verizon, a NYSE-traded ISP with decades of experience.

CloudFlare being unable to reach Verizon for eight hours probably didn’t improve their mood...

Also, it’s not like they were hit by a particularly insidious BGP bug (like the overly-long AS path crashing Cisco IOS, or unknown BGP attributes crashing another BGP implementation). They didn’t have a single mechanism in place that would prevent a fat-finger customer from bringing down their core routing.

Even worse, numerous mechanisms they could have used were known for decades - we’re not talking about RPKI or BGPsec but simple stuff like “don’t accept transit prefixes from your customers” and “limit the number of prefixes you accept from a customer”. Most of them are documented in Best Current Practice 194 (RFC 7454), and it’s just the matter of figuring them out and implementing them consistently... which brings me to the root problem.

I’m somewhat naive and believe that Verizon has enough great engineers and in-house BGP knowledge to (almost) guarantee the SNAFU was not caused by technical incompetence, but gross management-level negligence... and there’s only one way to fight that: on the stock market.

Time for a campfire story... A few years ago numerous European ISPs pretended they didn’t need IPv6. Head-in-the-sand strategy worked for most of them, but at least one got a rude awakening when an analyst asked the CEO about their IPv6 strategy on a quarterly results call.

So all we need to make Internet a slightly more secure place is industry analysts asking “what is your strategy to stop BGP hijacks from affecting your customers” on earning calls. Will it happen? This is where I usually stop being an optimist...

Blog posts in this series


  1. Maybe your wish doesn't come true because it's a "too big to fail" ISP...
Add comment