Internet peering disputes: follow the money

You’ve probably heard about the recent peering dispute between Level-3 and Comcast ... and might have enjoyed the frenzy with which the blogging pundits have followed the false net neutrality scent left by Level-3 spin doctors.

Facts first: Level-3 is trying to dump huge amount of data into Comcast’s network for free.

Follow the money: Comcast will eventually have to upgrade the network to carry the additional data. It cannot charge the end-users (or people will start screaming about net neutrality). Giving the users their fair share of bandwidth could be a problem - either due to outdated DOCSIS equipment or because Netflix streaming simply wouldn’t work on reduced amount of bandwidth ... raising new cries about violations of net neutrality.

I never understood how giving an end user a fair share of the core/access bandwidth is not neutral when the fair share happens to be too small to support the user’s application.

To support massive Netflix deployment, Comcast’s network would have to be upgraded. The party causing the massive increase in traffic (Level-3) is playing dumb and wants to get a free lunch. Comcast is trying to squeeze some money out of them. Business as usual.

Is anyone innocent? I doubt. However, the motives of Level-3 (and its tactics) are obvious for everyone to see (unless, of course, you hysterically fall into the neutrality lure). Comcast might prefer not to have Netflix on its network, but as long as Comcast doesn’t block or interfere with Netflix traffic, there’s no reasonable argument that it’s violating any rules.

What probably happened? Level 3 offered Netflix a better deal than their existing CDN partner and tried to push all the costs over to Comcast.

Why am I interested in this? It’s a perfect example of the discrepancies (and failing business models) in today’s Internet that cause most of the problems I’m describing in the Upcoming Internet Challenges webinar.

More information (thanks to my Twitter friends):

Recent posts in the same categories

26 comments:

  1. plapukhov 01 December 2010 16:11
    Well this is what you get in datagram networks with distributed control :) No end-to-end QoS, no end-to-end management (includes accountability), nothing end-to-end in fact. In turn, this effect is a large-scale reflection of the hop-by-hop independence principle introduced by packet networks. This has been one of the main challenges to "transport neutrality" for years - if X sends an *unsolicited* packet to Y across A,B,C and D, how would A,B,C and D bill X and Y for such transportation? If X and Y are paying, do they have any control over the path their packets take? And while nothing much we could do about it in existing architecture we can always blog about the good, the bad and the ugly! :)
  2. Jason Tracy 01 December 2010 17:05
    I disagree that Comcast can't raise its prices. The money model for ISPs is to charge THEIR customers for access to the Internet. Level 3 and Netflix are providing content *free of charge* to Comcast. Their content is the reason Comcast has customers that are willing to pay for Internet service.
    If Comcast's customers are using more bandwidth than Comcast can provide, they need to adjust how they are charging customers. They might raise the price on unlimited access, use bandwidth caps, or change to a pay by the gigabyte model. Regardless, this is Comcast's problem to solve, since *Comcast's users* are requesting the data.
  3. Jason Tracy 01 December 2010 17:08
    Also, your "fact" that Level 3 is "dumping" data into Comcast's network is false. They are delivering content.

    Dumping would imply that Level 3 expects Comcast to allow traffic to transit its network on the way to another ISP. If that was the case (and I've heard nothing supporting that), then I would completely side with Comcast on this.
  4. Josh Gant 01 December 2010 17:08
    If Comcast wants to have a network infrastructure that can deliver this new Netflix content to their customers in a way that is acceptable to their customers isn't it Comcast's responsibility to invest in their infrastructure? If they need to raise their prices to do this (which I highly doubt is "necessary") isn't that a perfectly fine solution?

    I fail to see how this is a problem with what Level 3 is doing.
  5. santino.rizzo 01 December 2010 17:31
    Agree with Jason and Josh. I made the same argument to Mike Fratto after he also claimed this was a simple peering issue. This is not traffic in transit across Comcast's network. This is content Comcast users are requesting, and bandwidth they are already oaying for. If Comcast cannot supply their own customers that bandwidth, it's their fault, not Level 3. The fact that the content (Netflix) competes with Comcast's own services makes this all the more suspicious and why Net Neutrality claims are not baseless.
  6. stretch 01 December 2010 17:56
    Glad to see Jason and Josh already wrote what I was going to. :) Far from "dumping" traffic onto Comcast's infrastructure, Comcast's paying customers are *requesting* this data as end users. Level 3 cannot be faulted for Comcast's failure to provide adequate service.
  7. Ivan Pepelnjak 01 December 2010 18:07
    Jason, Josh, Santino & Stretch,

    Let's go back to the basics. If Amazon wants to deliver you a book, they have to pay shipping charges, even though you (also a customer of US Post or whatever other postal service) have requested the book. Why should the digital world be any different? If I want to use your network to deliver my content, I have to pay for it (under the current business models, I have to pay for the bandwidth in/out of your network).

    Next, if it wouldn't be Level 3 but a regular CDN or content provider (or your business), they would have to be Comcast's customer if they would want to have direct access to Comcast network. Why is this different because Level 3 and Comcast USED TO HAVE a zero-settlement peering relationship? Why is Comcast violating Net Neutrality if they simply say "dear Level-3, with 5:1 traffic ratio, we don't think you're entitled to a zero-settlement peering"? Why is it OK for Cogent and Level-3 to fight, but when it's Comcast, it's all about net neutrality?

    Last, if you'd taken the time and read the CNN article I quoted at the end, you would have seen that Netflix used Akamai (which was acting as Comcast's customer) and switched to Level 3 (presumably due to lower prices offered by Level 3). Why would Akamai being Comcast's customer be OK, but Level 3 should get the same service for free?

    Things are plain and simple: Level 3 is gaming the system and using the Net Neutrality cloaking device to apply pressure to Comcast. I am sad to see this is so hard to grasp, it's Economics 101.
  8. Ivan Pepelnjak 01 December 2010 18:24
    FYI, if you want to know the whole story, you have to read this:

    http://blog.comcast.com/2010/11/comcasts-letter-to-fcc-on-level-3.html

    Level 3 asked for 30 new interconnection ports ... for free. If I would be Comcast, I would say "no way".
  9. Jason Tracy 01 December 2010 19:25
    And if I were Level 3, I would tell Comcast they need to pay me for access to my content.

    Point being, that content is going to find its way to Comcast either through Level 3 direct, or though some other carrier like AT&T. Comcast has the choice of getting 30 free ports, or having its ports with other carriers saturated.

    Personally, I don't understand why the other CDNs have been paying Comcast to directly connect. That is something that makes life easier on Comcast. Just because Akamai paid the bribe doesn't mean others should be silly enough to do so.

    Lastly, your point about Amazon delivering a book is a great one. What you are proposing is that Amazon pays for sending the package TO the postal carrier, and then I pay to get it FROM them. The way it works with mail is exactly what I want to see. The person requesting the goods pays for them to be delivered. When I buy something from Amazon, I pay shipping. It has its own line item, Amazon does not pay for it. When I buy a movie online, I pay for the shipping through my ISP. Same thing.
  10. Laurent Geyer 01 December 2010 19:31
    I wish it were as clear cut as you are trying to make it seem Ivan, but I feel like there is a fundamental flaw with your argument that needs to be addressed when having this discussion.

    You brought up Amazon book delivery as an analogy, and that's exactly where your argument falls apart in my humble opinion.

    "Content Delivery" may be a convenient way of describing the exchange of information on the Internet, but I think there's a semantic problem here and I don't think delivery accurately describes the flow of information here.

    From the consumer perspective, and that to me should really be the key focus, I could care less how network providers settle the exchange of information. What I know is this, I am contracting Comcast to provide me with access to the Internet, and unless Comcast starts re-qualifying exactly what that means, I am going to operate under the assumption that they will make every single network that's part of the global routing table available. The idea here being that I, as a consumer, am able to request information made available on the Internet at my own leisure and without interference from Comcast.

    Request is really the key word here, and going back to your Amazon analogy, delivery is covered by the consumer, and not the other way like you are trying to suggest. Taking this into account, I think I don't have to spell out how my monthly Comcast Internet subscription fits into this analogy.

    If Comcast feels that they don't want to absorb the capex of additional settlement-free interconnects, wouldn't it be more honest for them to admit that their oversubscription model is failing to scale in light of ever increasing services being delivered over the Internet?
  11. stretch 01 December 2010 19:36
    Ivan,

    As primarily a residential ISP (by far one of the largest in the US), it seems fair to assume that their subscribers will always consume considerably more content than they generate, as is the nature of end users. Does this mean Comcast should charge all other service providers for the privilege of delivering content to these millions of users, in addition to charging these millions of users for receiving that content?

    Yes, Level 3 relies on ISPs like Comcast to deliver content to end users. However, Comcast is equally reliant on Level 3 to provide their portion of Internet content.

    Of course, we're skirting the issue of whether Comcast wants Netflix (a direct competitor to Comcast's own video-on-demand service) on its network at all.
  12. Ivan Pepelnjak 01 December 2010 19:43
    You wouldn't believe, but we're in (almost) perfect agreement. However, before going forward, please do read the Comcast letter to the FCC (last link in my post)

    You wrote ... "From the consumer perspective, and that to me should really be the key focus, I could care less how network providers settle the exchange of information" ... absolutely agreed. Comcast never told their customers "you won't have XYZ", they just told Level-3 "if you want more pipes into our network (presumably to ensure quality of Netflix video), it's no longer free".

    You can still access any information you want, it will be delivered ... you'll most probably get a fair share of the bandwidth, but that might not be the bandwidth the application is expecting. If Netflix would be just another web app, they would sit somewhere in the middle of nowhere, using ISP XYZ and offer best-effort service with no real-time expectations to anyone around the globe, just like YouTube and Vimeo (at least those two look to me this way from where I'm sitting, I'm positive there's a lot of CDN gear involved).

    However, Level 3 wanted more bandwidth and Comcast said "not for free" ... and how is net neutrality entering the picture, apart from being misused by Level-3 spinsters?

    Last but not least, this time Comcast appears to be honest. They said "We can't absorb extra traffic for free" and now everyone is screaming at them. How does that make sense? Obviously you can't win.
  13. Ivan Pepelnjak 01 December 2010 19:46
    Stretch, it's a QoS problem: do you want your application delivered like any other application or do you want special treatment (extra interconnect pipes). If Level 3 would be ready to push the new content across existing links, I would have nothing to write about ... but of course Netflix users would not be happy and Netflix would have to change its CDN partner yet again ... maybe this time to someone more expensive who doesn't cut corners or game the system.
  14. Brian 01 December 2010 19:50
    Interesting post Ivan. Seems like we have some folks on this blog advocating a "free ride". I am going to leave the technology aside and rant a bit on the government intervention that is, in my opinion, at the core of much of this. Net Neutrality is a euphemism for government sponsored theft. I'd be hard pressed to come up with an instance where government regulation has increased rather than decreased innovation and creativity. This excess regulation is not necessary. There is no lack of competition. The intent is to replace the choices of consumers with government regulation as to how Internet traffic is to be managed. This is not a market driven model and if the scheme prevails we'll end up with a slower less efficient Internet. It's no wonder that most of the support for "Net Neutrality" comes from the content providers - Google, Amazon, eBay et al. As they add more high-bandwidth content they want to prevent the ISPs from being able to charge them for using a large percentage of available bandwidth. These content providers wants to use the government to prevent the ISPs from setting a market price on the bandwidth. This amount to government sponsored theft of the ISP's property. If this is allow to stand you end up with bureaucrats rather than the entrepreneurs and engineers determining how these networks will function.
  15. Bastiaan 01 December 2010 20:32
    "Let's go back to the basics. If Amazon wants to deliver you a book, they have to pay shipping charges, even though you (also a customer of US Post or whatever other postal service) have requested the book. Why should the digital world be any different?"

    Heh, that is a new twist of a flawed train of thought.
    I had not heard this one before.

    Level-3 is the postal service, netflix is amazon.
    The customer pays netflix for content and netflix pays level-3 to stream the content to the internet.
    You as an end user pays comcast for internet.

    Everything should pan out.
    However it starts going wrong when comcast wanta to earn money from all sides.
  16. Jason Tracy 01 December 2010 22:02
    Brian, your post would be correct if Comcast gave away Internet access for free. It might also be correct if Comcast didn't sell access to the Internet, but to their own Intranet.

    Again, Google, Netflix, eBay, etc, are NOT using Comcast's bandwidth. Comcast's CUSTOMERS are using Comcast's bandwidth.
  17. Brian 01 December 2010 22:40
    I believe the post is accurate, because the government is going to be in the position of determining the price for bandwidth. Comcast should be able to charge a market rate for the use of their infrastructure. Agreements between carriers for transit use are common practice with CDNs. You emphasize (NOT) that Google et al are able to operate in a vacuum and do not require the use of Comcast's bandwidth. I don't see how you can square that logic. Aren't Comcast's customers ipso facto Google's customers, Netflix's customers, eBay's customers? I'd say so to the tune of 16.7M potential subscribers. Those customers don't exist w/o Comcast's network. Comcast wants L3, provider for Netflix, to adhere to the same type of arrangement that it has with the other CDNs. L3, however, does not want to be subject to such an agreements and wants to have the gov't, via net "neutrality" set the price for Comcast's
  18. John 02 December 2010 04:25
    Are these 30 ports really at issue here? L3 wants more ports, so what? Dropping some additional 10G ports in a few *private* peering points can't possibly be that big a deal for a company the size of Comcast. Now, maybe Comcast's real problem is the capacity they have to the peering points. I still think that's a Comcast FAIL. Say for absurdity sake, they only brought in a T1 to their peering point. Is that L3's problem? Scale that up. When does CC's capacity issue stop being CC's problem and start being L3's?

    I think that Comcast is commiting douchebaggery for asking L3 to pay for the traffic that CC's own customers are asking L3 to feed to them. I also think that making meat-space analogies for these digital issues is fallacious. Your Amazon analogy is wrong because shipping is only paid once. Comcast is wanting to be paid for ingress and egress.

    Do you honestly think that Comcast will stop looking for revenue after they're done shaking down L3? If this proves successful, what's to stop them from demanding money from any other value-added peer (Google)? Then ATT will see there's money in it and so on. In the end, our pockets are the only ones lighter.
  19. stretch 02 December 2010 16:25
    Brian, do you mean something like the Carterfone decision in 1968? This was direct government regulation which allowed people to attach their own devices to the PSTN, spawning inventions like faxes and the dial-up modem. Network neutrality is not dissimilar.
  20. James Cape 03 December 2010 04:20
    Comcast makes a convincing case that Level3 is starting a new business as a CDN to deliver Netflix, and it's unfair they should carry this traffic to their customers.

    Unfortunately, Level3 and Ars Technica make (to my mind) a better case that since this isn't transit service, that's bullshit. Level3 just landed a deal to run 20% of the Internet. By comparison, if Level3 had 20% of the hosts on the internet, Comcast would be paying them to peer, not the other way around, and you would not be complaining.

    If Comcast weren't capable of handling it's customer demands to be 20% of the internet, then the proper response would be to tell Comcast to Evolve or Die. Now, it's kind of screwy that 20% of the Internet bandwidth can be used by *one* company, with *one* carrier, but that's the world we live in today.

    As it turns out, Level3's eyes were bigger than their stomachs and now Comcast is in a position to squeeze them, lest they lose the deal and go back to being just a Tier 1 on the verge of delisting.

    I appreciate the bind that Comcast finds itself in---the need to constantly upgrade it's user-facing service in order to one-up other competitors in the retail space vs. the need to constantly upgrade and rework their upstream network to keep up with the increased demand that the first need allows.

    But seriously, Comcast's business model problems are *Comcast's* business model problems.
  21. Gabriel 04 December 2010 15:46
    From what others (on nanog) say, Comcast was actually a customer of Level 3 (by buying transit) in which case the ratio argument doesn't really stand.
  22. Ivan Pepelnjak 04 December 2010 17:44
    In that case, the ratio argument is obviously bogus ... but you also don't go to your customer and tell them to upgrade the link, do you?
  23. Alexandra Stanovska 16 December 2010 23:52
    Hello Ivan,
    On Slashdot there was recently article about Comcast possibly violating their own rules about peering ratio 5:1 which they want to use as lever for negotiation with L3. Seems like they use more than 10:1 with Tata, which is where Netflix users currently go through.
    http://tech.slashdot.org/story/10/12/14/1335235/Comcast-Accused-of-Congestion-By-Choice
    Not that I am on one side or another, it's hard for me to judge with my knowledge and information, just posting it as a bit of information.
  24. Ivan Pepelnjak 17 December 2010 09:52
    Alexandra, thanks for the link! The graphs prove what we all know: there are no innocents in this particular spat.

    However, keep in mind that Comcast is a _customer_ of Tata, so the ratio on the links and the link utilization is nobody's (but Comcast's ... and their customers') business.

    The catch of the story is that Level3 tried to force Comcast to give them more bandwidth for free, citing peering relationship (which is a totally different beast).

    If Comcast is a customer of Level3 (which might be the case), Level3 cannot expect to have any influence on Comcast's link speeds or utilization - it's entirely up to Comcast to figure out what they need and what they're willing to pay for.

    On the other hand, if they have a zero-settlement peering agreement, there are explicit or implicit assumptions about traffic ratio (which have nothing to do with in/out ratios Comcast might have on the links it pays for as a customer) which Level3 tried to shift in their favor.
  25. Gabriel 18 December 2010 21:42
    In the meantime, Level 3 addressed a letter to the FCC. And Comcast responded [1].

    It seems Comcast keeps insisting this is a peering dispute (see first paragraph). And then they say there's a _paid_ peering agreement (I wonder how that differs from transit) between them and Level 3 since November 2010.

    I agree that Level 3 is in no position to impose Comcast to upgrade the interconnection links. I think that _Comcast_ should have asked for the upgrade (I don't know how the costs are divided in these situations). But they didn't and they run their links to Tata congested. And they refuse to upgrade other links as well (Global Crossing). But that's what (maybe unbiased) people from nanog say.

    If their goal is to force all CDNs to pay them to reach their (captive) customers, why not terminate de relationship to Level 3 and buy transit from another network? They run their transit links congested anyway, so that way, they force Level 3 (or at least the CDN part of it) to pay them (like the other CDNs do) or face the risk of losing Netflix as their client.

    [1] http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.html
  26. Gabriel 18 December 2010 21:44
    Forgot to add another link:

    http://www.xtranormal.com/watch/8124137/
Add comment
Home
Sidebar