P2P traffic is bad for the network

I’m positive you all know that. I also hope that you’re making sure it’s not hogging your enterprise network (if you’re managing one); in case you have to detect P2P traffic in your enterprise network, read the Catching the torrent guy post (or get in touch with our NIL Monitor guys if you need remote monitoring help ... the e-mail address is hidden at the bottom of a long marketese document).

Service Providers are not so fortunate – some Internet users claim using unlimited amounts of P2P traffic is their birthright. I don’t really care what kind of content these users transfer, they are consuming enormous amounts of network resources due to a combination of P2P client behavior (which is clearly “optimized” to grab as much as possible) and the default TCP/QoS interaction.

Let’s start with the P2P behavior:

  • P2P implementations (for example, BitTorrent clients) open a large amount of simultaneous TCP sessions. 200 concurrent sessions is a very “reasonable” number and some people claim that the 10 new sessions per second limit imposed by Windows XP SP2 is reducing their speed ... now you know how many sessions they tend to open.
  • P2P client can saturate any link for a very long time. I’m a heavy Internet user, but I still use around 1% of my access speed (long-term average). A P2P client can bring the long-term average close to 100%.

I wouldn’t mind the reckless implementations of P2P clients if the Internet would be an infrastructure where every user gets its fair share of bandwidth. Unfortunately, the idealistic design of the early Internet ensures that (using default router settings) every TCP session gets the same amount of bandwidth. A P2P user with 200 concurrent sessions thus gets 200 times the bandwidth of another user downloading her e-mail with a POP3 session. Clearly not a win-win situation (for anyone but the P2P guy) that could easily result in “a few” annoyed calls to the SP helpdesk.

What we would need to cope with the P2P users is per-user (per-IP-address) queuing, which is not available on any router that I’m familiar with (let alone on any high-speed platform). If you have other information, please share it in the comments.

The best solution I’m aware of is a dedicated box like Cisco’s SCE that can perform per-user measuring with real-time actions. Unfortunately, we cannot deploy SCE at every bottleneck in the network; it’s usually deployed somewhere between the access and core network with plenty of bandwidth surrounding it.

To address the P2P challenge with SCE, you could define very smart bandwidth groups or you could use SCE to track per-user quotas (actually per-IP-address quotas) and to act once a user exceeds her quota – for example, by hard-limiting the user’s bandwidth or remarking her packets with a low-priority DSCP value which can then be used to select a different (low-priority) queue on congested links.

Are you curious whether SCE can help you solve your problems? Why don’t you get in touch with our Professional Services team; these guys have deployed SCE-based solutions in over 20 large SP networks worldwide. Alternatively, you might want to attend our SCE training.

29 comments:

  1. Dmitri Kalintsev26 July, 2010 08:15

    Hi Ivan,

    Have a look at the following (apologies if you've known all that, but to me it didn't appear to have come through in your post):

    1) http://www.zdnet.com/blog/storage/p4p-faster-smarter-p2p/303

    2) http://en.wikipedia.org/wiki/P2P_caching

    3) http://blog.bittorrent.com/2009/10/05/changing-the-game-with-%CE%BCtp/

    ReplyDelete
  2. Dmitri Kalintsev26 July, 2010 08:16

    Forgot to subscribe to the replies :)

    ReplyDelete
  3. Shill for the Internet monopolies.

    ReplyDelete
  4. To be honest, much of the hoopla about P2P sounds like a bunch of Enterprise Network engineers failing to grasp that the model ISPs have built up is simply not sustainable using existing methods (and technologies)... and nobody in telecom has any vision about how to let paying customers do what they are trying to do.

    ReplyDelete
  5. While I agree about torrents being bad in corporate environment (people.supposed to work there) SPs should deal with any amount of bandwidth they sell, even.if user tends to occupy it 100% 24/7.

    ReplyDelete
  6. Ivan Pepelnjak26 July, 2010 13:34

    It's not "a bunch of Enterprise Network engineers", it's a (quite large & vocal) bunch of users who want to do whatever they wish and who believe nobody should touch their traffic.

    As for the telecom industry: they are in a vise into which they put themselves.

    ReplyDelete
  7. Ivan Pepelnjak26 July, 2010 13:35

    Absolutely in agreement. However, as I pointed out, P2P traffic is bad from the perspective that it cannot be coped with using existing QoS mechanisms.

    ReplyDelete
  8. Ivan Pepelnjak26 July, 2010 13:39

    Thanks for the links. Very interesting (as always).

    P4P solves the long-haul congestion and might improve the situation for hub-and-spoke access networks. Supposedly it makes the situation worse on cable (shared media) networks, because it's easier to saturate the cable with locally-sourced traffic.

    P2P caching is an interesting idea. It would be interesting to see how big a cache you need to have reasonable improvements.

    uTP - need to study the details (assuming they are published anywhere). As long as it runs on TCP, it's only a marginal improvement. TCP should back off automatically when experiencing delays; the problem is the large discrepancy between the number of TCP sessions created by a P2P user versus the number of sessions a regular web/e-mail user has opened (and their duration).

    ReplyDelete
  9. There is no such a thing as "bad" or "good" traffic. When you call any traffic "bad", you actually stops being an engineer and starts you existence as a marketing guy selling your seminars. Traffic can be either "in contract" or "out-of-contract". And there is a simple rule - don't sell a contract which you cannot fulfil. But human's greediness is pushing people here and there to violate this simple rule. They are inevitably punished, whether the punishment comes in a form of P2P application (for telecoms) or in a form of George Soros (for the Bank of England).

    ReplyDelete
  10. What if p2p users will be smart enough to protect traffic with ipsec? Mark ALL ipsec traffic (including business VPN) as "bad"? Prohibit ipsec entirely? Require to subscribe "enterprise/premium" plans? (AT&T do it)

    ReplyDelete
  11. I'm pretty sure the engineers at Facebook and Twitter would disagree with your thesis.

    http://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/

    ReplyDelete
  12. Ivan Pepelnjak26 July, 2010 18:15

    OK, let's get into hair-splitting mode: I never used the "bad traffic" term. Please don't make it my problem if you're upset by people claiming P2P is bad ;) I just explained the effects current P2P implementations might have an average ISP network ... and the effects are bad, pure and simple.

    Next, the whole reason Internet works as well as it does is the presumed cooperative behavior of its users that allowed designers to bypass numerous checks-and-balances that burdened the traditional telco technologies. If we can no longer rely on cooperative behavior, the costs of the Internet infrastructure will go up, as those checks will have to be implemented.

    As I wrote (I hope you did consider my technical arguments), the solution is per-IP-address queuing, which is not commonly implemented.

    Last but not least, the root cause of all the problems we're discussing is that THERE IS NO CONTRACT. The only parameter promised by an ISP is the access speed to the first router (and in a cable network, not even that). Everything else is implicitly assumed by one or the other party.

    ReplyDelete
  13. Ivan Pepelnjak26 July, 2010 18:17

    First of all, I'm not saying ISP should block P2P traffic. On the contrary, I'm against it. I'm just pointing out that with the current QoS mechanisms P2P clients are allowed to grab disproportionate amounts of shared goods (bandwidth) and that in my opinion per-IP-address queuing is the only solution.

    As soon as we have per-user or per-IP-address queuing or bandwidth allocation, P2P would stop being a problem and IPSec would not help you a bit.

    ReplyDelete
  14. Ivan Pepelnjak26 July, 2010 18:24

    Thanks for an excellent link. Using BT to distribute large amount of files to a large number of distributed servers is a marvelous use of the technology.

    I'm also positive (and I am sure you'd agree with me) that Facebook or Twitter would not allow their BitTorrent traffic to run rampant squeezing out all other traffic from their network.

    They have at least two options to control BT bandwidth utilization: It's quite easy to tightly control BT clients. You can specify # of TCP sessions (per-torrent or globally), per-torrent bandwidth, total bandwidth ... and you can mark desired BT traffic in a private network and queue it accordingly.

    However, while I absolutely enjoy the information you've shared, I fail to see how it's relevant to our discussion. It's like saying viruses are good because some retroviruses are used as DNA carriers in genetic engineering.

    ReplyDelete
  15. Just pointing out that you may want to qualify your thesis. Peer-to-peer protocols are not inherently "bad for the network."

    ReplyDelete
  16. Ivan Pepelnjak26 July, 2010 18:44

    Hair-splitting mode ON.

    I was not speaking about the protocols, but about the uncontrolled use of these protocols in an environment that was designed for a cooperative use.

    ReplyDelete
  17. Exactly. ISPs complaining about users using the traffic that they pay for. Few years ago there was a proposal from Anja Feldmann in Berlin about a monster called "Oracle" to deal with this stuff.

    http://www.net.t-labs.tu-berlin.de/research/isp-p2p/

    Killed instantly from the audience, not to mention.

    ReplyDelete
  18. Yes, "bad" traffic should set the evil bit on :)

    http://www.faqs.org/rfcs/rfc3514.html

    ReplyDelete
  19. this have a spike and a long tail, specially if we know, that p2p traffic is slowly moving to IPv6 protocol. Latest windows hosts have teredo on by default, so cache/catch that :)

    ReplyDelete
  20. subscribing to the stream :)

    ReplyDelete
  21. Ivan Pepelnjak26 July, 2010 19:18

    Use QuantumFlow processors on ASR to set it:

    http://blog.ioshints.info/2008/04/rfc-3514-implemented-by-asr-series-of.html

    ReplyDelete
  22. Ivan Pepelnjak26 July, 2010 19:20

    No change. If you do per-session QoS, you're fried, if you manage to do per-address QoS, it's the same as with IPv4.

    Now, here's a good question: does SCE support IPv6 ... not sure :-P, need to ask our gurus.

    ReplyDelete
  23. Exactly :) I wanted to ask that question, but I was sure you'll come to that :)

    Salute from IEF78 @Maastricht.

    ReplyDelete
  24. Petr Lapukhov26 July, 2010 20:00

    The root cause of P2P QoS problem is the flow-fairness and congestion-adaption model that has been in the Internet since the very first days. However, if you try to think of a more suitable fairness model you'll quickly run in the problems found by the game theory and the theory of public welness.

    How would you define fairness (e.g. egalitarian/utilitarian approaches) among ISP customers? What scale do you define fairness at (link, aggregation point, ISP, multiple-ISPs, Internet)? Is it possible to implement fairness at all, when ISP oversubscribe their networks and offer "unlimited" contracts to customers, based on statistical assumptions which are never satisfied asymptotically?

    The above questions are fundamental to any statistical multiplexing network that uses oversubscription to realize better value. The intuitive fairness is broken at the very beginning! :) All proposed solutions (e.g. changing flow mask) may only help to alleviate, but not solve the problem, as no one knows how fairness should be truly implemented in the deceptive world of ISP services :).

    Here is an example: let's say you aggregate flows based on customer IP address to deafeat single hosts sourcing barrage of flows. With the flow-fairness model it will give an enterprise customer with more hosts advantage over the customer having less hosts in their networks. It's the same problem, just different scale. The flow-fairness simple does not work as it should be. Thus, any solution within flow-fairness that works at the small scale may not work at a larger scale.

    Finally, I would recommend reading the following draft documet:

    http://bobbriscoe.net/projects/refb/draft-briscoe-tsvarea-fair-02.html

    It offers interesting discussion and some rather novel ideas for fairness management.

    ReplyDelete
  25. Dmitri Kalintsev27 July, 2010 00:19

    Ivan,

    Here's the uTP protocol description for your enjoyment: http://bittorrent.org/beps/bep_0029.html . No, it does not use TCP. :)

    Regarding the cache size for P2P - I think you might be surprised that you don't need terribly large storage to get high hit ratios. There is a tendency to download stuff that is "popular" at a given time, and a few terabytes of storage (well below 10) will easily take care of that. The reason behind this is *why* people download certain content - it is most often because their friends are talking about it. Particular movie, show or music album. The rest of the available content lives in relative obscurity.

    Also, most P2P caching solutions I've heard about provide mechanisms for controlling the download rate without the need to use DPI solutions.

    And, of course, there are signature-based DPI solutions, which can enforce QoS policies directly, without the need to cache anything.

    ReplyDelete
  26. yournamegoeshere27 July, 2010 01:56

    I'm always surprised by service provider-minded people blaming the users for this problem.

    As Ivan has pointed out in a previous comment, there is no contract other than an implicit access rate to the first hop router.

    Maybe there should be one? Blaming the users for their interpretation of the marketing materials is crazy. Users have no idea what the network is actually capable of, or what's expected of them. All they know is what's printed in 10m tall print on a billboard.

    Blame your marketing department for this debacle, not your users. :-P

    ReplyDelete
  27. Thanks for illustrating my point succinctly. Blaming customers for trying to do use what they're paying for in ways that a core chunk of the provider business model doesn't take into account.

    It's an attitude that reeks of large, top-down enterprise, where IT is in charge and fucking up the network will get you fired...as opposed to a competitive market where you have to innovate and serve your customers, or you will go out of business.

    I'm saying the network sucks because it can't let people do what they want to do. Users *should* be able to do whatever they damned well please, and if the technology doesn't exist to let them, then that's a tremendous opportunity for engineers to tell the "hey, I help allow [this torrenty-like-thing we all use and love now]" story 20 years from now, and businesses to make a shitload of cash in the meantime.

    Perhaps also a way for the IETF to atone for PIM ;-)

    ReplyDelete
  28. Ivan Pepelnjak27 July, 2010 18:01

    James, the technology is there and it's very simple:

    * fine-tune your BitTorrent settings to consume 1/10 to 1/4 of your access speed and limit the number of TCP connections to 50-100.
    * implement the same fine-tuning mechanism automatically in BitTorrent client;
    * use BitTorrent client with uTP (supposedly it works better than BT over TCP, need to study it a bit)
    * accept that BT traffic is classified as lower-priority traffic. Not rate-limited, not throttled, not dropped, just easily recognizable and lower priority. As BT represents 50-60% of Internet traffic, giving the other traffic priority will not hurt it badly.

    These modifications would make [this torrenty-thing we all use] more lovable without a major sacrifice. Unfortunately, the argument has left the technical waters years ago and became political (best case) or (usually) religious.

    Of course some SPs (particularly those that love ridiculously low usage caps) would still try all sorts of tricks to raise their ARPU, but the market will eventually deal with them.

    ReplyDelete
  29. "who believe nobody should touch their traffic."

    I am one of them. I work as a network security admin - but when I am at home I am a paying customer.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.