Building Network Automation Solutions
6 week online course starting in September 2017

Do We Need QoS in the Data Center?

Whenever I get asked about QoS in the data center, my stock reply is “bandwidth is cheaper than QoS-induced complexity.” This is definitely true in most cases, and ideally the elephant problems should be solved higher up in the application stack, not with network-layer kludges, but are there situations where you actually need data center QoS?

Congestion detection and TCP ECN marking might be a good use case and can be done with minimal interface configuration – all it takes is a few configuration lines on Arista EOS and Cisco Nexus OS. Data Center TCP uses ECN markings to detect congestion and reduce transmission rate before packets get dropped (packets drops could result in not-so-insignificant performance degradation because they kick the NICs out of TCP offload mode).

There might be cases where you need QoS to reduce latency, but I don’t think VoIP qualifies. At 10Gbps speeds, you need 1 MB of packets sitting in the output queue to generate an additional millisecond of latency.

Finally, if you’re forced to implement queuing to reduce the impact of elephant flows (for example), insist on behaving like a service provider and keeping the network configuration as clean as possible – police ingress traffic (if needed) and queue packets based on DSCP or 802.1p markings. Application-aware processing (hopefully resulting in DSCP marking) belongs to hypervisors or the end-hosts, not to the ToR switch.

Anything else? Share your thoughts in the comments.

11 comments:

  1. I believe that FCoE (or any converged storage & networking solution, including iSCSI or whatever storage solution) is the paradigmatic example of QoS requirement in the DC. The Enhanced Transmission Selection (802.1Qaz) in Data Center Ethernet handles specifically this case by guaranteeing bandwidth reservations for different traffic classes. Also, since some storage solutions (FC) or different traffic types (VoIP) do not run over TCP, they also designed a congestion avoidance mechanism in layer-2 (802.1Qau, congestion notification).

    ReplyDelete
    Replies
    1. Maybe you should start with these blog posts:

      http://blog.ipspace.net/2010/09/introduction-to-8021qaz-enhanced.html
      http://blog.ipspace.net/2010/10/pfcets-and-storage-traffic-real-story.html
      http://blog.ipspace.net/2013/07/iscsi-with-pfc.html
      http://blog.ipspace.net/2010/11/does-fcoe-need-qcn-8021qau.html

      Delete
  2. Good post Ivan. This discussion keeps coming up. Equipment vendors keep talking up their DC QoS features, but I don't see need to QoS in DC unless there is distributed storage....Would it make sense to create a separate network for IP/distributed storage with Ethernet being so cheap?

    ReplyDelete
    Replies
    1. "Would it make sense to create a separate network for IP/distributed storage with Ethernet being so cheap?" - absolutely. A lot of people (including Amazon) are doing that.

      Delete
    2. Does it make sense have separate fastpath for elephant as suggested in http://networkheresy.com/2013/11/01/of-mice-and-elephants/ ?

      Delete
  3. Not sure if applicable to DC, but the author of http://habrahabr.ru/post/246791/ (in russian) makes a case from SP perspective: there is a problem of microbursts and oversubscription in thei network.

    ReplyDelete
    Replies
    1. Exactly this (after translating it). Microbursts require QoS. There are plenty of applications, especially in the data center, that assume a lossless environment and will not act well without QoS, even on 10 GE. Luckily the policy usually doesn't have to be complex: just have a lossless queue somewhere.

      Delete
  4. Also see that with NFV vCPE use case, SP customers are looking for QoS features needed on End VNFs acting as vCPE (not end applications). Is QoS applicable in this scenario??

    ReplyDelete
    Replies
    1. http://m.youtube.com/watch?v=wXlvy3sTTBk

      Delete
  5. Depends of the DC.. For example, if we are talking about DC with VDI, there is no big difference from office network. And QoS should be implemented.
    Also, as it was mentioned here - microburst can added some small issues time to time.

    ReplyDelete
    Replies
    1. "Depends of the DC" << In principle of course I have to agree

      "if we are talking about DC with VDI" << hope you'll have 10GE links to the servers and 40GE or 100GE uplinks. How much traffic does a VDI session generate?

      Delete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.