IOS Queuing 101

A while ago I've been a co-mentor to a graduate student who investigated the difference between shaping and policing for his thesis. While doing the lab tests, he stumbled across an interesting problem: he could not configure fancy queuing (for example, CB-WFQ) on a point-to-point Frame Relay subinterface. The solution was to configure a QoS hierarchy, doing shaping first and queuing within the shaping queues.

A few days ago, Arden Packeer covered the same topic in one of his blog posts, so it looks like the underlying problem (and the solution) is not as widely known as I had assumed. Time to write a tutorial on Queuing Principles in Cisco IOS.

5 comments:

  1. The size of the hardware queue can be configured with the tx-ring-limit interface configuration command and inspected with the show controllers interface command (look for the tx_limit field or the TX ring size). This setting should be changed only if the long hardware queue size causes unacceptable jitter or delay on QoS-sensitive applications (for example, VoIP traffic).

    --

    Here's the trade off:

    A longer rx/tx_ring means that bursty traffic will be queued up, for busy interfaces, a larger ring size is desirable (generally). The problem is that under LOW traffic conditions, a large ring size is bad for jitter and latency. An interrupt should be triggered by one of two conditions, whichever happens first:

    1) A certain fraction of the ring is full. Generally, an interrupt gets triggered when 1/4-1/3 of the ring is full.

    2) A timeout is hit, and traffic is flushed

    On a low load interface, the timeout will usually be hit, while a heavily utilized link will hit the fraction first. So the busy interface keeps sending those interrupts and flushing out part of the buffer.

    If you compare 2 systems, heavily utilized and low utilization, you'll see dramatic differences in the latency - up to ~4 times! And the shocker for many people is that the latency increases, the lower the traffic levels.

    Your trade off is latency vs ability to handle traffic bursts. Tuning these values is not recommended unless you're taking serious hits.

    ReplyDelete
  2. Hello guys,

    I have one question regarding policer.

    Suppose i've set a policer to 5Mbps.

    One of my customer, is pinging with MTU size of 1500 repeat of 1000 for example. An he sees packet drop.

    Of course i told him, our policer is dropping the packet.

    My question is, how do i calculate the exact amount of bandwidth the ping is utilize. So that i can pint point for every reply you will have this amount of timeout since the policer will discard any traffic that goes beyond the 5 Mbps


    Regards
    Wan T

    ReplyDelete
  3. @Anonymous#1: Thanks for the excellent explanation.

    @Wan (aka Anonymous #2): You cannot generate 5 Mbps of traffic with ping. Ping is a request-response protocol, the second packet is only sent after the first reply has been received (or after the timeout expires).

    To calculate the approximate bandwidth used by ping, do the following:

    pps = 1000 / average-ping-response[in msec]
    bw[bps] = pps * (ping-packet-size +24) * 8

    ReplyDelete
  4. I have a problem that is related to.
    I have traffic-shaping in the main interface and a LL/QCB-WFQ service-policy inside the frame-relay map. It works fine for some interfaces. However, other interfaces never put voice packets inside the high queue od the dual-fifo interface.
    Looking the interfaces, I discovered that interfaces that work fine presents matching packets at L3 queue. Then, they go to the high queue.

    Serial1/2 voice never goes to high dual-fifo, get stucked with data
    fragments in the output queue.

    Serial30 voice goes to the high queue.

    See that L3 queueing of s1/2 has no match, even with matches at ip dscp EF shows more than 1 million of packets...

    :(


    look the print (sorry, but it is long...)

    mac-pn-230b-dr002#sh int s1/2

    Serial1/2 is up, line protocol is up

    Hardware is M4T

    Description: ==To Noble Muravenko==

    MTU 1500 bytes, BW 192 Kbit, DLY 20000 usec,

    reliability 255/255, txload 245/255, rxload 46/255

    Encapsulation FRAME-RELAY, crc 16, loopback not set

    Keepalive set (10 sec)

    Restart-Delay is 0 secs

    LMI enq sent 0, LMI stat recvd 0, LMI upd recvd 0

    LMI enq recvd 6123, LMI stat sent 6123, LMI upd sent 0, DCE LMI up

    LMI DLCI 1023 LMI type is CISCO frame relay DCE

    FR SVC disabled, LAPF state down

    Broadcast queue 0/64, broadcasts sent/dropped 8165/0, interface broadcasts 6123

    Last input 00:00:05, output 00:00:00, output hang never

    Last clearing of "show interface" counters 17:00:49

    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4875

    Queueing strategy: dual fifo

    Output queue: high size/max/dropped 0/256/0

    Output queue: 10/128 (size/max)

    30 second input rate 35000 bits/sec, 59 packets/sec

    30 second output rate 185000 bits/sec, 140 packets/sec

    3639099 packets input, 253448953 bytes, 0 no buffer

    Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

    312 input errors, 201 CRC, 0 frame, 0 overrun, 0 ignored, 111 abort

    4659821 packets output, 597264717 bytes, 0 underruns

    0 output errors, 0 collisions, 0 interface resets

    0 output buffer failures, 0 output buffers swapped out

    2 carrier transitions DCD=up DSR=down DTR=up RTS=up CTS=down



    mac-pn-230b-dr002#sh queue s1/2

    Output queue for Serial1/2 is 12/128



    Packet 1, linktype: fr-fragment, length: 246, flags: 0x8

    DLCI: 300



    Packet 2, linktype: fr-fragment, length: 246, flags: 0x8

    DLCI: 300



    Packet 3, linktype: compressed-rtp, length: 19, flags: 0x88, CID: 1

    source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,

    TOS: 184 prot: 17, source port 17878, destination port 17218



    Packet 4, linktype: fr-fragment, length: 246, flags: 0x8

    DLCI: 300



    Packet 5, linktype: fr-fragment, length: 246, flags: 0x8

    DLCI: 300



    Packet 6, linktype: compressed-rtp, length: 29, flags: 0x88, CID: 1

    source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,

    TOS: 184 prot: 17, source port 17878, destination port 17218



    Packet 7, linktype: fr-fragment, length: 246, flags: 0x8

    DLCI: 300



    Packet 8, linktype: fr-fragment, length: 48, flags: 0x10000088

    DLCI: 300



    Packet 9, linktype: fr-fragment, length: 246, flags: 0x8

    DLCI: 300



    Packet 10, linktype: fr-fragment, length: 246, flags: 0x8

    DLCI: 300



    Packet 11, linktype: compressed-rtp, length: 28, flags: 0x88, CID: 1

    source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,

    TOS: 184 prot: 17, source port 17878, destination port 17218



    Packet 12, linktype: compressed-rtp, length: 26, flags: 0x88, CID: 1

    source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,

    TOS: 184 prot: 17, source port 17878, destination port 17218



    mac-pn-230b-dr002#

    mac-pn-230b-dr002#sh fram pvc int s1/2 100



    PVC Statistics for interface Serial1/2 (Frame Relay DCE)



    DLCI = 100, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial1/2.100



    input pkts 2030261 output pkts 1607788 in bytes 50304341

    out bytes 40998431 dropped pkts 0 in pkts dropped 0

    out pkts dropped 0 out bytes dropped 0

    in FECN pkts 0 in BECN pkts 0 out FECN pkts 0

    out BECN pkts 0 in DE pkts 0 out DE pkts 0

    out bcast pkts 7146 out bcast bytes 839182

    30 second input rate 9000 bits/sec, 42 packets/sec

    30 second output rate 4000 bits/sec, 23 packets/sec

    pvc create time 29w2d, last time pvc status changed 06:03:00

    fragment type end-to-end fragment size 240

    cir 64000 bc 640 be 0 limit 80 interval 10

    mincir 64000 byte increment 80 BECN response no IF_CONG no

    frags 56797 bytes 6354709 frags delayed 5464 bytes delayed 797186

    shaping inactive

    traffic shaping drops 0

    service policy CPRK-VoIP

    Serial1/2.100: DLCI 100 -



    Service-policy output: CPRK-VoIP



    Class-map: RTP (match-any)

    1552390 packets, 92206236 bytes

    30 second offered rate 10000 bps, drop rate 0 bps

    Match: ip dscp ef

    1552390 packets, 92206236 bytes

    30 second rate 10000 bps

    Match: dscp ef

    0 packets, 0 bytes

    30 second rate 0 bps

    Match: protocol rtp audio

    0 packets, 0 bytes

    30 second rate 0 bps

    Match: ip rtp 16384 16383

    0 packets, 0 bytes

    30 second rate 0 bps

    QoS Set

    dscp ef

    Packets marked 1552390

    Queueing

    Strict Priority

    Output Queue: Conversation 24

    Bandwidth 90 (%)

    Bandwidth 57 (kbps) Burst 1425 (Bytes)

    (pkts matched/bytes matched) 0/0

    (total drops/bytes drops) 0/0



    Class-map: H323 (match-any)

    18883 packets, 2473464 bytes

    30 second offered rate 0 bps, drop rate 0 bps

    Match: ip dscp af31

    18883 packets, 2473464 bytes

    30 second rate 0 bps

    Queueing

    Output Queue: Conversation 25

    Bandwidth 5 (%)

    Bandwidth 3 (kbps) Max Threshold 64 (packets)

    (pkts matched/bytes matched) 18883/2473464

    (depth/total drops/no-buffer drops) 0/0/0



    Class-map: class-default (match-any)

    36515 packets, 3867255 bytes

    30 second offered rate 0 bps, drop rate 0 bps

    Match: any

    Queueing

    Flow Based Fair Queueing

    Maximum Number of Hashed Queues 16

    (total queued/total drops/no-buffer drops) 0/0/0

    Output queue size 0/max total 600/drops 0

    mac-pn-230b-dr002#

    mac-pn-230b-dr002#sh fram pvc int s1/2 300



    PVC Statistics for interface Serial1/2 (Frame Relay DCE)



    DLCI = 300, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial1/2.300



    input pkts 1160692 output pkts 1181504 in bytes 199842592

    out bytes 556850656 dropped pkts 107 in pkts dropped 107

    out pkts dropped 4875 out bytes dropped 972170

    late-dropped out pkts 4875 late-dropped out bytes 972170

    in FECN pkts 0 in BECN pkts 0 out FECN pkts 0

    out BECN pkts 0 in DE pkts 0 out DE pkts 0

    out bcast pkts 1021 out bcast bytes 349182

    30 second input rate 74000 bits/sec, 31 packets/sec

    30 second output rate 178000 bits/sec, 29 packets/sec

    pvc create time 29w2d, last time pvc status changed 06:03:05

    fragment type end-to-end fragment size 240

    cir 192000 bc 1920 be 0 limit 240 interval 10

    mincir 128000 byte increment 240 BECN response no IF_CONG yes

    frags 3048008 bytes 556787812 frags delayed 2667707 bytes delayed 521876100

    shaping active

    traffic shaping drops 4875

    service policy Noble

    Serial1/2.300: DLCI 300 -



    Service-policy output: Noble



    Class-map: NobleSAP (match-any)

    2478 packets, 310198 bytes

    30 second offered rate 0 bps, drop rate 0 bps

    Match: access-group 111

    2478 packets, 310198 bytes

    30 second rate 0 bps

    Queueing

    Strict Priority

    Output Queue: Conversation 24

    Bandwidth 96 (kbps) Burst 2400 (Bytes)

    (pkts matched/bytes matched) 2478/310198

    (total drops/bytes drops) 0/0



    Class-map: class-default (match-any)

    1183901 packets, 544459264 bytes

    30 second offered rate 166000 bps, drop rate 0 bps

    Match: any

    Queueing

    Flow Based Fair Queueing

    Maximum Number of Hashed Queues 16

    (total queued/total drops/no-buffer drops) 68/4875/0

    exponential weight: 9



    dscp Transmitted Random drop Tail drop Minimum Maximum Mark

    pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

    af11 15/4185 0/0 0/0 32 40 1/10

    af12 3/136 0/0 0/0 28 40 1/10

    af13 4/200 0/0 0/0 24 40 1/10

    af21 7/350 0/0 0/0 32 40 1/10

    af22 57/2960 0/0 0/0 28 40 1/10

    af23 0/0 0/0 0/0 24 40 1/10

    af31 7/344 0/0 0/0 32 40 1/10

    af32 0/0 0/0 0/0 28 40 1/10

    af33 0/0 0/0 0/0 24 40 1/10

    af41 0/0 0/0 0/0 32 40 1/10

    af42 0/0 0/0 0/0 28 40 1/10

    af43 0/0 0/0 0/0 24 40 1/10

    cs1 1384/509353 1/64 2/128 22 40 1/10

    cs2 34/2922 0/0 0/0 24 40 1/10

    cs3 7/358 0/0 0/0 26 40 1/10

    cs4 36/4963 0/0 0/0 28 40 1/10

    cs5 37/37755 0/0 0/0 30 40 1/10

    cs6 7/390 0/0 0/0 32 40 1/10

    cs7 109/6874 0/0 0/0 34 40 1/10

    ef 1/58 0/0 0/0 36 40 1/10

    rsvp 0/0 0/0 0/0 36 40 1/10

    default 1177331/542923501 1153/491036 3719/480942 20 40 1/10



    Output queue size 67/max total 600/drops 4875

    mac-pn-230b-dr002#

    mac-pn-230b-dr002#sh fram pvc int s3/0 100



    PVC Statistics for interface Serial3/0 (Frame Relay DCE)



    DLCI = 100, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial3/0.100



    input pkts 1362 output pkts 806 in bytes 40669

    out bytes 26542 dropped pkts 0 in pkts dropped 0

    out pkts dropped 0 out bytes dropped 0

    in FECN pkts 0 in BECN pkts 0 out FECN pkts 0

    out BECN pkts 0 in DE pkts 0 out DE pkts 0

    out bcast pkts 26 out bcast bytes 3128

    5 minute input rate 6000 bits/sec, 6 packets/sec

    5 minute output rate 4000 bits/sec, 6 packets/sec

    pvc create time 29w2d, last time pvc status changed 2d16h

    fragment type end-to-end fragment size 320

    cir 256000 bc 2560 be 0 limit 320 interval 10

    mincir 64000 byte increment 320 BECN response no IF_CONG no

    frags 828 bytes 26870 frags delayed 8 bytes delayed 1408

    shaping inactive

    traffic shaping drops 0

    service policy Austral-VoIP

    Serial3/0.100: DLCI 100 -



    Service-policy output: Austral-VoIP



    Class-map: RTP (match-any)

    708 packets, 40028 bytes

    30 second offered rate 8000 bps, drop rate 0 bps

    Match: ip dscp ef

    708 packets, 40028 bytes

    30 second rate 8000 bps

    Match: dscp ef

    0 packets, 0 bytes

    30 second rate 0 bps

    Match: protocol rtp audio

    0 packets, 0 bytes

    30 second rate 0 bps

    Match: ip rtp 16384 16383

    0 packets, 0 bytes

    30 second rate 0 bps

    QoS Set

    dscp ef

    Packets marked 708

    Queueing

    Strict Priority

    Output Queue: Conversation 24

    Bandwidth 40 (kbps) Burst 1000 (Bytes)

    (pkts matched/bytes matched) 708/13721

    (total drops/bytes drops) 0/0



    mac-pn-230b-dr002#sh fram pvc int s3/0 100



    PVC Statistics for interface Serial3/0 (Frame Relay DCE)



    DLCI = 100, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial3/0.100



    input pkts 1467 output pkts 914 in bytes 43405

    out bytes 29009 dropped pkts 0 in pkts dropped 0

    out pkts dropped 0 out bytes dropped 0

    in FECN pkts 0 in BECN pkts 0 out FECN pkts 0

    out BECN pkts 0 in DE pkts 0 out DE pkts 0

    out bcast pkts 26 out bcast bytes 3128

    5 minute input rate 7000 bits/sec, 7 packets/sec

    5 minute output rate 4000 bits/sec, 7 packets/sec

    pvc create time 29w2d, last time pvc status changed 2d16h

    fragment type end-to-end fragment size 320

    cir 256000 bc 2560 be 0 limit 320 interval 10

    mincir 64000 byte increment 320 BECN response no IF_CONG no

    frags 922 bytes 29113 frags delayed 8 bytes delayed 1408

    shaping inactive

    traffic shaping drops 0

    service policy Austral-VoIP

    Serial3/0.100: DLCI 100 -



    Service-policy output: Austral-VoIP



    Class-map: RTP (match-any)

    801 packets, 45642 bytes

    30 second offered rate 8000 bps, drop rate 0 bps

    Match: ip dscp ef

    801 packets, 45642 bytes

    30 second rate 8000 bps

    Match: dscp ef

    0 packets, 0 bytes

    30 second rate 0 bps

    Match: protocol rtp audio

    0 packets, 0 bytes

    30 second rate 0 bps

    Match: ip rtp 16384 16383

    0 packets, 0 bytes

    30 second rate 0 bps

    QoS Set

    dscp ef

    Packets marked 802

    Queueing

    Strict Priority

    Output Queue: Conversation 24

    Bandwidth 40 (kbps) Burst 1000 (Bytes)

    (pkts matched/bytes matched) 802/15870

    (total drops/bytes drops) 0/0



    mac-pn-230b-dr002#sh queue s3/0

    Output queue for Serial3/0 is 10/128



    Packet 1, linktype: ip, length: 843, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x96FD, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x0333 0x33D8 0x4A00 0x1401 0x7400

    0x0100 0x0001 0x424B 0xB4FD 0x3F2B 0xB7FC 0xE19C



    Packet 2, linktype: ip, length: 202, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x96FE, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x00B2 0x7262 0x4A00 0x1401 0x7500

    0x0100 0x0001 0x4267 0x020F 0x0FE9 0xFFFD 0xA34D



    Packet 3, linktype: ip, length: 45, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x96FF, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x0015 0xE30F 0x4460 0x0001 0x0000

    0x0000 0x0001 0xE801 0x0081 0xC04E 0x544C 0x4D53



    Packet 4, linktype: ip, length: 45, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x9700, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x0015 0xE30F 0x4460 0x0001 0x0000

    0x0000 0x0001 0xE801 0x0053 0x004B 0x0041 0x004E



    Packet 5, linktype: ip, length: 202, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x9701, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x00B2 0x9DD1 0x4A00 0x1401 0x7600

    0x0100 0x0001 0x4277 0xB01E 0x8602 0xF7FE 0x6258



    Packet 6, linktype: ip, length: 1344, flags: 0x10000888

    source: 68.142.101.254, destination: 201.23.192.234, id: 0x79E9, ttl: 52,

    TOS: 0 prot: 6, source port 80, destination port 1923

    data: 0x0050 0x0783 0x3F2A 0xFD33 0xB4AD 0x5AC6 0x5010

    0xFFFF 0xC4DC 0x0000 0xA747 0x82D4 0x71C7 0x124D



    Packet 7, linktype: ip, length: 202, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x9702, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x00B2 0xB9F4 0x4A00 0x1401 0x7700

    0x0100 0x0001 0x4256 0x04B5 0x9901 0xB0FF 0xF4B7



    Packet 8, linktype: ip, length: 202, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x9703, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x00B2 0xF037 0x4A00 0x1401 0x7800

    0x0100 0x0001 0x4276 0x3089 0xC2A9 0xBFFB 0xDA6D



    Packet 9, linktype: ip, length: 202, flags: 0x10000888

    source: 201.9.13.87, destination: 201.23.192.234, id: 0x9704, ttl: 117,

    TOS: 0 prot: 17, source port 58548, destination port 43832

    data: 0xE4B4 0xAB38 0x00B2 0x469A 0x4A00 0x1401 0x7900

    0x0100 0x0001 0x424B 0xB08F 0x8603 0x7FE3 0xF6A0



    Packet 10, linktype: ip, length: 1344, flags: 0x10000888

    source: 68.142.101.254, destination: 201.23.192.234, id: 0x7C60, ttl: 52,

    TOS: 0 prot: 6, source port 80, destination port 1923

    data: 0x0050 0x0783 0x3F2B 0x0247 0xB4AD 0x5AC6 0x5010

    0xFFFF 0xE26A 0x0000 0x21FA 0x0000 0x0000 0x2FFF



    mac-pn-230b-dr002#

    mac-pn-230b-dr002#

    mac-pn-230b-dr002#sh queue s3/0 0



    mac-pn-230b-dr002#sh queue s3/0 0

    Output queue for Serial3/0 is 1/256



    Packet 1, linktype: compressed-rtp, length: 10, flags: 0x10000808



    mac-pn-230b-dr002#sh queue s3/0 0

    Output queue for Serial3/0 is 2/256



    Packet 1, linktype: compressed-rtp, length: 26, flags: 0x10000808



    Packet 2, linktype: compressed-rtp, length: 9, flags: 0x10000808



    mac-pn-230b-dr002#

    mac-pn-230b-dr002#

    mac-pn-230b-dr002#

    ReplyDelete
  5. Dear Vl@d,

    to start with, problems of this complexity are best handled by dedicated support engineers. I would suggest you open a case with your system integrator or Cisco TAC.

    Furthermore, even if I would have the time to troubleshoot your problem, doing so without router configuration is impossible.

    Last but definitely not least, you should have asked before posting the dump of your router queues as a comment. I doubt its contents are of much interest to future readers of this post.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.