IOS Queuing 101
A while ago I've been a co-mentor to a graduate student who investigated the difference between shaping and policing for his thesis. While doing the lab tests, he stumbled across an interesting problem: he could not configure fancy queuing (for example, CB-WFQ) on a point-to-point Frame Relay subinterface. The solution was to configure a QoS hierarchy, doing shaping first and queuing within the shaping queues.
A few days ago, Arden Packeer covered the same topic in one of his blog posts, so it looks like the underlying problem (and the solution) is not as widely known as I had assumed. Time to write a tutorial on Queuing Principles in Cisco IOS.
A few days ago, Arden Packeer covered the same topic in one of his blog posts, so it looks like the underlying problem (and the solution) is not as widely known as I had assumed. Time to write a tutorial on Queuing Principles in Cisco IOS.
--
Here's the trade off:
A longer rx/tx_ring means that bursty traffic will be queued up, for busy interfaces, a larger ring size is desirable (generally). The problem is that under LOW traffic conditions, a large ring size is bad for jitter and latency. An interrupt should be triggered by one of two conditions, whichever happens first:
1) A certain fraction of the ring is full. Generally, an interrupt gets triggered when 1/4-1/3 of the ring is full.
2) A timeout is hit, and traffic is flushed
On a low load interface, the timeout will usually be hit, while a heavily utilized link will hit the fraction first. So the busy interface keeps sending those interrupts and flushing out part of the buffer.
If you compare 2 systems, heavily utilized and low utilization, you'll see dramatic differences in the latency - up to ~4 times! And the shocker for many people is that the latency increases, the lower the traffic levels.
Your trade off is latency vs ability to handle traffic bursts. Tuning these values is not recommended unless you're taking serious hits.
I have one question regarding policer.
Suppose i've set a policer to 5Mbps.
One of my customer, is pinging with MTU size of 1500 repeat of 1000 for example. An he sees packet drop.
Of course i told him, our policer is dropping the packet.
My question is, how do i calculate the exact amount of bandwidth the ping is utilize. So that i can pint point for every reply you will have this amount of timeout since the policer will discard any traffic that goes beyond the 5 Mbps
Regards
Wan T
@Wan (aka Anonymous #2): You cannot generate 5 Mbps of traffic with ping. Ping is a request-response protocol, the second packet is only sent after the first reply has been received (or after the timeout expires).
To calculate the approximate bandwidth used by ping, do the following:
pps = 1000 / average-ping-response[in msec]
bw[bps] = pps * (ping-packet-size +24) * 8
I have traffic-shaping in the main interface and a LL/QCB-WFQ service-policy inside the frame-relay map. It works fine for some interfaces. However, other interfaces never put voice packets inside the high queue od the dual-fifo interface.
Looking the interfaces, I discovered that interfaces that work fine presents matching packets at L3 queue. Then, they go to the high queue.
Serial1/2 voice never goes to high dual-fifo, get stucked with data
fragments in the output queue.
Serial30 voice goes to the high queue.
See that L3 queueing of s1/2 has no match, even with matches at ip dscp EF shows more than 1 million of packets...
:(
look the print (sorry, but it is long...)
mac-pn-230b-dr002#sh int s1/2
Serial1/2 is up, line protocol is up
Hardware is M4T
Description: ==To Noble Muravenko==
MTU 1500 bytes, BW 192 Kbit, DLY 20000 usec,
reliability 255/255, txload 245/255, rxload 46/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive set (10 sec)
Restart-Delay is 0 secs
LMI enq sent 0, LMI stat recvd 0, LMI upd recvd 0
LMI enq recvd 6123, LMI stat sent 6123, LMI upd sent 0, DCE LMI up
LMI DLCI 1023 LMI type is CISCO frame relay DCE
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 8165/0, interface broadcasts 6123
Last input 00:00:05, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:00:49
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4875
Queueing strategy: dual fifo
Output queue: high size/max/dropped 0/256/0
Output queue: 10/128 (size/max)
30 second input rate 35000 bits/sec, 59 packets/sec
30 second output rate 185000 bits/sec, 140 packets/sec
3639099 packets input, 253448953 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
312 input errors, 201 CRC, 0 frame, 0 overrun, 0 ignored, 111 abort
4659821 packets output, 597264717 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions DCD=up DSR=down DTR=up RTS=up CTS=down
mac-pn-230b-dr002#sh queue s1/2
Output queue for Serial1/2 is 12/128
Packet 1, linktype: fr-fragment, length: 246, flags: 0x8
DLCI: 300
Packet 2, linktype: fr-fragment, length: 246, flags: 0x8
DLCI: 300
Packet 3, linktype: compressed-rtp, length: 19, flags: 0x88, CID: 1
source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,
TOS: 184 prot: 17, source port 17878, destination port 17218
Packet 4, linktype: fr-fragment, length: 246, flags: 0x8
DLCI: 300
Packet 5, linktype: fr-fragment, length: 246, flags: 0x8
DLCI: 300
Packet 6, linktype: compressed-rtp, length: 29, flags: 0x88, CID: 1
source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,
TOS: 184 prot: 17, source port 17878, destination port 17218
Packet 7, linktype: fr-fragment, length: 246, flags: 0x8
DLCI: 300
Packet 8, linktype: fr-fragment, length: 48, flags: 0x10000088
DLCI: 300
Packet 9, linktype: fr-fragment, length: 246, flags: 0x8
DLCI: 300
Packet 10, linktype: fr-fragment, length: 246, flags: 0x8
DLCI: 300
Packet 11, linktype: compressed-rtp, length: 28, flags: 0x88, CID: 1
source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,
TOS: 184 prot: 17, source port 17878, destination port 17218
Packet 12, linktype: compressed-rtp, length: 26, flags: 0x88, CID: 1
source: 10.128.0.10, destination: 10.128.9.14, id: 0x2F09, ttl: 253,
TOS: 184 prot: 17, source port 17878, destination port 17218
mac-pn-230b-dr002#
mac-pn-230b-dr002#sh fram pvc int s1/2 100
PVC Statistics for interface Serial1/2 (Frame Relay DCE)
DLCI = 100, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial1/2.100
input pkts 2030261 output pkts 1607788 in bytes 50304341
out bytes 40998431 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 7146 out bcast bytes 839182
30 second input rate 9000 bits/sec, 42 packets/sec
30 second output rate 4000 bits/sec, 23 packets/sec
pvc create time 29w2d, last time pvc status changed 06:03:00
fragment type end-to-end fragment size 240
cir 64000 bc 640 be 0 limit 80 interval 10
mincir 64000 byte increment 80 BECN response no IF_CONG no
frags 56797 bytes 6354709 frags delayed 5464 bytes delayed 797186
shaping inactive
traffic shaping drops 0
service policy CPRK-VoIP
Serial1/2.100: DLCI 100 -
Service-policy output: CPRK-VoIP
Class-map: RTP (match-any)
1552390 packets, 92206236 bytes
30 second offered rate 10000 bps, drop rate 0 bps
Match: ip dscp ef
1552390 packets, 92206236 bytes
30 second rate 10000 bps
Match: dscp ef
0 packets, 0 bytes
30 second rate 0 bps
Match: protocol rtp audio
0 packets, 0 bytes
30 second rate 0 bps
Match: ip rtp 16384 16383
0 packets, 0 bytes
30 second rate 0 bps
QoS Set
dscp ef
Packets marked 1552390
Queueing
Strict Priority
Output Queue: Conversation 24
Bandwidth 90 (%)
Bandwidth 57 (kbps) Burst 1425 (Bytes)
(pkts matched/bytes matched) 0/0
(total drops/bytes drops) 0/0
Class-map: H323 (match-any)
18883 packets, 2473464 bytes
30 second offered rate 0 bps, drop rate 0 bps
Match: ip dscp af31
18883 packets, 2473464 bytes
30 second rate 0 bps
Queueing
Output Queue: Conversation 25
Bandwidth 5 (%)
Bandwidth 3 (kbps) Max Threshold 64 (packets)
(pkts matched/bytes matched) 18883/2473464
(depth/total drops/no-buffer drops) 0/0/0
Class-map: class-default (match-any)
36515 packets, 3867255 bytes
30 second offered rate 0 bps, drop rate 0 bps
Match: any
Queueing
Flow Based Fair Queueing
Maximum Number of Hashed Queues 16
(total queued/total drops/no-buffer drops) 0/0/0
Output queue size 0/max total 600/drops 0
mac-pn-230b-dr002#
mac-pn-230b-dr002#sh fram pvc int s1/2 300
PVC Statistics for interface Serial1/2 (Frame Relay DCE)
DLCI = 300, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial1/2.300
input pkts 1160692 output pkts 1181504 in bytes 199842592
out bytes 556850656 dropped pkts 107 in pkts dropped 107
out pkts dropped 4875 out bytes dropped 972170
late-dropped out pkts 4875 late-dropped out bytes 972170
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 1021 out bcast bytes 349182
30 second input rate 74000 bits/sec, 31 packets/sec
30 second output rate 178000 bits/sec, 29 packets/sec
pvc create time 29w2d, last time pvc status changed 06:03:05
fragment type end-to-end fragment size 240
cir 192000 bc 1920 be 0 limit 240 interval 10
mincir 128000 byte increment 240 BECN response no IF_CONG yes
frags 3048008 bytes 556787812 frags delayed 2667707 bytes delayed 521876100
shaping active
traffic shaping drops 4875
service policy Noble
Serial1/2.300: DLCI 300 -
Service-policy output: Noble
Class-map: NobleSAP (match-any)
2478 packets, 310198 bytes
30 second offered rate 0 bps, drop rate 0 bps
Match: access-group 111
2478 packets, 310198 bytes
30 second rate 0 bps
Queueing
Strict Priority
Output Queue: Conversation 24
Bandwidth 96 (kbps) Burst 2400 (Bytes)
(pkts matched/bytes matched) 2478/310198
(total drops/bytes drops) 0/0
Class-map: class-default (match-any)
1183901 packets, 544459264 bytes
30 second offered rate 166000 bps, drop rate 0 bps
Match: any
Queueing
Flow Based Fair Queueing
Maximum Number of Hashed Queues 16
(total queued/total drops/no-buffer drops) 68/4875/0
exponential weight: 9
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob
af11 15/4185 0/0 0/0 32 40 1/10
af12 3/136 0/0 0/0 28 40 1/10
af13 4/200 0/0 0/0 24 40 1/10
af21 7/350 0/0 0/0 32 40 1/10
af22 57/2960 0/0 0/0 28 40 1/10
af23 0/0 0/0 0/0 24 40 1/10
af31 7/344 0/0 0/0 32 40 1/10
af32 0/0 0/0 0/0 28 40 1/10
af33 0/0 0/0 0/0 24 40 1/10
af41 0/0 0/0 0/0 32 40 1/10
af42 0/0 0/0 0/0 28 40 1/10
af43 0/0 0/0 0/0 24 40 1/10
cs1 1384/509353 1/64 2/128 22 40 1/10
cs2 34/2922 0/0 0/0 24 40 1/10
cs3 7/358 0/0 0/0 26 40 1/10
cs4 36/4963 0/0 0/0 28 40 1/10
cs5 37/37755 0/0 0/0 30 40 1/10
cs6 7/390 0/0 0/0 32 40 1/10
cs7 109/6874 0/0 0/0 34 40 1/10
ef 1/58 0/0 0/0 36 40 1/10
rsvp 0/0 0/0 0/0 36 40 1/10
default 1177331/542923501 1153/491036 3719/480942 20 40 1/10
Output queue size 67/max total 600/drops 4875
mac-pn-230b-dr002#
mac-pn-230b-dr002#sh fram pvc int s3/0 100
PVC Statistics for interface Serial3/0 (Frame Relay DCE)
DLCI = 100, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial3/0.100
input pkts 1362 output pkts 806 in bytes 40669
out bytes 26542 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 26 out bcast bytes 3128
5 minute input rate 6000 bits/sec, 6 packets/sec
5 minute output rate 4000 bits/sec, 6 packets/sec
pvc create time 29w2d, last time pvc status changed 2d16h
fragment type end-to-end fragment size 320
cir 256000 bc 2560 be 0 limit 320 interval 10
mincir 64000 byte increment 320 BECN response no IF_CONG no
frags 828 bytes 26870 frags delayed 8 bytes delayed 1408
shaping inactive
traffic shaping drops 0
service policy Austral-VoIP
Serial3/0.100: DLCI 100 -
Service-policy output: Austral-VoIP
Class-map: RTP (match-any)
708 packets, 40028 bytes
30 second offered rate 8000 bps, drop rate 0 bps
Match: ip dscp ef
708 packets, 40028 bytes
30 second rate 8000 bps
Match: dscp ef
0 packets, 0 bytes
30 second rate 0 bps
Match: protocol rtp audio
0 packets, 0 bytes
30 second rate 0 bps
Match: ip rtp 16384 16383
0 packets, 0 bytes
30 second rate 0 bps
QoS Set
dscp ef
Packets marked 708
Queueing
Strict Priority
Output Queue: Conversation 24
Bandwidth 40 (kbps) Burst 1000 (Bytes)
(pkts matched/bytes matched) 708/13721
(total drops/bytes drops) 0/0
mac-pn-230b-dr002#sh fram pvc int s3/0 100
PVC Statistics for interface Serial3/0 (Frame Relay DCE)
DLCI = 100, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial3/0.100
input pkts 1467 output pkts 914 in bytes 43405
out bytes 29009 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 26 out bcast bytes 3128
5 minute input rate 7000 bits/sec, 7 packets/sec
5 minute output rate 4000 bits/sec, 7 packets/sec
pvc create time 29w2d, last time pvc status changed 2d16h
fragment type end-to-end fragment size 320
cir 256000 bc 2560 be 0 limit 320 interval 10
mincir 64000 byte increment 320 BECN response no IF_CONG no
frags 922 bytes 29113 frags delayed 8 bytes delayed 1408
shaping inactive
traffic shaping drops 0
service policy Austral-VoIP
Serial3/0.100: DLCI 100 -
Service-policy output: Austral-VoIP
Class-map: RTP (match-any)
801 packets, 45642 bytes
30 second offered rate 8000 bps, drop rate 0 bps
Match: ip dscp ef
801 packets, 45642 bytes
30 second rate 8000 bps
Match: dscp ef
0 packets, 0 bytes
30 second rate 0 bps
Match: protocol rtp audio
0 packets, 0 bytes
30 second rate 0 bps
Match: ip rtp 16384 16383
0 packets, 0 bytes
30 second rate 0 bps
QoS Set
dscp ef
Packets marked 802
Queueing
Strict Priority
Output Queue: Conversation 24
Bandwidth 40 (kbps) Burst 1000 (Bytes)
(pkts matched/bytes matched) 802/15870
(total drops/bytes drops) 0/0
mac-pn-230b-dr002#sh queue s3/0
Output queue for Serial3/0 is 10/128
Packet 1, linktype: ip, length: 843, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x96FD, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x0333 0x33D8 0x4A00 0x1401 0x7400
0x0100 0x0001 0x424B 0xB4FD 0x3F2B 0xB7FC 0xE19C
Packet 2, linktype: ip, length: 202, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x96FE, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x00B2 0x7262 0x4A00 0x1401 0x7500
0x0100 0x0001 0x4267 0x020F 0x0FE9 0xFFFD 0xA34D
Packet 3, linktype: ip, length: 45, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x96FF, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x0015 0xE30F 0x4460 0x0001 0x0000
0x0000 0x0001 0xE801 0x0081 0xC04E 0x544C 0x4D53
Packet 4, linktype: ip, length: 45, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x9700, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x0015 0xE30F 0x4460 0x0001 0x0000
0x0000 0x0001 0xE801 0x0053 0x004B 0x0041 0x004E
Packet 5, linktype: ip, length: 202, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x9701, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x00B2 0x9DD1 0x4A00 0x1401 0x7600
0x0100 0x0001 0x4277 0xB01E 0x8602 0xF7FE 0x6258
Packet 6, linktype: ip, length: 1344, flags: 0x10000888
source: 68.142.101.254, destination: 201.23.192.234, id: 0x79E9, ttl: 52,
TOS: 0 prot: 6, source port 80, destination port 1923
data: 0x0050 0x0783 0x3F2A 0xFD33 0xB4AD 0x5AC6 0x5010
0xFFFF 0xC4DC 0x0000 0xA747 0x82D4 0x71C7 0x124D
Packet 7, linktype: ip, length: 202, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x9702, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x00B2 0xB9F4 0x4A00 0x1401 0x7700
0x0100 0x0001 0x4256 0x04B5 0x9901 0xB0FF 0xF4B7
Packet 8, linktype: ip, length: 202, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x9703, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x00B2 0xF037 0x4A00 0x1401 0x7800
0x0100 0x0001 0x4276 0x3089 0xC2A9 0xBFFB 0xDA6D
Packet 9, linktype: ip, length: 202, flags: 0x10000888
source: 201.9.13.87, destination: 201.23.192.234, id: 0x9704, ttl: 117,
TOS: 0 prot: 17, source port 58548, destination port 43832
data: 0xE4B4 0xAB38 0x00B2 0x469A 0x4A00 0x1401 0x7900
0x0100 0x0001 0x424B 0xB08F 0x8603 0x7FE3 0xF6A0
Packet 10, linktype: ip, length: 1344, flags: 0x10000888
source: 68.142.101.254, destination: 201.23.192.234, id: 0x7C60, ttl: 52,
TOS: 0 prot: 6, source port 80, destination port 1923
data: 0x0050 0x0783 0x3F2B 0x0247 0xB4AD 0x5AC6 0x5010
0xFFFF 0xE26A 0x0000 0x21FA 0x0000 0x0000 0x2FFF
mac-pn-230b-dr002#
mac-pn-230b-dr002#
mac-pn-230b-dr002#sh queue s3/0 0
mac-pn-230b-dr002#sh queue s3/0 0
Output queue for Serial3/0 is 1/256
Packet 1, linktype: compressed-rtp, length: 10, flags: 0x10000808
mac-pn-230b-dr002#sh queue s3/0 0
Output queue for Serial3/0 is 2/256
Packet 1, linktype: compressed-rtp, length: 26, flags: 0x10000808
Packet 2, linktype: compressed-rtp, length: 9, flags: 0x10000808
mac-pn-230b-dr002#
mac-pn-230b-dr002#
mac-pn-230b-dr002#
to start with, problems of this complexity are best handled by dedicated support engineers. I would suggest you open a case with your system integrator or Cisco TAC.
Furthermore, even if I would have the time to troubleshoot your problem, doing so without router configuration is impossible.
Last but definitely not least, you should have asked before posting the dump of your router queues as a comment. I doubt its contents are of much interest to future readers of this post.