An Observation By : Tim Evens, CCIE #6109 (SP & RS) Email: [email protected] / [email protected] Date : 10/2009 Overview   Cisco ME3750’s, 7600 SIP-­‐400/600, and ASR1000’s (not verified on ASR’s yet) implement network processer based shaping at the microsecond level, supporMng ingress and egress shaping. Shaping is performed using approximately a 48us Tc. IOS based routers, which are preTy much all customer edge devices, support shaping egress at a minimum Tc of 4ms. Tc Matters   The Tc value matters because it defines the burst window of 48us 48us 4,800 bits per Tc 4800 bits 48us 2400 bits 2400 bits 48us 1200 bits 48,000 bits per Tc 48us 1200 bits 1200 bits 48us 100Mbps 48us 48000 bits 48us 24000 bits 12000 bits 48us 12000 bits 12000 bits 24000bits 1Gbps conformance. Below illustrates a burst window of 48us for 1Gbps and 100Mbps 48us   There are 20,833 samples/windows at 48us. With a fixed Tc, the sample/shape window remains the same, therefore the allowed burst shrinks per window as the bits per second decreases   In IOS, Tc is derived by using Tc=bc/CIR. Tc is not configurable in IOS but Bc and CIR are. The lowest value for IOS is 4ms Tc Compare   Using a 100Mbps rate, at 4ms the burst is 400,000 bits per shaping window. In other words; 400,000 bits * 250 shape windows is 100,000,000 bits per second.   The Tc at 48us gives us 20,833 shape windows with 4800 bits per window. This isn’t even enough bits for a 1500 byte packet (12,000 bits). In this case, a window will be skipped in order to credit enough bits. 1Gbps to 100Mbps (shaped)   To calculate proportions, we need to bring values to the same base or interval in this case, which is microseconds, not milliseconds.   1Gbps line rate is 1000bits per microsecond, resulting in 48000 bits per 48us shape window.   At 100Mbps, the allowed 48us shaped window burst is 4800 bits, not 48,000 bits. What happens to the remaining 43,200 bits?   The answer is that if Hierarchical Queuing Framework (HQF) is used, the third or second level policy can queue the excess packets in order to buffer long enough before they they are policed/dropped. If a 2 or 3 level queue is not used, the packets will be policed (dropped) in the ME3750 and 7600/SIP-­‐400/600. 1Gbps to 100Mbps (shaped) -­‐ Config   Below is a ME3750 configuration example showing how to queue the excess packets that are shaped  
policy-­‐map PE-­‐1G-­‐phy_in description -­‐-­‐==| Physical policy – 1Gbps |==-­‐-­‐ class class-­‐default shape average 1000000000 service-­‐policy PE-­‐VLAN_shape  
policy-­‐map PE-­‐VLAN_shape Description -­‐-­‐==| QoS/shape at VLAN level |==-­‐-­‐ class Cust-­‐vlan shape average 100000000  100Mbps service-­‐policy PE-­‐QoS_default   policy-­‐map PE-­‐QoS_default Description -­‐-­‐==| QoS/queue traffic *** class class-­‐default bandwidth remaining percent 100 queue-­‐limit 9000  Enough packets to prevent or reduce drops Problem? The burst size from the customer is 4ms (400,000 bits) transmitted at 1Gig line rate which could be transmitted in 400 microseconds or in bursts over the 4ms window. As long as the IOS 4ms window is not exceeded, the shaper doesn't queue, thus slow down the traffic. For this reason, microbursts at 1Gbps line rates slip through the IOS 4ms shaper but run into a brick wall on the microsecond shaper implemented in the ME3750 and 7600 SIP-­‐400/600.   The customer cannot reduce their egress traffic down to the configured microsecond shape burst window unless the customer implements the same hardware with microsecond shaping   If the service provider also shapes egress on the customer edge, then microbust convergence in the cloud is likely and cannot be controlled by the customer unless symmetric forwarding (layer 2 & 3) or EVC/VLAN mapping are used   Service providers normally charge extra for QoS and therefore will not typically queue/buffer using the third (second in SIP-­‐400/600) policy, which is required with microsecond shapers due to customer equipment restrictions of 4ms minimum shapers Illustrations Problem examples with Qwest QMOE (Layer 2 MPLS/VPLS) services Shaping Tc Mismatch Problem Remote Packets are transmiTed at 100Mbps rate, illustrated by the long dash line Qwest shapes/polices outgoing rate to purchased rate of 100Mbps. Packets at micro-­‐second level are dropped. Packets are transmiTed at 1Gbps rate, illustrated by the short dash line HUB Resolu;on is to shape this at the micro-­‐second level to match the purchased rate for the desMnaMon Qwest MAN SP Microburst Convergence Problem Remote Rate is 100Mbps and contains both HUB site flows. Packets converge to go out to the remote site. Qwest switch sees both HUB site flows at 100Mbps each, combined rate is 200Mbps. Qwest drops packets egress due to the over subscribed rate of 200Mbps. Packets are transmiTed at 100Mbps rate per desMnaMon purchased rate HUB 1 Packets are transmiTed at 100Mbps rate per desMnaMon purchased rate Qwest MAN Resolu;on is to remove one of these flows or to shape each ½ the rate so that each flow does not use more than purchased rate when combined. Or pay for EVC’s or QoS within Qwest. HUB 2 Shared VLAN Microburst Multicast Problem Remote 1 Remote site 2 receives remote site 1’s mulMcast because of the shared Qwest VLAN. This remote requests to receive mulMcast from HUB site 1. HUB 2 receives remote site 1’s mulMcast because of the shared Qwest VLAN. HUB 1 sends mulMcast to remote site 1 as requested. HUB 1 Remote 2 Qwest MAN Resolu;on is to forward this mulMcast traffic via unicast over the Qwest shared vlan or purchase EVC’s where Qwest maps VLANS to specific sites. HUB 2 Conclusion Conclusion Cisco introduced ME3750 and 7600 SIP-­‐400/600 devices that support microsecond shaping. Customers are still using hardware that only supports a minimum of 4ms shaping. Due to the microsecond shaping, the customer traffic at 4ms shaping can/will microburst greater than the 48us shaper window, resulting in packet drops in the service provider network. The customer is left hanging and either forced to purchase new hardware to terminate the service provider connections, EVC’s to map layer 2 VLANS, or QoS to buffer the packets that would have been dropped by the microsecond shaper. If microsecond shapers are used by service providers, then service providers should by default buffer customer traffic enough to handle the microburst up to 4ms.