Buffer and Queue Management
Hardware datapath configuration manages packet buffering, queueing and scheduling in hardware.
The /usr/lib/python2.7/dist-packages/cumulus/__chip_config/[bcm|mlx]/datapath.conf assigns buffer space and egress queues. The default thresholds defined in the datapath.conf file are intended for data center environments, but certain workloads may require additional tuning. It is best to make small, incremental changes to validate the changes with your application performance. Be sure to back up the original file before making changes.
Each packet is assigned to an ASIC Class of Service (CoS) value based on the priority value of the packet stored in the 802.1p (Class of Service) or DSCP (Differentiated Services Code Point) header field. The choice to schedule packets based on COS or DSCP is a configurable option in the /etc/cumulus/datapath/traffic.conf file.
Priority groups include:
- Control: Highest priority traffic
- Service: Second-highest priority traffic
- Bulk: All remaining traffic
The scheduler is configured to use a hybrid scheduling algorithm. It applies strict priority to control traffic queues and a weighted round robin selection from the remaining queues. Unicast packets and multicast packets with the same priority value are assigned to separate queues, which are assigned equal scheduling weights.
You can configure Quality of Service (QoS) for switches on the following platforms only:
- Broadcom Tomahawk, Trident II, Trident II+, and Trident3
- Mellanox Spectrum, Spectrum-2, and Spectrum-3
Traffic Marking
You can mark traffic for egress packets through iptables or ip6tables rule classifications. To enable these rules, you do one of the following:
- Mark DSCP values in egress packets.
- Mark 802.1p CoS values in egress packets.
To enable traffic marking, use cl-acltool. Add the -p option to specify the location of the policy file. By default, if you do not include the -p option, cl-acltool looks for the policy file in /etc/cumulus/acl/policy.d/.
The iptables-/ip6tables-based marking is supported with the following action extension:
-j SETQOS --set-dscp 10 --set-cos 5
For ebtables, the setqos keyword must be in lowercase, as in:
[ebtables]
-A FORWARD -o swp5 -j setqos --set-cos 5
You can specify one of the following targets for SETQOS/setqos:
| Option |
Description |
|---|---|
--set-cos INT |
Sets the datapath resource/queuing class value. Values are defined in IEEE P802.1p. |
--set-dscp value |
Sets the DSCP field in packet header to a value, which can be either a decimal or hex value. |
--set-dscp-class class |
Sets the DSCP field in the packet header to the value represented by the DiffServ class value. This class can be EF, BE or any of the CSxx or AFxx classes. |
You can specify either --set-dscp or --set-dscp-class, but not both.
Here are two example rules:
[iptables]
-t mangle -A FORWARD --in-interface swp+ -p tcp --dport bgp -j SETQOS --set-dscp 10 --set-cos 5
[ip6tables]
-t mangle -A FORWARD --in-interface swp+ -j SETQOS --set-dscp 10
You can put the rule in either the mangle table or the default filter table; the mangle table and filter table are put into separate TCAM slices in the hardware.
To put the rule in the mangle table, include -t mangle; to put the rule in the filter table, omit -t mangle.
Priority Flow Control
Priority flow control, as defined in the IEEE 802.1Qbb standard, provides a link-level flow control mechanism that can be controlled independently for each Class of Service (CoS) with the intention to ensure no data frames are lost when congestion occurs in a bridged network.
PFC is not supported on switches with the Helix4 ASIC.
PFC is a layer 2 mechanism that prevents congestion by throttling packet transmission. When PFC is enabled for received packets on a set of switch ports, the switch detects congestion in the ingress buffer of the receiving port and signals the upstream switch to stop sending traffic. If the upstream switch has PFC enabled for packet transmission on the designated priorities, it responds to the downstream switch and stops sending those packets for a period of time.
PFC operates between two adjacent neighbor switches; it does not provide end-to-end flow control. However, when an upstream neighbor throttles packet transmission, it could build up packet congestion and propagate PFC frames further upstream: eventually the sending server could receive PFC frames and stop sending traffic for a time.
The PFC mechanism can be enabled for individual switch priorities on all or specific switch ports for received and/or transmitted traffic. The ingress buffer occupancy of the switch port is used to measure congestion. If congestion is present, the switch transmits flow control frames to the upstream switch. Packets with priority values that do not have PFC configured are not counted during congestion detection and they do not get throttled by the upstream switch when it receives flow control frames.
PFC congestion detection is implemented on the switch using xoff and xon threshold values for the specific ingress buffer used by the targeted switch priorities. When a packet enters the buffer and the buffer occupancy is above the xoff threshold, the switch transmits an Ethernet PFC frame to the upstream switch to signal packet transmission must stop. When the buffer occupancy drops below the xon threshold, the switch sends another PFC frame upstream to signal that packet transmission can resume. (PFC frames contain a quanta value to indicate a timeout value for the upstream switch: packet transmission can resume after the timer has expired or when a PFC frame with quanta == 0 is received from the downstream switch.)
After the downstream switch sends a PFC frame upstream, it continues to receive packets until the upstream switch receives and responds to the PFC frame. The downstream ingress buffer must be large enough to store those additional packets after the xoff threshold is reached.
Priority flow control is fully supported on both Broadcom (including the Edgecore Minipack-AS8000/Trident3) and Mellanox switches.
PFC is disabled by default in Cumulus Linux. To configure PFC, update and uncomment the settings in the priority flow control section of the /etc/cumulus/datapath/traffic.conf file.
# to configure priority flow control on a group of ports:
# -- assign cos value(s) to the cos list
# -- add or replace port group names in the port group list
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- set a PFC buffer size in bytes for each port in the group
# -- set the xoff byte limit (buffer limit that triggers PFC frames transmit to start)
# -- set the xon byte delta (buffer limit that triggers PFC frames transmit to stop)
# -- enable PFC frame transmit and/or PFC frame receive
# priority flow control
pfc.port_group_list = [pfc_port_group]
pfc.pfc_port_group.cos_list = []
pfc.pfc_port_group.port_set = swp1-swp4,swp6
pfc.pfc_port_group.port_buffer_bytes = 25000
pfc.pfc_port_group.xoff_size = 10000
pfc.pfc_port_group.xon_delta = 2000
pfc.pfc_port_group.tx_enable = true
pfc.pfc_port_group.rx_enable = true
#
# Specify cable length in mts
pfc.pfc_port_group.cable_length = 10
| PFC Setting | Description |
|---|---|
pfc.port_group_list |
The name of the port group in brackets. |
pfc.pfc_port_group.cos_list |
The CoS value to the ports. |
pfc.pfc_port_group.port_set |
The ports in the port group. |
pfc.pfc_port_group.port_buffer_bytes |
The PFC buffer size. This is the maximum number of bytes allocated for storing bursts of packets, guaranteed at the ingress port. The default is 25000 bytes. This setting is optional. If not provided, the value is derived from the port speed, port MTU, or port cable length. |
pfc.pfc_port_group.xoff_size |
The xoff byte limit. This is a threshold for the PFC buffer; when this limit is reached, an xoff transition is initiated, signaling the upstream port to stop sending traffic, during which time packets continue to arrive due to the latency of the communication. The default is 10000 bytes. This setting is optional. If not provided, the value is derived from the port speed, port MTU, or port cable length. |
pfc.pfc_port_group.xon_delta |
The xon delta limit. This is the number of bytes to subtract from the xoff limit, which results in a second threshold at which the egress port resumes sending traffic. After the xoff limit is reached and the upstream port stops sending traffic, the buffer begins to drain. When the buffer reaches 8000 bytes (assuming default xoff and xon settings), the egress port signals that it can start receiving traffic again. The default is 2000 bytes. This setting is optional. If not provided, the value is derived from the port speed, port MTU, or port cable length. |
pfc.pfc_port_group.tx_enable |
Enables the egress port to signal the upstream port to stop sending traffic. The default is true. |
pfc.pfc_port_group.rx_enable |
Enables the egress port to receive notifications and act on them. The default is true. |
pfc.pfc_port_group.cable_length |
The length of the port group cable. |
On Broadcom switches, after you modify the settings in the /etc/cumulus/datapath/traffic.conf file, you must restart switchd for the changes to take effect; run the cumulus@switch:~$ sudo systemctl restart switchd.service command.
cumulus@switch:~$ sudo systemctl restart switchd.service
Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.
On NVIDIA Spectrum switches, changes to the settings in the /etc/cumulus/datapath/traffic.conf file do not require you to restart switchd. However, you must run the echo 1 > /cumulus/switchd/config/traffic/reload command to apply the settings.
cumulus@switch:~$ echo 1 > /cumulus/switchd/config/traffic/reload
Always run the syntax checker syntax checker before applying the configuration changes.
Port Groups
A port group refers to one or more sequences of contiguous ports. You can define multiple port groups by adding:
- A comma-separated list of port group names to the
port_group_list. - The
port_set,rx_enable, andtx_enableconfiguration lines for each port group.
You can specify the set of ports in a port group in comma-separate sequences of contiguous ports; you can see which ports are contiguous in the /var/lib/cumulus/porttab file. The syntax supports:
- A single port (swp1s0 or swp5).
- A sequence of regular swp ports (swp2-swp5).
- A sequence within a breakout swp port (swp6s0-swp6s3).
- A sequence of regular and breakout ports, provided they are all in a contiguous range. For example:
...
swp2
swp3
swp4
swp5
swp6s0
swp6s1
swp6s2
swp6s3
swp7
...
Link Pause
The PAUSE frame is a flow control mechanism that halts the transmission of the transmitter for a specified period of time, which might be needed if a server or other network node within the data center receives traffic faster than it can handle. In Cumulus Linux, you can configure individual ports to execute link pause by:
- Transmitting pause frames when the ingress buffers become congested (TX pause enable).
- Responding to received pause frames (RX pause enable).
Link pause is disabled by default. To enable link pause, you must configure settings in the /etc/cumulus/datapath traffic.conf file.
Here is an example configuration that enables TX pause and RX pause for swp1 through swp4 and swp6:
# to configure pause on a group of ports:
# -- add or replace port group names in the port group list
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- set a pause buffer size in bytes for each port
# -- set the xoff byte limit (buffer limit that triggers pause frames transmit to start)
# -- set the xon byte delta (buffer limit that triggers pause frames transmit to stop)
# -- enable pause frame transmit and/or pause frame receive
link pause
link_pause.port_group_list = [pause_port_group]
link_pause.pause_port_group.port_set = swp1-swp4,swp6
link_pause.pause_port_group.port_buffer_bytes = 25000
link_pause.pause_port_group.xoff_size = 10000
link_pause.pause_port_group.xon_delta = 2000
link_pause.pause_port_group.rx_enable = true
link_pause.pause_port_group.tx_enable = true
# Specify cable length in mts
link_pause.pause_port_group.cable_length = 10
This link_pause.pause_port_group.port_buffer_bytes, link_pause.pause_port_group.xoff_size, and link_pause.pause_port_group.xon_delta settings are optional. If not provided, the values are derived from the port speed, port MTU, or port cable length.
On Broadcom switches, after you modify the settings in the /etc/cumulus/datapath/traffic.conf file, you must restart switchd for the changes to take effect; run the cumulus@switch:~$ sudo systemctl restart switchd.service command.
cumulus@switch:~$ sudo systemctl restart switchd.service
Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.
On NVIDIA Spectrum switches, changes to the settings in the /etc/cumulus/datapath/traffic.conf file do not require you to restart switchd. However, you must run the echo 1 > /cumulus/switchd/config/traffic/reload command to apply the settings.
cumulus@switch:~$ echo 1 > /cumulus/switchd/config/traffic/reload
Always run the syntax checker syntax checker before applying the configuration changes.
Cut-through Mode and Store and Forward Switching
Cut-through mode is disabled in Cumulus Linux by default on switches with Broadcom ASICs. On NVIDIA Spectrum switches, you cannot disable cut-through mode.
# Cut-through is disabled by default on all chips with the exception of
# Spectrum. On Spectrum cut-through cannot be disabled.
#cut_through_enable = false
If cut-though mode is enabled and link pause is asserted, Cumulus Linux generates a TOVR and TUFL ERROR; certain error counters increment on a given physical port.
cumulus@switch:~$ sudo ethtool -S swp49 | grep Error
HwIfInDot3LengthErrors: 0
HwIfInErrors: 0
HwIfInDot3FrameErrors: 0
SoftInErrors: 0
SoftInFrameErrors: 0
HwIfOutErrors: 35495749
SoftOutErrors: 0
cumulus@switch:~$ sudo ethtool -S swp50 | grep Error
HwIfInDot3LengthErrors: 3038098
HwIfInErrors: 297595762
HwIfInDot3FrameErrors: 293710518
To work around this issue, disable link pause by commenting out the link_pause* section in the /etc/cumulus/datapath/traffic.conf file:
cumulus@switch:~$ sudo nano /etc/cumulus/datapath/traffic.conf
# link pause
# link_pause.port_group_list = [pause_port_group]
# link_pause.pause_port_group.port_set = swp1-swp4,swp6
# link_pause.pause_port_group.port_buffer_bytes = 25000
# link_pause.pause_port_group.xoff_size = 10000
# link_pause.pause_port_group.xon_delta = 2000
# link_pause.pause_port_group.rx_enable = true
# link_pause.pause_port_group.tx_enable = true
#
# Specify cable length in mts
# link_pause.pause_port_group.cable_length = 10
To enable store and forward switching, set cut_through_enable to false in the /etc/cumulus/datapath/traffic.conf file:
cumulus@switch:~$ sudo nano /etc/cumulus/datapath/traffic.conf
cut_through_enable = false
On Broadcom switches, after you modify the settings in the /etc/cumulus/datapath/traffic.conf file, you must restart switchd for the changes to take effect; run the cumulus@switch:~$ sudo systemctl restart switchd.service command. Always run the syntax checker syntax checker before applying the configuration changes.
cumulus@switch:~$ sudo systemctl restart switchd.service
Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.
On switches using Broadcom Tomahawk, Trident II, Trident II+, and Trident3 ASICs, Cumulus Linux supports store and forward switching but does not support cut-through mode.
On switches with the Mellanox Spectrum ASIC, Cumulus Linux supports cut-through mode but does not support store and forward switching.
Congestion Notification
Explicit Congestion Notification (ECN) is defined by RFC 3168. ECN enables the Cumulus Linux switch to mark a packet to signal impending congestion instead of dropping the packet, which is how TCP typically behaves when ECN is not enabled.
ECN is a layer 3 end-to-end congestion notification mechanism only. Packets can be marked as ECN-capable transport (ECT) by the sending server. If congestion is observed by any switch while the packet is getting forwarded, the ECT-enabled packet can be marked by the switch to indicate the congestion. The end receiver can respond to the ECN-marked packets by signaling the sending server to slow down transmission. The sending server marks a packet ECT by setting the least two significant bits in an IP header DiffServ (ToS) field to 01 or 10. A packet that has the least teo significant bits set to 00 indicates a non-ECT-enabled packet.
The ECN mechanism on a switch only marks packets to notify the end receiver. It does not take any other action or change packet handling in any way, nor does it respond to packets that have already been marked ECN by an upstream switch.
On Trident II switches only, if ECN is enabled on a specific queue, the ASIC also enables RED on the same queue. If the packet is ECT marked (the ECN bits are 01 or 10), the ECN mechanism executes as described above. However, if it is entering an ECN-enabled queue but is not ECT marked (the ECN bits are 00), then the RED mechanism uses the same threshold and probability values to decide whether to drop the packet. Packets entering a non-ECN-enabled queue do not get marked or dropped due to ECN or RED in any case.
ECN is implemented on the switch using minimum and maximum threshold values for the egress queue length. When a packet enters the queue and the average queue length is between the minimum and maximum threshold values, a configurable probability value will determine whether the packet is marked. If the average queue length is above the maximum threshold value, the packet is always marked.
The downstream switches with ECN enabled perform the same actions as the traffic is received. If the ECN bits are set, they remain set. The only way to overwrite ECN bits is to set the ECN bits to 11.
ECN is supported on Broadcom Tomahawk, Tomahawk2, Trident II, Trident II+ and Trident3, and Mellanox Spectrum ASICs.
ECN is disabled by default in Cumulus Linux. You can enable ECN for individual switch priorities on specific switch ports in the /etc/cumulus/datapath/traffic.conf file:
- Specify the name of the port group in
ecn.port_group_listin brackets; for example,ecn.port_group_list = [ecn_port_group]. - Assign a CoS value to the port group in
ecn.ecn_port_group.cos_list. If the CoS value of a packet matches the value of this setting, ECN is applied. - Populate the port group with its member ports (
ecn.ecn_port_group.port_set). Congestion is measured on the egress port queue for the ports listed here, using the average queue length: if congestion is present, a packet entering the queue can be marked to indicate that congestion was observed. Marking a packet involves setting the least 2 significant bits in the IP header DiffServ (ToS) field to 11. - The switch priority value(s) are mapped to specific egress queues for the target switch ports.
- The
ecn.ecn_port_group.probabilityvalue indicates the probability of a packet being marked if congestion is experienced.
The following configuration example shows ECN configured for ports swp1 through swp4 and swp6:
# Explicit Congestion Notification
# to configure ECN and RED on a group of ports:
# -- add or replace port group names in the port group list
# -- assign cos value(s) to the cos list
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- to enable RED requires the latest traffic.conf
ecn_red.port_group_list = [ecn_red_port_group]
ecn_red.ecn_red_port_group.cos_list = [3]
ecn_red.ecn_red_port_group.port_set = swp1-swp4,swp6
ecn_red.ecn_red_port_group.ecn_enable = true
ecn_red.ecn_red_port_group.red_enable = false
ecn_red.ecn_red_port_group.min_threshold_bytes = 40000
ecn_red.ecn_red_port_group.max_threshold_bytes = 200000
ecn_red.ecn_red_port_group.probability = 100
On Broadcom switches, after you modify the settings in the /etc/cumulus/datapath/traffic.conf file, you must restart switchd for the changes to take effect; run the cumulus@switch:~$ sudo systemctl restart switchd.service command.
cumulus@switch:~$ sudo systemctl restart switchd.service
Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.
On NVIDIA Spectrum switches, changes to the settings in the /etc/cumulus/datapath/traffic.conf file do not require you to restart switchd. However, you must run the echo 1 > /cumulus/switchd/config/traffic/reload command to apply the settings.
cumulus@switch:~$ echo 1 > /cumulus/switchd/config/traffic/reload
Always run the syntax checker syntax checker before applying the configuration changes.
Scheduling Weights Per Egress Queue
On NVIDIA Spectrum switches, you can set the scheduling weight per egress queue, which determines the amount of bandwidth assigned to the queue. Cumulus Linux supports eight queues per port. You can either use a default profile that each port inherits or create separate profiles that map a different set of ports. Each profile, including the default profile, has weights configured for each egress queue (0-7).
You set the weights per egress queue as a percentage. The total weight percentages for all egress queues cannot be greater than 100. If you do not define a weight for an egress queue, no scheduling is done for packets on this queue if congestion occurs. If you want to configure strict scheduling on an egress queue (always send every single packet in the queue) set the value to 0.
You can configure per queue egress scheduling with NCLU commands or manually by editing the /etc/cumulus/datapath/traffic.conf file.
Cumulus Linux provides a default profile. You can either enable the default profile or configure a non-default profile.
The following example commands enable the default profile:
cumulus@switch:~$ net add qos egress-sched default_profile
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit
In the default profile, the egress queue weights are set as follows. You cannot modify these values with NCLU.
...
default_egress_sched.egr_queue_0.bw_percent = 12
default_egress_sched.egr_queue_1.bw_percent = 12
default_egress_sched.egr_queue_2.bw_percent = 24
default_egress_sched.egr_queue_3.bw_percent = 12
default_egress_sched.egr_queue_4.bw_percent = 12
default_egress_sched.egr_queue_5.bw_percent = 12
default_egress_sched.egr_queue_6.bw_percent = 12
default_egress_sched.egr_queue_7.bw_percent = 0
The following commands create a non-default profile for port group port_group1 for swp2 and swp3, set the weight to 30 percent on egress queue 2 and strict scheduling on egress queue 3:
cumulus@switch:~$ net add qos egress-sched profile port_set swp2-swp3
cumulus@switch:~$ net add qos egress_sched profile sched_port_group1 queue 2 dwrr bw_percent 30
cumulus@switch:~$ net add qos egress_sched profile sched_port_group1 queue 3 strict
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit
The NCLU commands save the configuration in the /etc/cumulus/datapath/traffic.conf file. For example:
...
egress_sched.port_group_list = [sched_port_group1]
egress_sched.sched_port_group1.port_set = swp2-swp3
egress_sched.sched_port_group1.egr_queue_2.bw_percent = 30
egress_sched.sched_port_group1.egr_queue_3.bw_percent = 0
...
- To configure a non-default profile with NCLU, you must configure the port set for the profile before you configure the bandwidth percent for the egress queues.
- The total bandwidth percent for all egress queues cannot be greater than 100.
- If you delete the port set for a non-default profile, the bandwidth percent for all the queues in that profile are deleted.
To configure per queue egress scheduling manually in the /etc/cumulus/datapath/traffic.conf file, update and uncomment the settings in the default egress scheduling weight per egress queue section of the /etc/cumulus/datapath/traffic.conf file.
The following example enables the default profile, and sets the weight to 30 percent for egress queue 2 and 10 percent for the remaining egress queues. The settings are applied to all ports.
# default egress scheduling weight per egress queue
# To be applied to all the ports if port_group profile not configured
# If you do not specify any bw_percent of egress_queues, those egress queues
# will assume DWRR weight 0 - no egress scheduling for those queues
# '0' indicates strict priority
default_egress_sched.egr_queue_0.bw_percent = 10
default_egress_sched.egr_queue_1.bw_percent = 10
default_egress_sched.egr_queue_2.bw_percent = 30
default_egress_sched.egr_queue_3.bw_percent = 10
default_egress_sched.egr_queue_4.bw_percent = 10
default_egress_sched.egr_queue_5.bw_percent = 10
default_egress_sched.egr_queue_6.bw_percent = 10
default_egress_sched.egr_queue_7.bw_percent = 10
The following example creates a non-default profile for port group port_group1, sets the weight to 30 percent for egress queue 1 and 2, to 0 for egress queue 6 and 7 (always send every single packet from egress queue 6 and 7 before any other queue), and 10 percent for the remaining egress queues:
...
egress_sched.port_group_list = [sched_port_group1]
egress_sched.sched_port_group1.port_set = swp2
egress_sched.sched_port_group1.egr_queue_0.bw_percent = 10
egress_sched.sched_port_group1.egr_queue_1.bw_percent = 30
egress_sched.sched_port_group1.egr_queue_2.bw_percent = 30
egress_sched.sched_port_group1.egr_queue_3.bw_percent = 10
egress_sched.sched_port_group1.egr_queue_4.bw_percent = 10
egress_sched.sched_port_group1.egr_queue_5.bw_percent = 10
egress_sched.sched_port_group1.egr_queue_6.bw_percent = 0
egress_sched.sched_port_group1.egr_queue_7.bw_percent = 0
On Broadcom switches, after you modify the settings in the /etc/cumulus/datapath/traffic.conf file, you must restart switchd for the changes to take effect; run the cumulus@switch:~$ sudo systemctl restart switchd.service command.
cumulus@switch:~$ sudo systemctl restart switchd.service
Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.
On NVIDIA Spectrum switches, changes to the settings in the /etc/cumulus/datapath/traffic.conf file do not require you to restart switchd. However, you must run the echo 1 > /cumulus/switchd/config/traffic/reload command to apply the settings.
cumulus@switch:~$ echo 1 > /cumulus/switchd/config/traffic/reload
Always run the syntax checker syntax checker before applying the configuration changes.
Traffic Shaping
Configure traffic shaping to regulate network traffic by using a lower bitrate than the physical interface is capable of. Traffic shaping prevents packets from being dropped or lost due to bandwidth limits or congestion.
To configure traffic shaping, update and uncomment the settings in the Hierarchical traffic shaping section of the the /etc/cumulus/datapath/traffic.conf file. You can configure traffic shaping per egress queue or aggregated at the port level.
The egress shaping rate configured in the /etc/cumulus/datapath/traffic.conf is always the layer 1 rate. The calculated shaping rate considers overheads in the Ethernet frame like the interframe gap, preamble, cyclic redundancy check (CRC) and so on. The egress layer 3 throughput measured is always less than the maximum shaper rate configured.
The following example shows the Hierarchical traffic shaping section of the /etc/cumulus/datapath/traffic.conf file.
...
# Hierarchical traffic shaping
# to configure shaping at 2 levels:
# - per egress queue egr_queue_0 - egr_queue_7
# - port level aggregate
# -- add or replace a port group names in the port group list
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- set min and max rates in kbps for each egr_queue [min, max]
# -- set max rate in kbps at port level
shaping.port_group_list = [shaper_port_group]
shaping.shaper_port_group.port_set = swp1-swp3
shaping.shaper_port_group.egr_queue_0.shaper = [50000, 100000]
shaping.shaper_port_group.egr_queue_1.shaper = [51000, 150000]
shaping.shaper_port_group.egr_queue_2.shaper = [52000, 200000]
shaping.shaper_port_group.egr_queue_3.shaper = [53000, 250000]
shaping.shaper_port_group.egr_queue_4.shaper = [54000, 300000]
shaping.shaper_port_group.egr_queue_5.shaper = [55000, 350000]
shaping.shaper_port_group.egr_queue_6.shaper = [56000, 400000]
shaping.shaper_port_group.egr_queue_7.shaper = [57000, 450000]
# shaping.shaper_port_group.port.shaper = 900000
The settings are described below:
| Traffic Shaping Setting | Description |
|---|---|
shaping.port_group_list |
The name of the port group. You must enclose the name in square brackets; for example, shaping.port_group_list = [shaper_port_group1]. |
shaping.shaper_port_group.port_set |
The list of ports in the port group. |
shaping.shaper_port_group.egr_queue_0.shaper |
The minimum and maximum rates in kbps for egress queue 0. You must enclose the values in square brackets. |
shaping.shaper_port_group.egr_queue_1.shaper |
The minimum and maximum rates in kbps for egress queue 1. You must enclose the values in square brackets. |
shaping.shaper_port_group.egr_queue_2.shaper |
The minimum and maximum rates in kbps for egress queue 2. You must enclose the values in square brackets. |
shaping.shaper_port_group.egr_queue_3.shaper |
The minimum and maximum rates in kbps for egress queue 3. You must enclose the values in square brackets. |
shaping.shaper_port_group.egr_queue_4.shaper |
The minimum and maximum rates in kbps for egress queue 4. You must enclose the values in square brackets. |
shaping.shaper_port_group.egr_queue_5.shaper |
The minimum and maximum rates in kbps for egress queue 5. You must enclose the values in square brackets. |
shaping.shaper_port_group.egr_queue_6.shaper |
The minimum and maximum rates in kbps for egress queue 6. You must enclose the values in square brackets. |
shaping.shaper_port_group.egr_queue_7.shaper |
The minimum and maximum rates in kbps for egress queue 7. You must enclose the values in square brackets. |
shaping.shaper_port_group.port.shaper |
The maximum rate in kbps at the port level. At the port level, only the maximum shaper rate is supported. |
scheduling.algorithm |
Cumulus Linux supports the Deficit Weighted Round Robin (DWRR) scheduling algorithm only. |
In Cumulus Linux, the burst size is set to twice the maximum rate internally; the setting is not configurable.
On Broadcom switches, when you modify the configuration in the /etc/cumulus/datapath/traffic.conf file, you must restart switchd for the changes to take effect; run the cumulus@switch:~$ sudo systemctl restart switchd.service command.
cumulus@switch:~$ sudo systemctl restart switchd.service
Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.
On Broadcom switches, after you modify the settings in the /etc/cumulus/datapath/traffic.conf file, you must restart switchd for the changes to take effect; run the cumulus@switch:~$ sudo systemctl restart switchd.service command.
cumulus@switch:~$ sudo systemctl restart switchd.service
Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.
On NVIDIA Spectrum switches, changes to the settings in the /etc/cumulus/datapath/traffic.conf file do not require you to restart switchd. However, you must run the echo 1 > /cumulus/switchd/config/traffic/reload command to apply the settings.
cumulus@switch:~$ echo 1 > /cumulus/switchd/config/traffic/reload
Always run the syntax checker syntax checker before applying the configuration changes.
Interface Buffer Status
On switches with ASICs, you can collect a fine-grained history of queue lengths using histograms maintained by the ASIC; see the ASIC Monitoring for details.
Example Configuration File
The following example /etc/cumulus/datapath/traffic.conf datapath configuration file applies to 10G, 40G, and 100G switches on Broadcom Tomahawk, Trident II, Trident II+, or Trident3 and Mellanox Spectrum platforms only.
- For the default source packet fields and mapping, each selected packet field must have a block of mapped values. Any packet field value that is not specified in the configuration is assigned to a default internal switch priority. The configuration applies to every forwarding port unless a custom remark configuration is defined for that port (see below).
- For the default remark packet fields and mapping, each selected packet field should have a block of mapped values. Any internal switch priority value that is not specified in the configuration is assigned to a default packet field value. The configuration applies to every forwarding port unless a custom remark configuration is defined for that port (see below).
- Per-port source packet fields and mapping apply to the designated set of ports.
- Per-port remark packet fields and mapping apply to the designated set of ports.
On switches with Spectrum ASICs, you must enable packet priority remark on the ingress port. A packet received on a remark-enabled port is remarked according to the priority mapping configured on the egress port. If you configure packet priority remark the same way on every port, the default configuration example above is correct. However, per-port customized configurations require two port groups, one for the ingress ports and one for the egress ports, as below:
remark.port_group_list = [ingress_remark_group, egress_remark_group]
remark.ingress_remark_group.packet_priority_remark_set = [dscp]
remark.remark_port_group.port_set = swp1-swp4,swp6
remark.egress_remark_group.port_set = swp10-swp20
remark.egress_remark_group.cos_0.priority_remark.dscp = [2]
remark.egress_remark_group.cos_1.priority_remark.dscp = [10]
remark.egress_remark_group.cos_2.priority_remark.dscp = [18]
remark.egress_remark_group.cos_3.priority_remark.dscp = [26]
remark.egress_remark_group.cos_4.priority_remark.dscp = [34]
remark.egress_remark_group.cos_5.priority_remark.dscp = [42]
remark.egress_remark_group.cos_6.priority_remark.dscp = [50]
remark.egress_remark_group.cos_7.priority_remark.dscp = [58]
Syntax Checker
Cumulus Linux provides a syntax checker for the /etc/cumulus/datapath/traffic.conf file to check for errors, such missing parameters, or invalid parameter labels and values.
On Broadcom switches, the syntax checker runs automatically during switchd initialization and reports syntax errors to the /var/log/switchd.log file.
On both Broadcom and NVIDIA switches, you can run the syntax checker manually from the command line by issuing the cl-consistency-check --datapath-syntax-check command. If errors exist, they are written to stderr by default. If you run the command with -q, errors are written to the /var/log/switchd.log file.
The cl-consistency-check --datapath-syntax-check command takes the following options:
Option |
Description |
|---|---|
| -h | Displays this list of command options. |
| -q | Runs the command in quiet mode. Errors are written to the /var/log/switchd.log file instead of stderr. |
-t <file-name> |
Runs the syntax check on a non-default traffic.conf file; for example, /mypath/test-traffic.conf. |
You can run the syntax checker when switchd is either running or stopped.
Example Commands
The following example command runs the syntax checker on the default /etc/cumulus/datapath/traffic.conf file and shows that no errors are detected:
cumulus@switch:~$ cl-consistency-check --datapath-syntax-check
No errors detected in traffic config file /etc/cumulus/datapath/traffic.conf
The following example command runs the syntax checker on the default /etc/cumulus/datapath/traffic.conf file in quiet mode. If errors exist, they are written to the /var/log/switchd.log file.
cumulus@switch:~$ cl-consistency-check --datapath-syntax-check -q
The following example command runs the syntax checker on the /mypath/test-traffic.conf file and shows that errors are detected:
cumulus@switch:~$ cl-consistency-check --datapath-syntax-check -t /path/test-traffic.conf
Traffic source 8021p: missing mapping for priority value '7'
Errors detected while checking traffic config file /mypath/test-traffic.conf
The following example command runs the syntax checker on the /mypath/test-traffic.conf file in quiet mode. If errors exist, they are written to the /var/log/switchd.log file.
cumulus@switch:~$ cl-consistency-check --datapath-syntax-check -t /path/test-traffic.conf -q