RoCE
In NetQ, use either the UI or CLI to monitor RoCE in your network.
The syntax for RoCE commands is:
netq [<hostname>] show roce-counters [<text-port>] tx | rx [roce | general] [around <text-time>] [json]
netq [<hostname>] show roce-config [<text-port>] [around <text-time>] [json]
netq [<hostname>] show roce-counters pool [json]
netq [<hostname>] show events [message_type tca_roce]
netq [<hostname>] show events [message_type roceconfig]
View the RoCE Configuration
To view the RoCE configuration, run netq show roce-config.
View RoCE Counters
You can view the following RoCE counters for a given switch through the UI or CLI:
- Rx and Tx counters
- General, CNP, and RoCE-specific counters
- Counter pools
- Port-specific counters
View Rx Counters
- To view Rx counters, open the large switch card, then click the RoCE icon (
). - Expand the card to the largest size, then select RoCE Counters from the side menu:
To view general and CNP Rx counters, run netq show roce-counters rx general:
cumulus@switch:~$ netq show roce-counters rx general
Matching roce records:
Hostname Interface PG packets PG bytes no buffer discard buffer usage buffer max usage PG usage PG max usage
----------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
switch swp1s1 1627273 152582910 0 0 1 0 1
switch swp1s2 1627273 152582910 0 0 1 0 1
switch swp63s1 1618361 160178796 0 0 2 0 2
switch swp1s0 1627273 152582910 0 0 1 0 1
switch swp63s3 1618361 160178796 0 0 2 0 2
switch swp1s3 1627273 152582910 0 0 1 0 1
switch swp63s0 1094532 120228456 0 0 1 0 1
switch swp63s2 1618361 160178796 0 0 2 0 2
To view RoCE-specific Rx counters, run netq show roce-counters rx roce:
cumulus@switch:~$ netq show roce-counters rx roce
Matching roce records:
Hostname Interface PG packets PG bytes no buffer discard PFC pause packets PFC pause duration buffer usage buffer max usage PG usage PG max usage
----------------- --------------- ------------ ------------ ------------------ ------------------ ------------------ ------------ ------------------ ------------ ---------------
switch swp1s1 0 0 0 0 0 0 0 0 0
switch swp1s2 0 0 0 0 0 0 0 0 0
switch swp63s1 0 0 0 0 0 0 0 0 0
switch swp1s0 0 0 0 0 0 0 0 0 0
switch swp63s3 0 0 0 0 0 0 0 0 0
switch swp1s3 0 0 0 0 0 0 0 0 0
switch swp63s0 0 0 0 0 0 0 0 0 0
switch swp63s2 0 0 0 0 0 0 0 0 0
View Tx Counters
- To view Tx counters, open the large switch card, then click the RoCE icon ().
- Expand the card to the largest size, then select RoCE Counters from the side menu.
- Select Tx from the toggle above the table:
To view general and CNP Tx counters, run netq show roce-counters tx general:
cumulus@switch:~$ netq show roce-counters tx general
Matching roce records:
Hostname Interface ECN marked packets TC packets TC bytes unicast no buffer discard buffer usage buffer max usage TC usage TC max usage
----------------- --------------- -------------------- ------------ ------------ ------------------------- ------------ ------------------ ------------ ------------
switch swp1s1 0 0 0 0 0 0 0 0
switch swp1s2 0 0 0 0 0 0 0 0
switch swp63s1 0 0 0 0 0 0 0 0
switch swp1s0 0 0 0 0 0 0 0 0
switch swp63s3 0 0 0 0 0 0 0 0
switch swp1s3 0 0 0 0 0 0 0 0
switch swp63s0 0 0 0 0 0 0 0 0
switch swp63s2 0 0 0 0 0 0 0 0
cumulus@switch :~$
To view RoCE-specific Tx counters, run netq show roce-counters tx roce:
cumulus@switch:~$ netq show roce-counters tx roce
Matching roce records:
Hostname Interface TC packets TC bytes unicast no buffer discard PFC pause packets PFC pause duration buffer usage buffer max usage TC usage TC max usage
----------------- --------------- ---------- ---------- ------------------------- ------------------ ------------------ ------------ ------------------ ---------- ---------------
switch swp1s1 0 0 0 0 0 0 0 0 0
switch swp1s2 0 0 0 0 0 0 0 0 0
switch swp63s1 0 0 0 0 0 0 0 0 0
switch swp1s0 0 0 0 0 0 0 0 0 0
switch swp63s3 0 0 0 0 0 0 0 0 0
switch swp1s3 0 0 0 0 0 0 0 0 0
switch swp63s0 0 0 0 0 0 0 0 0 0
switch swp63s2 0 0 0 0 0 0 0 0 0
View RoCE Counter Pools
- To view RoCE counter pools, open the large switch card, then click the RoCE icon ().
- Switch to the full-screen card, then click RoCE Counters. Look for these columns: Lossy Default Ingress Size, RoCE Reserved Ingress Size, Lossy Default Egress Size, and RoCE Reserved Egress Size.
To view the RoCE counter pools, run netq show roce-counters pool:
cumulus@switch:~$ netq show roce-counters pool
Matching roce records:
Hostname Lossy Default Ingress Size Roce Reserved Ingress Size Lossy Default Egress Size Roce Reserved Egress Size
----------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------
switch 104823 104823 104823 104823
View Counters for a Specific Switch Port
- Open the large switch card, then click the RoCE icon ().
- Select a port from the list on the left:
To view counters for a specific switch port, include the switch name with the command:
cumulus@switch:~$ netq show roce-counters swp1s1 rx general
Matching roce records:
Hostname Interface PG packets PG bytes no buffer discard buffer usage buffer max usage PG usage PG max usage
----------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
switch swp1s1 1643392 154094520 0 0 1 0 1
View Results from a Time in the Past
To view counters for a time period in the past:
- Open the large switch card, then click the RoCE icon ().
- Click
in the header and select a different time period.
Use around with any RoCE-related command to view counters from a previous time period:
cumulus@switch:~$ netq show roce-counters swp1s1 rx general around 1h
Matching roce records:
Hostname Interface PG packets PG bytes no buffer discard buffer usage buffer max usage PG usage PG max usage
----------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
switch swp1s1 661 61856 0 0 1 0 1
Disable RoCE Monitoring
To disable RoCE monitoring:
-
Edit
/etc/netq/commands/cl4-netq-commands.ymland comment out the following lines:cumulus@netq-ts:~$ sudo nano /etc/netq/commands/cl4-netq-commands.yml #- period: "60" # key: "roce" # isactive: true # command: "/usr/lib/cumulus/mlxcmd --json roce counters" # parser: "local" -
Delete the
/var/run/netq/netq_commands.ymlfile:cumulus@netq-ts:~$ sudo rm /var/run/netq/netq_commands.yml -
Restart the NetQ Agent:
cumulus@netq-ts:~$ netq config agent restart