Configure and Monitor Threshold-Crossing Events

1. Click Menu and navigate to Threshold crossing rules.

2. Select the tab that reflects the event type for the rule.

3. Click Create a rule. Enter a name for the rule and assign a severity, then click Next.

4. Select the attribute you want to monitor. The listed attributes change depending on the type of event you chose in the previous step.

5. Click Next.

6. On the Set threshold step, enter a threshold value.

7. Define the scope of the rule.

   • If you want to restrict the rule based on a particular parameter, enter values for one or more of the available attributes. For What Just Happened rules, select a reason from the available list.

   • If you want the rule to apply to across the network, select the Apply rule to entire network toggle.

8. Click Next.

9. (Optional) Select a notification channel where you want the events to be sent.

   Only previously created channels are available for selection. If no channel is available or selected, the notifications can only be retrieved from the database. You can add a channel at a later time and then add it to the rule.

10. Click Finish. The rules may take several minutes to appear in the UI.

The simplest configuration you can create is one that sends a TCA event generated by all devices and all interfaces to a single notification application. Use the netq add tca command to configure the event. Its syntax is:

netq add tca event_id <text-event-id-anchor>
    [scope <text-scope-anchor>]
    [severity info | severity error]
    [is_active true | is_active false]
    [suppress_until <text-suppress-ts>]
    [threshold_type user_set | threshold_type vendor_set]
    [threshold <text-threshold-value>]
    [channel <text-channel-name-anchor> | channel drop <text-drop-channel-name>]

Note that the event ID is case-sensitive and must be in all uppercase.

For example, this rule tells NetQ to deliver an event notification to the tca_slack_ifstats pre-configured Slack channel when the CPU utilization exceeds 95% of its capacity on any monitored switch:

cumulus@switch:~$ netq add tca event_id TCA_CPU_UTILIZATION_UPPER scope '*' threshold 95 channel tca_slack_ifstats 

This rule tells NetQ to deliver an event notification to the tca_pd_ifstats PagerDuty channel when the number of transmit bytes per second (Bps) on the leaf12 switch exceeds 20,000 Bps on any interface:

cumulus@switch:~$ netq add tca event_id TCA_TXBYTES_UPPER scope leaf12,'*' threshold 20000 channel tca_pd_ifstats

This rule tells NetQ to deliver an event notification to the syslog-netq syslog channel when the temperature on sensor temp1 on the leaf12 switch exceeds 32 degrees Celcius:

cumulus@switch:~$ netq add tca event_id TCA_SENSOR_TEMPERATURE_UPPER scope leaf12,temp1 threshold 32 channel syslog-netq

This rule tells NetQ to deliver an event notification to the tca-slack channel when the total number of ACL drops on the leaf04 switch exceeds 20,000 for any reason, ingress port, or drop type.

cumulus@switch:~$ netq add tca event_id TCA_WJH_ACL_DROP_AGG_UPPER scope leaf04,'*','*','*' threshold 20000 channel tca-slack

For a Slack channel, the event messages should be similar to this:

Set the Severity of a Threshold-crossing Event

In addition to defining a scope for TCA rule, you can also set a severity of either info or error. To add a severity to a rule, use the severity option.

For example, if you want to add an error severity to the CPU utilization rule you created earlier:

cumulus@switch:~$ netq add tca event_id TCA_CPU_UTILIZATION_UPPER scope '*' severity error threshold 95 channel tca_slack_resources

Or if an event is important, but not an error. Set the severity to info:

cumulus@switch:~$ netq add tca event_id TCA_TXBYTES_UPPER scope leaf12,'*' severity info threshold 20000 channel tca_pd_ifstats 

Set the Threshold for Digital Optics Events

Digital optics have the additional option of applying user- or vendor-defined thresholds, using the threshold_type and threshold options.

This example shows how to send an error to channel ch1 when the upper threshold for module voltage exceeds the vendor-defined thresholds for interface swp31 on the mlx-2700-04 switch.

cumulus@switch:~$ netq add tca event_id TCA_DOM_MODULE_VOLTAGE_ALARM_UPPER scope 'mlx-2700-04,swp31' severity error is_active true threshold_type vendor_set channel ch1
Successfully added/updated tca

This example shows how to send an error to channel ch1 when the upper threshold for module voltage exceeds the user-defined threshold of 3V for interface swp31 on the mlx-2700-04 switch.

cumulus@switch:~$ netq add tca event_id TCA_DOM_MODULE_VOLTAGE_ALARM_UPPER scope 'mlx-2700-04,swp31' severity error is_active true threshold_type user_set threshold 3 channel ch1
Successfully added/updated tca

1. Click Menu and navigate to Threshold crossing rules.

2. Select the relevant tab. The UI displays each rule and its parameters as a card. Each attribute is displayed on the rule card as a regular expression:

• Equals is displayed as an equals sign (=)
• Starts with is displayed as a caret (^)
• Blank (all) is displayed as an asterisk (*)

This example indicates that the rule applies across all interfaces on the exit-1 switch.

After creating a rule, you can use the filters that appear above the rule cards to filter by status, severity, channel, and/or events.

To view TCA rules, run netq show tca:

netq show tca [tca_id <text-tca-id-anchor>] [json]

This example displays all TCA rules:

cumulus@switch:~$ netq show tca
Matching config_tca records:
TCA Name                     Event Name           Scope                      Severity Channel/s          Active Threshold          Unit     Threshold Type Suppress Until
---------------------------- -------------------- -------------------------- -------- ------------------ ------ ------------------ -------- -------------- ----------------------------
TCA_CPU_UTILIZATION_UPPER_1  TCA_CPU_UTILIZATION_ {"hostname":"leaf01"}      info     pd-netq-events,slk True   87                 %        user_set       Fri Oct  9 15:39:35 2020
                             UPPER                                                    -netq-events
TCA_CPU_UTILIZATION_UPPER_2  TCA_CPU_UTILIZATION_ {"hostname":"*"}           error    slk-netq-events    True   93                 %        user_set       Fri Oct  9 15:39:56 2020
                             UPPER
TCA_DOM_BIAS_CURRENT_ALARM_U TCA_DOM_BIAS_CURRENT {"hostname":"leaf*","ifnam error    slk-netq-events    True   0                  mA       vendor_set     Fri Oct  9 16:02:37 2020
PPER_1                       _ALARM_UPPER         e":"*"}
TCA_DOM_RX_POWER_ALARM_UPPER TCA_DOM_RX_POWER_ALA {"hostname":"*","ifname":" info     slk-netq-events    True   0                  mW       vendor_set     Fri Oct  9 15:25:26 2020
_1                           RM_UPPER             *"}
TCA_SENSOR_TEMPERATURE_UPPER TCA_SENSOR_TEMPERATU {"hostname":"leaf","s_name error    slk-netq-events    True   32                 degreeC  user_set       Fri Oct  9 15:40:18 2020
_1                           RE_UPPER             ":"temp1"}
TCA_TCAM_IPV4_ROUTE_UPPER_1  TCA_TCAM_IPV4_ROUTE_ {"hostname":"*"}           error    pd-netq-events     True   20000              %        user_set       Fri Oct  9 16:13:39 2020
                             UPPER

This example displays a specific TCA rule:

cumulus@switch:~$ netq show tca tca_id TCA_TXMULTICAST_UPPER_1
Matching config_tca records:
TCA Name                     Event Name           Scope                      Severity         Channel/s          Active Threshold          Suppress Until
---------------------------- -------------------- -------------------------- ---------------- ------------------ ------ ------------------ ----------------------------
TCA_TXMULTICAST_UPPER_1      TCA_TXMULTICAST_UPPE {"ifname":"swp3","hostname info             tca-tx-bytes-slack True   0                  Sun Dec  8 16:40:14 2269
                             R                    ":"leaf01"}

To modify the threshold:

1. Locate the rule you want to modify and hover over the top of the card.

2. Click Edit.

3. Enter a new threshold value, then select Update rule.

To modify the threshold, run netq add tca:

netq add tca tca_id <text-tca-id-anchor> threshold <text-threshold-value>

This example changes the threshold for the rule TCA_CPU_UTILIZATION_UPPER_1 to a value of 96 percent. This overwrites the existing threshold value.

cumulus@switch:~$ netq add tca tca_id TCA_CPU_UTILIZATION_UPPER_1 threshold 96

To modify the scope:

1. Locate the rule you want to modify and hover over the top of the card.

2. Click Edit.

3. Select the toggle to either apply the rule to the entire network or individual hosts.

4. Click Update rule.

To modify the scope, run:

netq add tca event_id <text-event-id-anchor> scope <text-scope-anchor> threshold <text-threshold-value>

This example changes the scope for the rule TCA_CPU_UTILIZATION_UPPER to apply only to switches beginning with a hostname of leaf. You must also provide a threshold value. This example case uses a value of 95 percent. Note that this overwrites the existing scope and threshold values.

cumulus@switch:~$ netq add tca event_id TCA_CPU_UTILIZATION_UPPER scope hostname^leaf threshold 95
Successfully added/updated tca

cumulus@switch:~$ netq show tca

Matching config_tca records:
TCA Name                     Event Name           Scope                      Severity         Channel/s          Active Threshold          Suppress Until
---------------------------- -------------------- -------------------------- ---------------- ------------------ ------ ------------------ ----------------------------
TCA_CPU_UTILIZATION_UPPER_1  TCA_CPU_UTILIZATION_ {"hostname":"*"}           error            onprem-email       True   93                 Mon Aug 31 20:59:57 2020
                             UPPER
TCA_CPU_UTILIZATION_UPPER_2  TCA_CPU_UTILIZATION_ {"hostname":"hostname^leaf info                                True   95                 Tue Sep  1 18:47:24 2020
                             UPPER                "}

1. Locate the rule you want to modify and hover over the top of the card.

2. Click Edit.

3. Select the Channels tab.

4. Select one or more channels.

5. Click Update rule.

To change a channel association, run:

netq add tca tca_id <text-tca-id-anchor> channel <text-channel-name-anchor>

This overwrites the existing channel association.

This example shows the changing of the channel for the disk utilization 1 rule to a PagerDuty channel pd-netq-events.

cumulus@switch:~$ netq add tca tca_id TCA_DISK_UTILIZATION_UPPER_1 channel pd-netq-events
Successfully added/updated tca TCA_DISK_UTILIZATION_UPPER_1

To remove a channel association (stop sending events to a particular channel), run:

netq add tca tca_id <text-tca-id-anchor> channel drop <text-drop-channel-name>

This example removes the tca_slack_resources channel from the disk utilization 1 rule.

cumulus@switch:~$ netq add tca tca_id TCA_DISK_UTILIZATION_UPPER_1 channel drop tca_slack_resources
Successfully added/updated tca TCA_DISK_UTILIZATION_UPPER_1

1. Locate the rule you want to disable and click Disable.

2. Select the Date/Time field to set when you want the rule to be reenabled.

3. Click Disable.

Using the suppress_until option allows you to prevent the rule from being applied for a designated amout of time (in seconds). When this time has passed, the rule is automatically reenabled.

To suppress a rule, run:

netq add tca tca_id <text-tca-id-anchor> suppress_until <text-suppress-ts>

This example suppresses the maximum cpu utilization event for 24 hours:

cumulus@switch:~$ netq add tca tca_id TCA_CPU_UTILIZATION_UPPER_2 suppress_until 86400
Successfully added/updated tca TCA_CPU_UTILIZATION_UPPER_2

To disable a rule that is currently active:

1. Locate the rule you want to disable.

2. Click Disable.

3. Leave the Date/Time field blank.

4. Click Disable.

To disable a rule that is currently suppressed, click Disable forever.

To disable a rule, run:

netq add tca tca_id <text-tca-id-anchor> is_active false

This example disables the maximum disk utilization 1 rule:

cumulus@switch:~$ netq add tca tca_id TCA_DISK_UTILIZATION_UPPER_1 is_active false
Successfully added/updated tca TCA_DISK_UTILIZATION_UPPER_1

To reenable the rule, set the is_active option to true.

To delete a rule:

1. Locate the rule you want to remove and hover over the card.

2. In the card’s top-right corner, select Delete.

To remove a rule altogether, run:

netq del tca tca_id <text-tca-id-anchor>

This example deletes the maximum receive bytes rule:

cumulus@switch:~$ netq del tca tca_id TCA_RXBYTES_UPPER_1
Successfully deleted TCA TCA_RXBYTES_UPPER_1