gNMI Streaming

You can use gRPC Network Management Interface (gNMI) to collect system resource, interface, and counter information from Cumulus Linux and export it to your own gNMI client.

Configure the gNMI Agent

The gNMI agent is disabled by default. To enable it, run:

 cumulus@switch:~$ netq config add agent gnmi-enable true

The gNMI agent listens over port 9339. You can change the default port in case you use that port in another application. The /etc/netq/netq.yml file stores the configuration.

Use the following commands to adjust the settings:

  1. Disable the gNMI agent:

    cumulus@switch:~$ netq config add agent gnmi-enable false
    
  2. Change the default port over which the gNMI agent listens:

    cumulus@switch:~$ netq config add agent gnmi-port <gnmi_port>
    
  3. Restart the NetQ agent to incorporate the configuration changes:

    cumulus@switch:~$ netq config restart agent
    

Use the gNMI Agent Only

NVIDIA recommends collecting data with both the gNMI and NetQ agents. However, if you do not want to collect data with both agents, you can disable the NetQ agent. Data is then sent exclusively to the gNMI agent.

To disable the NetQ agent, use the following command:

cumulus@switch:~$ netq config add agent opta-enable false

You cannot disable both the NetQ and gNMI agents. If both agents are enabled on Cumulus Linux and a NetQ server is unreachable, the data from the following models are not sent to gNMI:

  • openconfig-interfaces
  • openconfig-if-ethernet
  • openconfig-if-ethernet-ext
  • openconfig-system
  • nvidia-if-ethernet-ext

WJH, openconfig-platform, and openconfig-lldp data continue streaming to gNMI in this state. If you are only using gNMI and a NetQ telemetry server does not exist, you should disable the NetQ agent by setting opta-enable to false.

Supported Models

Cumulus Linux supports the following OpenConfig models:

Model Supported Data
openconfig-interfaces Name, Operstatus, AdminStatus, IfIndex, MTU, LoopbackMode, Enabled, Counters (InPkts, OutPkts, InOctets, InUnicastPkts, InDiscards, InMulticastPkts, InBroadcastPkts, InErrors, OutOctets, OutUnicastPkts, OutMulticastPkts, OutBroadcastPkts, OutDiscards, OutErrors)
openconfig-if-ethernet AutoNegotiate, PortSpeed, MacAddress, NegotiatedPortSpeed, Counters (InJabberFrames, InOversizeFrames,​ InUndersizeFrames)
openconfig-if-ethernet-ext Frame size counters (InFrames_64Octets, InFrames_65_127Octets, InFrames_128_255Octets, InFrames_256_511Octets, InFrames_512_1023Octets, InFrames_1024_1518Octets)
openconfig-system Memory, CPU
openconfig-platform Platform data (Name, Description, Version)
openconfig-lldp LLDP data (PortIdType, PortDescription, LastUpdate, SystemName, SystemDescription, ChassisId, Ttl, Age, ManagementAddress, ManagementAddressType, Capability)

gNMI clients can also use the following NVIDIA models:

Model Supported Data
nvidia-if-wjh-drop-aggregate Aggregated WJH drops, including L1, L2, router, ACL, tunnel, and buffer drops
nvidia-if-ethernet-ext Extended Ethernet counters (AlignmentError, InAclDrops, InBufferDrops, InDot3FrameErrors, InDot3LengthErrors, InL3Drops, InPfc0Packets, InPfc1Packets, InPfc2Packets, InPfc3Packets, InPfc4Packets, InPfc5Packets, InPfc6Packets, InPfc7Packets, OutNonQDrops, OutPfc0Packets, OutPfc1Packets, OutPfc2Packets, OutPfc3Packets, OutPfc4Packets, OutPfc5Packets, OutPfc6Packets, OutPfc7Packets, OutQ0WredDrops, OutQ1WredDrops, OutQ2WredDrops, OutQ3WredDrops, OutQ4WredDrops, OutQ5WredDrops, OutQ6WredDrops, OutQ7WredDrops, OutQDrops, OutQLength, OutWredDrops, SymbolErrors, OutTxFifoFull)

The client should use the following YANG models as a reference:

nvidia-if-ethernet-ext
nvidia-if-wjh-drop-aggregate

Collect WJH Data Using gNMI

You can export What Just Happened data from the NetQ agent to your own gNMI client. Refer to the previous section for the nvidia-if-wjh-drop-aggregate reference YANG model.

Supported Features

  • The gNMI agent supports capability and stream subscribe requests for WJH events.
  • If you are using SONiC, WJH data can only be collected using gNMI.

WJH Drop Reasons

The data NetQ sends to the gNMI agent is in the form of WJH drop reasons. The reasons are generated by the SDK and are stored in the /usr/etc/wjh_lib_conf.xml file on the switch. Use this file as a guide to filter for specific reason types (L1, ACL, and so forth), reason IDs, or event severities.

L1 Drop Reasons

Reason ID Reason Description
10021 Port admin down Validate port configuration
10022 Auto-negotiation failure Set port speed manually, disable auto-negotiation
10023 Logical mismatch with peer link Check cable/transceiver
10024 Link training failure Check cable/transceiver
10025 Peer is sending remote faults Replace cable/transceiver
10026 Bad signal integrity Replace cable/transceiver
10027 Cable/transceiver is not supported Use supported cable/transceiver
10028 Cable/transceiver is unplugged Plug cable/transceiver
10029 Calibration failure Check cable/transceiver
10030 Cable/transceiver bad status Check cable/transceiver
10031 Other reason Other L1 drop reason

L2 Drop Reasons

Reason ID Reason Severity Description
201 MLAG port isolation Notice Expected behavior
202 Destination MAC is reserved (DMAC=01-80-C2-00-00-0x) Error Bad packet was received from the peer
203 VLAN tagging mismatch Error Validate the VLAN tag configuration on both ends of the link
204 Ingress VLAN filtering Error Validate the VLAN membership configuration on both ends of the link
205 Ingress spanning tree filter Notice Expected behavior
206 Unicast MAC table action discard Error Validate MAC table for this destination MAC
207 Multicast egress port list is empty Warning Validate why IGMP join or multicast router port does not exist
208 Port loopback filter Error Validate MAC table for this destination MAC
209 Source MAC is multicast Error Bad packet was received from peer
210 Source MAC equals destination MAC Error Bad packet was received from peer

Router Drop Reasons

Reason ID Reason Severity Description
301 Non-routable packet Notice Expected behavior
302 Blackhole route Warning Validate routing table for this destination IP
303 Unresolved neighbor/next hop Warning Validate ARP table for the neighbor/next hop
304 Blackhole ARP/neighbor Warning Validate ARP table for the next hop
305 IPv6 destination in multicast scope FFx0:/16 Notice Expected behavior - packet is not routable
306 IPv6 destination in multicast scope FFx1:/16 Notice Expected behavior - packet is not routable
307 Non-IP packet Notice Destination MAC is the router, packet is not routable
308 Unicast destination IP but multicast destination MAC Error Bad packet was received from the peer
309 Destination IP is loopback address Error Bad packet was received from the peer
310 Source IP is multicast Error Bad packet was received from the peer
311 Source IP is in class E Error Bad packet was received from the peer
312 Source IP is loopback address Error Bad packet was received from the peer
313 Source IP is unspecified Error Bad packet was received from the peer
314 Checksum or IPver or IPv4 IHL too short Error Bad cable or bad packet was received from the peer
315 Multicast MAC mismatch Error Bad packet was received from the peer
316 Source IP equals destination IP Error Bad packet was received from the peer
317 IPv4 source IP is limited broadcast Error Bad packet was received from the peer
318 IPv4 destination IP is local network (destination=0.0.0.0/8) Error Bad packet was received from the peer
320 Ingress router interface is disabled Warning Validate your configuration
321 Egress router interface is disabled Warning Validate your configuration
323 IPv4 routing table (LPM) unicast miss Warning Validate routing table for this destination IP
324 IPv6 routing table (LPM) unicast miss Warning Validate routing table for this destination IP
325 Router interface loopback Warning Validate the interface configuration
326 Packet size is larger than router interface MTU Warning Validate the router interface MTU configuration
327 TTL value is too small Warning Actual path is longer than the TTL

Tunnel Drop Reasons

Reason ID Reason Severity Description
402 Overlay switch - Source MAC is multicast Error The peer sent a bad packet
403 Overlay switch - Source MAC equals destination MAC Error The peer sent a bad packet
404 Decapsulation error Error The peer sent a bad packet

ACL Drop Reasons

Reason ID Reason Severity Description
601 Ingress port ACL Notice Validate ACL configuration
602 Ingress router ACL Notice Validate ACL configuration
603 Egress router ACL Notice Validate ACL configuration
604 Egress port ACL Notice Validate ACL configuration

Buffer Drop Reasons

Reason ID Reason Severity Description
503 Tail drop Warning Monitor network congestion
504 WRED Warning Monitor network congestion
505 Port TC congestion threshold crossed Notice Monitor network congestion
506 Packet latency threshold crossed Notice Monitor network congestion

gNMI Client Requests

You can use your gNMI client on a host server to request capabilities and data that the agent is subscribed to.

The following example shows a gNMI client request for interface speed:

gnmi_client -target_addr 10.209.37.121:9339 -xpath "/interfaces/interface[name=swp1]/ethernet/state/port-speed" -once
{
   "Response": {
      "Update": {
         "update": [
            {
               "val": {
                  "Value": {
                     "StringVal": "SPEED_40GB"
                  }
               },
               "path": {
                  "elem": [
                     {
                        "name": "state"
                     },
                     {
                        "name": "port-speed"
                     }
                  ]
               }
            }
         ],
         "timestamp": 1636910588085654861,
         "prefix": {
            "target": "netq",
            "elem": [
               {
                  "name": "interfaces"
               },
               {
                  "name": "interface",
                  "key": {
                     "name": "swp1"
                  }
               },
               {
                  "name": "ethernet"
               }
            ]
         }
      }
   }
}


The following example shows a gNMI client request for WJH drop data:

gnmi_client -target_addr 10.209.37.121:9339 -xpath "/interfaces/interface[name=swp8]/wjh/aggregate/l2/reasons/reason[id=210]"
{
   "Response": {
      "Update": {
         "update": [
            {
               "val": {
                  "Value": {
                     "StringVal": "[{
									  "IngressPort": "swp8",
									  "DropType": "L2",
									  "Reason": "Source MAC equals destination MAC",
									  "Severity": "Error",
									  "Smac": "00:02:10:00:00:01",
									  "Dmac": "00:02:10:00:00:01",
									  "Proto": 6,
									  "Sport": 15,
									  "Dport": 16,
									  "Sip": "1.1.1.1"
									  "Dip": "2.2.2.2",
									  "AggCount": 192,
									  "FirstTimestamp": 1636907412,
									  "EndTimestamp": 1636907432,
								   }]"

                  }
               },
               "path": {
                  "elem": [
                     {
                        "name": "state"
                     },
                     {
                        "name": "drop"
                     }
                  ]
               }
            }
         ],
         "prefix": {
            "elem": [
               {
                  "name": "interfaces"
               },
               {
                  "key": {
                     "name": "swp8"
                  },
                  "name": "interface"
               },
               {
                  "name": "wjh"
               },
               {
                  "name": "aggregate"
               },
               {
                  "name": "l2"
               },
               {
                  "name": "reasons"
               },
               {
                  "key" : {
                     "severity": "error",
                     "id": "210"
                  },
                  "name" : "reason"
               }
            ],
            "target": "netq"
         },
         "timestamp": 1636907442362981645
      }
   }
}