Cumulus User Experience - CUE

Cumulus User Experience (CUE) is an early access feature currently in ALPHA and open to customer feedback.

CUE is not currently intended to run in production and is not supported through NVIDIA networking support.

Your evaluation is welcome and appreciated as we start to roll out this new Cumulus Linux CLI. You can provide feedback by sending an email to net-cl-cue-ea-feedback@nvidia.com.

What is CUE?

CUE is an object-oriented, schema driven model of a complete Cumulus Linux system (hardware and software) providing a robust API that allows for multiple interfaces to both view (show) and configure (set and unset) any element within a system running the CUE software. The CUE CLI and the REST API leverage the same API to interface with Cumulus Linux.

CUE follows a declarative model, removing context-specific commands and settings. It is structured as a big tree that represents the entire state of a Cumulus Linux instance. At the base of the tree are high level branches representing objects, such as router and interface. Under each of these branches are further branches. As you navigate through the tree, you gain a more specific context. At the leaves of the tree are actual attributes, represented as key/value pairs. The path through the tree is similar to a filesystem path.

In this ALPHA release of CUE, you have full access to the new CLI, which leverages the underlying CUE API. Future releases will provide access to the API through REST, Python and more.

This documentation describes how to access CUE and navigate the CUE CLI to configure and monitor Cumulus Linux.

Install CUE

CUE is not installed by default on Cumulus Linux. To install CUE, follow the procedure below.

Do not install CUE in a production environment.

Stop then disable the netd service:

cumulus@switch:~$ sudo systemctl stop netd
cumulus@switch:~$ sudo systemctl disable netd

Update the local cache listing of available packages, then install the CUE package:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get install python3-cue

Enable then start the CUE service:

cumulus@switch:~$ sudo systemctl enable cued
cumulus@switch:~$ sudo systemctl start cued

Log out of the switch, then log back in to get the CUE CLI prompt.

Command Line Basics

The CUE command line has a flat structure as opposed to a modal structure. This means that you can run all commands from the primary prompt instead of only in a specific mode.

Command Syntax

CUE commands all begin with cl and fall into one of three syntax categories:

Configuration (cl set and cl unset)
Monitoring (cl show)
Configuration management (cl config).

Command Completion

As you enter commands, you can get help with the valid keywords or options using the Tab key. For example, using Tab completion with cl set displays the possible objects for the command, and returns you to the command prompt to complete the command.

cumulus@switch:~$ cl set <<press Tab>>
bridge     interface  nve        router     vrf
evpn       mlag       platform   system

cumulus@switch:~$ cl set

Command Help

As you enter commands, you can get help with command syntax by entering -h or --help at various points within a command entry. For example, to find out what options are available for cl set interface, enter cl set interface -h or cl set interface --help.

cumulus@switch:~$ cl set interface -h
Usage:
cl set interface <interface-id> ...

Description:
  Interfaces

Identifiers:
  <interface-id>    Interface

General Options:
  -h, --help        Show help.

Command List

You can list all the CUE commands by running cl list-commands. See List All CUE Commands below.

Command History

At the command prompt, press the Up Arrow and Down Arrow keys to move back and forth through the list of commands previously entered. When you find a given command, you can run the command by pressing Enter. Optionally, you can modify the command before you run it.

Command Categories

The CUE CLI has a flat structure; however, the commands are conceptually grouped into three functional categories:

Configuration
Monitoring
Configuration Management

Configuration Commands

The CUE configuration commands modify switch configuration. You can set and unset configuration options.

The cl set and cl unset commands are grouped into the following categories. Each command group includes arguments. Use command completion (Tab key) to list the subcommands.

Command Group	Description
`cl set router` `cl unset router`	Configures router policies, such as prefix list rules and route maps, and global BGP options. This is where you enable and disable BGP, set the ASN and the router ID, and configure BGP graceful restart and shutdown.
`cl set platform` `cl unset platform`	Configures hostname options, such as the static hostname for the switch, the local domain, and whether DHCP is allowed to override the hostname. You can also set how configuration apply operations are performed (such as which files to ignore and which files to overwrite).
`cl set bridge` `cl unset bridge`	Configures a bridge domain. This is where you configure the bridge type (such as VLAN-aware), 802.1Q encapsulation, the STP state and priority, and the VLANs in the bridge domain.
`cl set mlag` `cl unset mlag`	Configures MLAG. This is where you configure the backup IP address or interface, MLAG system MAC address, peer IP address, MLAG priority, and the delay before bonds are brought up.
`cl set evpn` `cl unset evpn`	Configures EVPN. This is where you enable and disable the EVPN control plane, and set EVPN route advertise options, default gateway configuration for centralized routing, and duplicate address detection options.
`cl set interface <interface-id>` `cl unset interface <interface-id>`	Configures the switch interfaces. Use this command to configure bond interfaces, bridge interfaces, interface IP addresses, VLAN IDs, and links (MTU, FEC, speed, duplex, and so on).
`cl set system` `cl unset system`	Configures global system settings, such as NTP, DHCP servers, DNS, LLDP, and syslog.
`cl set vrf <vrf-id>` `cl unset vrf <vrf-id>`	Configures VRFs. This is where you configure VRF-level router configuration such as BGP, including BGP for the default VRF.
`cl set service` `cl unset service`	Configures DHCP relays. This is where you configure the DHCP relay server IP address, the set of interfaces on which to handle DHCP relay traffic, the DHCP relay gateway IP address on the interfaces, and the source IP address to use on the relayed packet.
`cl set nve` `cl unset nve`	Configures network virtualization (VXLAN) settings. This is where you configure the UDP port for VXLAN frames, control dynamic MAC learning over VXLAN tunnels, and configure how Cumulus Linux handles BUM traffic in the overlay.

Monitoring Commands

The CUE monitoring commands show various parts of the network configuration. For example, you can show the complete network configuration or only interface configuration. The monitoring commands are grouped into the following categories. Each command group includes subcommands. Use command completion (Tab key) to list the subcommands.

Command Group	Description
`cl show router`	Shows router configuration, such as router policies and global BGP configuration.
`cl show platform`	Shows platform configuration, such as hardware and software components, and the hostname of the switch.
`cl show bridge`	Shows bridge domain configuration.
`cl show mlag`	Shows MLAG configuration.
`cl show evpn`	Shows EVPN configuration.
`cl show interface`	Shows interface configuration.
`cl show system`	Shows global system settings, such as NTP, DHCP server, DNS, syslog and LLDP.
`cl show service`	Shows DHCP relay configuration, such as the DHCP relay server IP address, the set of interfaces on which DHCP relay traffic is handled, and the DHCP relay gateway IP address on the interfaces.
`cl show vrf`	Shows VRF configuration.
`cl show nve`	Shows network virtualization configuration, such as VXLAN-specfic MLAG configuration and VXLAN flooding.

The following example shows the cl show router commands after pressing the TAB key, then shows the output of the cl show router bgp command.

cumulus@leaf01:mgmt:~$ cl show router <<press Tab>>
bgp     policy
cumulus@leaf01:mgmt:~$ cl show router bgp
                                running  applied      pending      description
------------------------------  -------  -----------  -----------  ----------------------------------------------------------------------
enable                                   on                        Turn the feature 'on' or 'off'.  The default is 'off'.
autonomous-system                        65101                     ASN for all VRFs, if a single AS is in use.  If "none", then ASN mu...
graceful-shutdown                        off                       Graceful shutdown enable will initiate the GSHUT community to be an...
policy-update-timer                      5                         Wait time in seconds before processing updates to policies to ensur...
router-id                                10.10.10.1                BGP router-id for all VRFs, if a common one is used.  If "none", th...
convergence-wait
  establish-wait-time                    0                         Maximum time to wait to establish BGP sessions. Any peerswhich do...
  time                                   0                         Time to wait for peers to send end-of-RIB before router performs pa...
graceful-restart
  mode                                   helper-only               Role of router during graceful restart. helper-only, router is in h...
  path-selection-deferral-time           360                       Used by the restarter as an upper-bounds for waiting for peering es...
  restart-time                           120                       Amount of time taken to restart by router. It is advertised to the...
  stale-routes-time                      360                       Specifies an upper-bounds on how long we retain routes from a ....

If there are no pending or applied configuration changes, the cl show command only shows the running configuration.

Revision options are available for the cl show commands. You can choose the configuration you want to show (pending, applied, startup, or running):

Option	Description
`--rev <revision>`	Shows a detached pending configuration. See the `cl config detach` configuration management command below.
`--pending`	Shows the configuration you `set` and `unset` but have not yet applied or saved.
`--applied`	Shows the last set of commands applied with the `cl config apply` command.
`--startup`	Shows the set of commands saved with the `cl config save` command. This will be the configuration after the switch boots.
`--running`	Shows the running configuration (the actual system state). The running and applied configuration should be the same. If different, inspect the logs.

The following example shows pending BGP graceful restart configuration:

cumulus@switch:~$ cl show router bgp graceful-restart --pending
                             pending_20210128_212626_4WSY  description
----------------------------  ----------------------------  ----------------------------------------------------------------------
mode                          helper-only                   Role of router during graceful restart. helper-only, router is in h...
path-selection-deferral-time  360                           Used by the restarter as an upper-bounds for waiting for peeringes...
restart-time                  120                           Amount of time taken to restart by router. It is advertised to the...
stale-routes-time             360                           Specifies an upper-bounds on how long we retain routes from a resta...

Configuration Management Commands

The CUE configuration management commands manage and apply configurations.

Command	Description
`cl config apply`	Applies the pending configuration to become the applied configuration. You can also use these prompt options: `--y` or `--assume-yes` to automatically reply `yes` to all prompts. `--assume-no` to automatically reply `no` to all prompts. The configuration is applied but not saved and does not persist after a reboot.
`cl config detach`	Detaches the configuration from the current pending configuration. The detached configuration is called `pending` and includes a timestamp with extra characters. For example: `pending_20210128_212626_4WSY`
`cl config diff <revision> <revision>`	Shows differences between configurations, such as the pending configuration and the applied configuration or the detached configuration and the pending configuration.
`cl config patch <cue-file>`	Updates the pending configuration with the specified YAML configuration file.
`cl config replace <cue-file>`	Replaces the pending configuration with the specified YAML configuration file.
`cl config save`	Overwrites the startup configuration with the applied configuration by writing to the `/etc/cue.d/startup.yaml` file. The configuration persists after a reboot.

List all CUE Commands

To show the full list of CUE commands, run cl list-commands. For example:

cumulus@switch:~$ cl list-commands
...
cl show interface <interface-id> link lldp neighbor
cl show interface <interface-id> link lldp neighbor <neighbor-id>
cl show interface <interface-id> link lldp neighbor <neighbor-id> bridge
cl show interface <interface-id> link lldp neighbor <neighbor-id> bridge vlan
cl show interface <interface-id> link lldp neighbor <neighbor-id> bridge vlan <vid>
cl show interface <interface-id> link stats
cl show system
cl show system global
cl show system ntp
cl show system ntp server
cl show system ntp server <server-id>
cl show system ntp pool
cl show system ntp pool <server-id>
cl show system dhcp-server
...

You can show the list of commands for a command grouping and for subcommands. For example, to show the list of interface commands:

cumulus@switch:~$ cl list-commands interface
cl show interface
cl show interface <interface-id>
cl show interface <interface-id> bond
cl show interface <interface-id> bond member
cl show interface <interface-id> bond member <member-id>
cl show interface <interface-id> bond mlag
cl show interface <interface-id> bridge
cl show interface <interface-id> bridge domain
cl show interface <interface-id> bridge domain <domain-id>
cl show interface <interface-id> bridge domain <domain-id> stp
cl show interface <interface-id> bridge domain <domain-id> vlan
cl show interface <interface-id> bridge domain <domain-id> vlan <vid>
cl show interface <interface-id> ip
...

Use the Tab key to get help for the command lists you want to see. For example, to show the list of command options available for the interface swp1, run:

cumulus@switch:~$ cl list-commands interface swp1 <<press Tab>>
bond    bridge  ip      link
cumulus@switch:~$ cl list-commands interface swp1 bond
cl show interface <interface-id> bond
cl show interface <interface-id> bond member
cl show interface <interface-id> bond member <member-id>
cl show interface <interface-id> bond mlag
cl set interface <interface-id> bond
cl set interface <interface-id> bond member <member-id>
cl set interface <interface-id> bond mlag
cl set interface <interface-id> bond mlag id (1-65535|auto)
cl set interface <interface-id> bond down-delay 0-65535
cl set interface <interface-id> bond lacp-bypass (on|off)
cl set interface <interface-id> bond lacp-rate (fast|slow)
cl set interface <interface-id> bond mode (lacp|static)
cl set interface <interface-id> bond up-delay 0-65535
cl unset interface <interface-id> bond
cl unset interface <interface-id> bond member
cl unset interface <interface-id> bond member <member-id>
cl unset interface <interface-id> bond mlag
cl unset interface <interface-id> bond mlag id
cl unset interface <interface-id> bond down-delay
cl unset interface <interface-id> bond lacp-bypass
cl unset interface <interface-id> bond lacp-rate
cl unset interface <interface-id> bond mode
cl unset interface <interface-id> bond up-delay

Example Configuration Commands

This section provides examples of how to configure a Cumulus Linux switch using CUE commands.

Configure the System Hostname

The example below shows the CUE commands required to change the hostname for the switch to leaf01:

cumulus@switch:~$ cl set platform hostname value leaf01
cumulus@switch:~$ cl config apply

Configure the System DNS Server

The example below shows the CUE commands required to define the DNS server for the switch:

cumulus@switch:~$ cl set system dns server 192.168.200.1
cumulus@switch:~$ cl config apply

Configure an Interface

The example below shows the CUE commands required to bring up swp1.

cumulus@switch:~$ cl set interface swp1 link state up
cumulus@switch:~$ cl config apply

Configure a Bond

The example below shows the CUE commands required to configure the front panel port interfaces swp1 thru swp4 to be slaves in bond0.

cumulus@switch:~$ cl set interface bond0 bond member swp1-4
cumulus@switch:~$ cl config apply

Configure a Bridge

The example below shows the CUE commands required to create a VLAN-aware bridge that contains two switch ports (swp1 and swp2) and includes 3 VLANs; tagged VLANs 10 and 20 and an untagged (native) VLAN of 1.

With CUE, there is a default bridge called br_default, which has no ports assigned to it. The example below configures this default bridge.

cumulus@switch:~$ cl set interface swp1-2 bridge domain br_default
cumulus@switch:~$ cl set bridge domain br_default vlan 10,20
cumulus@switch:~$ cl set bridge domain br_default untagged 1
cumulus@switch:~$ cl config apply

Configure MLAG

The example below shows the CUE commands required to configure MLAG on leaf01. The commands:

Place swp1 into bond1 and swp2 into bond2.
Configure the MLAG ID to 1 for bond1 and to 2 for bond2.
Add bond1 and bond2 to the default bridge (br_default).
Create the inter-chassis bond (swp49 and swp50) and the peer link (peerlink)
Set the peer link IP address to linklocal, the MLAG system MAC address to 44:38:39:BE:EF:AA, and the backup interface to 10.10.10.2.

cumulus@leaf01:~$ cl set interface bond1 bond member swp1
cumulus@leaf01:~$ cl set interface bond2 bond member swp2
cumulus@leaf01:~$ cl set interface bond1 bond mlag id 1
cumulus@leaf01:~$ cl set interface bond2 bond mlag id 2
cumulus@switch:~$ cl set interface bond1-2 bridge domain br_default 
cumulus@leaf01:~$ cl set interface peerlink bond member swp49-50
cumulus@leaf01:~$ cl set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:~$ cl set mlag backup 10.10.10.2
cumulus@leaf01:~$ cl set mlag peer-ip linklocal
cumulus@leaf01:~$ cl config apply

Configure BGP Unnumbered

The example below shows the CUE commands required to configure BGP unnumbered on leaf01. The commands:

Assign the ASN for this BGP node to 65101.
Set the router ID to 10.10.10.1.
Distribute routing information to the peer on swp51.
Originate prefixes 10.10.10.1/32 from this BGP node.

cumulus@leaf01:~$ cl set router bgp autonomous-system 65101
cumulus@leaf01:~$ cl set router bgp router-id 10.10.10.1
cumulus@leaf01:~$ cl set vrf default router bgp peer swp51 remote-as external
cumulus@leaf01:~$ cl set vrf default router bgp address-family ipv4-unicast static-network 10.10.10.1/32
cumulus@leaf01:~$ cl config apply

Example Monitoring Commands

This section provides monitoring command examples.

Show Installed Software

The following example command lists the software installed on the switch:

cumulus@switch:~$ cl show platform software
             running                                applied  description
-----------  -------------------------------------  -------  --------------------------
[installed]  acpi                                            List of installed software
[installed]  acpi-support-base
[installed]  acpid
[installed]  adduser
[installed]  apt
[installed]  arping
[installed]  arptables
[installed]  auditd
[installed]  base-files
[installed]  base-passwd
[installed]  bash
[installed]  bash-completion
[installed]  bind9-host
[installed]  binutils
[installed]  binutils-common
[installed]  binutils-x86-64-linux-gnu
...

Show Interface Configuration

The following example command shows the running and applied swp1 interface configuration. There is no pending configuration.

cumulus@leaf01:~$ cl show interface swp1
                        running     applied      description
-----------------------  ----------  -----------  ----------------------------------------------------------------------
type                     swp         swp          The type of interface
bridge
  [domain]               br_default  br_default   Bridge domains on this interface
  [domain]                           bridge
ip
  vrf                                default      Virtual routing and forwarding
  ipv4                               forward      IPv4 support on the interface. A value of 'on' means IPv4 is enable...
  ipv6                               forward      IPv6 support on the interface. A value of 'on' means IPv6 is enable...
  [address]                          10.1.1.1/30  ipv4 and ipv6 address
link
  auto-negotiate                     on           Link speed and characteritic auto negotiation
  breakout                           1x           sub-divide, aggregate, or disable ports (only valid on plug interfa...
  duplex                             full         Link duplex
  fec                                auto         Link forward error correction mechanism
  mtu                    9216        9216         interface mtu
  speed                              auto         Link speed
  dot1x
    mab                              off          bypass MAC authentication
    parking-vlan                     off          VLAN for unauthorized MAC addresses
  state                  down        up           The state of the interface
  stats
    carrier-transitions  3                        Number of times the interface state has transitioned between up and...
    in-bytes             0                        total number of bytes received on the interface
    in-drops             0                        number of received packets dropped
    in-errors            0                        number of received packets with errors
    in-pkts              0                        total number of packets received on the interface
    out-bytes            65700                    total number of bytes transmitted out of the interface
    out-drops            0                        The number of outbound packets that were chosen to be discarded eve...
    out-errors           0                        The number of outbound packets that could not be transmitted becaus...
    out-pkts             934                      total number of packets transmitted out of the interface

Example Configuration Management Commands

This section provides examples of how to use the configuration management commands to apply, save, and detach configurations.

Apply and Save a Configuration

The following example command configures the front panel port interfaces swp1 thru swp4 to be slaves in bond0. The configuration is only in a pending configuration state. The configuration is not applied. CUE has not yet made any changes to the running configuration.

cumulus@switch:~$ cl set interface bond0 bond member swp1-4

To apply the pending configuration to the running configuration, run the cl config apply command. The configuration does not persist after a reboot.

cumulus@switch:~$ cl config apply

To save the applied configuration to the startup configuration, run the cl config save command. This command overwrites the startup configuration with the applied configuration by writing to the /etc/cue.d/startup.yaml file. The configuration persists after a reboot.

cumulus@switch:~$ cl config save

Detach a Pending Configuration

The following example configures the IP address of the loopback interface, then detaches the configuration from the current pending configuration. The detached configuration is saved to a file called pending that includes a timestamp with extra characters to distinguish it from other pending configurations; for example, pending_20210128_212626_4WSY.

cumulus@switch:~$ cl set interface lo ip address 10.10.10.1
cumulus@switch:~$ cl config detach

View Differences between Configurations

To view differences between configurations, run the cl config diff command.

To view differences between two detached pending configurations, run the cl config diff «TAB» command to list all the current detached pending configurations, then run the cl config diff command with the pending configurations you want to diff:

cumulus@switch:~$ cl config diff <<press Tab>>
cumulus@switch:~$ cl config diff
applied                              pending_20210208_201140_MJ0V
pending_20210208_195315_MJ0P         pending_20210208_204655_MJ12
pending_20210208_195937_MJ0S         startup

cumulus@switch:~$ cl config diff pending_20210208_201140_MJ0V pending_20210208_195315_MJ0P

To view differences between a detached pending configuration and the applied configuration:

cumulus@switch:~$ cl config diff pending_20210208_201140_MJ0V applied

Replace and Patch a Pending Configuration

The following example replaces the pending configuration with the contents of the YAML configuration file called cl-02/13/2021.yaml located in the /deps directory:

cumulus@switch:~$ cl config replace /deps/cl-02/13/2021.yaml

The following example patches the pending configuration (runs the set or unset commands from the configuration in the cl-02/13/2021.yaml file located in the /deps directory):

cumulus@switch:~$ cl config patch /deps/cl-02/13/2021.yaml

How is CUE Different from NCLU?

This section lists some of the differences between CUE and the NCLU command line interface to help you navigate configuration.

Configuration File

When you save network configuration using CUE, the configuration is written to the /etc/cue.d/startup.yaml file.

CUE also writes to underlying Linux files when you apply a configuration, such as the /etc/network/interfaces and /etc/frr/frr.conf files. You can view these configuration files; however NVIDIA recommends that you do not manually edit them while using CUE.

Bridge Configuration

You set global bridge configuration on the bridge domain. For example:

cumulus@leaf01:~$ cl set bridge domain br_default vlan 10,20

However, you set specific bridge interface options with interface commands. For example:

cumulus@leaf01:~$ cl set interface swp1 bridge domain br_default learning on

The default vlan-aware bridge in CUE is br_default. The default vlan-aware bridge in NCLU is bridge.

BGP Configuration

You can set global BGP configuration, such as the ASN, router ID, graceful shutdown and restart with the cl set router bgp command. For example:

cumulus@leaf01:~$ cl set router bgp autonomous-system 65101

However, BGP peer and peer group, route information, timer, and address family configuration requires a VRF. For example:

cumulus@leaf01:~$ cl set vrf default router bgp peer swp51 remote-as external