layout-3 layout-6 layout-1 layout-top
NVIDIA Docs Hubchevron_right NVIDIA Networkingchevron_right Networking Softwarechevron_right Switch Softwarechevron_right

Cumulus Linux

layout-top layout-6 layout-1

Upgrading Cumulus Linux

You can upgrade Cumulus Linux in one of two ways:

  • Install a new Cumulus Linux image with ONIE.
  • Upgrade only changed packages with package upgrade.

Cumulus Linux also provides ISSU to upgrade an active switch with minimal disruption to the network. See In-Service-System-Upgrade-ISSU.

Before You Upgrade

Package upgrade do not overwrite configuration files on the switch, however upgrading Cumulus Linux with ONIE is destructive and any configuration files on the switch are not saved. Before you start an upgrade with ONIE, back up configuration files to a different server.

For troubleshooting any upgrade issues, create a cl-support file before you start and after you complete the upgrade.

Back up Configuration Files

Understanding the location of configuration data is important for successful upgrades, migrations, and backup. As with other Linux distributions, the /etc directory is the primary location for all configuration data in Cumulus Linux. The following list contains the files you need to back up and migrate to a new release. Make sure you examine any changed files. Make the following files and directories part of a backup strategy.

▼ Network Configuration Files
▼ Commonly Used Files
▼ Never Migrate Files

To show a list of files changed from the previous Cumulus Linux install, run the sudo dpkg --verify command. To show a list of generated /etc/default/isc-* files changed from the previous Cumulus Linux install, run the egrep -v '^$|^#|=""$' /etc/default/isc-dhcp-* command.

Back Up and Restore Configuration with NVUE

You can back up and restore the configuration file with NVUE only if you used NVUE commands to configure the switch you want to upgrade.

To back up and restore the configuration file:

  1. Save the configuration to the /etc/nvue.d/startup.yaml file with the nv config save command:

    cumulus@switch:~$ nv config save
saved

  2. Copy the /etc/nvue.d/startup.yaml file off the switch to a different location.

  3. After upgrade is complete, restore the configuration. Copy the /etc/nvue.d/startup.yaml file to the switch, run the nv config patch command, then run the nv config apply command. In the following example startup.yaml is in the /home/cumulus directory on the switch:

    cumulus@switch:~$ nv config patch /home/cumulus/startup.yaml
cumulus@switch:~$ nv config apply

When you restore an NVUE configuration file that includes TACACS, you see an unrecoverable error when running additional NVUE commands. To work around this issue, restart the NVUE service with the systemctl restart nvued.service command.

For information about the NVUE object model and commands, see NVIDIA User Experience - NVUE.

As NVUE supports more features and introduces new syntax, snippets and flexible snippets become invalid.

Before you upgrade Cumulus Linux to a new release, make sure to:

  • Review the What's New for new NVUE syntax.
  • If NVUE introduces new syntax for the feature that a snippet configures, you must remove the snippet before upgrading.

Create a cl-support File

Before and after you upgrade the switch, run the cl-support script to create a cl-support archive file. The file is a compressed archive of useful information for troubleshooting. If you experience any issues during upgrade, you can send this archive file to the Cumulus Linux support team to investigate.

  1. Create the cl-support archive file with either the NVUE nv action generate system tech-support command or the Linux sudo cl-support command:
cumulus@switch:~$ nv action generate system tech-support

  1. Copy the cl-support file off the switch to a different location.

  2. After upgrade is complete, create a new archive file:

cumulus@switch:~$ nv action generate system tech-support

Image Upgrade

ONIE is an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on a bare metal switch.

  • Installing a Cumulus Linux image with ONIE is destructive; any configuration files on the switch are not saved; copy them to a different server before you start the Cumulus Linux image install.
  • You must move configuration data to the new network operating system using ZTP or automation while the operating system is first booted, or soon afterwards using out-of-band management.
  • Moving a configuration file can cause issues.
  • Identifying all the locations that include configuration data is not always an easy task. See Before You Upgrade Cumulus Linux above.
  • Merge conflicts with configuration file changes in the new release sometimes go undetected.
  • If configuration files do not restore correctly, you cannot ssh to the switch from in-band management. Use out-of-band connectivity (eth0 or the console).
  • You must reinstall and reconfigure third-party applications after upgrade.

Upgrading an MLAG pair requires additional steps. If you are using MLAG to dual connect two Cumulus Linux switches in your environment, follow the steps in Upgrade Switches in an MLAG Pair below to ensure a smooth upgrade.

To upgrade the switch:

  1. Back up the configurations off the switch.

  2. Download the Cumulus Linux image.

  3. Install the Cumulus Linux image with the onie-install -a -i <image-location> command, which boots the switch into ONIE. The following example command installs the image from a web server, then reboots the switch. There are additional ways to install the Cumulus Linux image, such as using FTP, a local file, or a USB drive. For more information, see Installing a New Cumulus Linux Image.

    cumulus@switch:~$ sudo onie-install -a -i http://10.0.1.251/cumulus-linux-5.11.0-mlx-amd64.bin && sudo reboot

  4. Restore the configuration files to the new release (NVIDIA does not recommend restoring files with automation).

  5. Verify correct operation with the old configurations on the new release.

  6. Reinstall third party applications and associated configurations.

Package Upgrade

Cumulus Linux completely embraces the Linux and Debian upgrade workflow, where you use an installer to install a base image, then perform any upgrades within that release train with package upgrade. Any packages that have been changed since the base install get upgraded in place from the repository. All switch configuration files remain untouched, or in rare cases merged during the package upgrade.

When you use package upgrade to upgrade the switch, configuration data stays in place during the upgrade. If the new release updates a previously changed configuration file, the upgrade process prompts you to either specify the version you want to use or evaluate the differences.

Upgrading an MLAG pair requires additional steps. If you are using MLAG to dual connect two Cumulus Linux switches in your environment, follow the steps in Upgrade Switches in an MLAG Pair below to ensure a smooth upgrade.

  • You cannot upgrade the switch to a new release train. For example, you cannot use package upgrade to upgrade the switch from 4.x to 5.x.
  • Package upgrade only supports the current version plus two. For example, you can upgrade from Cumulus Linux 5.9 to Cumulus Linux 5.11 with package upgrade only if you installed a 5.9 binary image.
  • Package upgrade always updates to the latest available release in the Cumulus Linux repository. For example, if you are currently running Cumulus Linux 5.9 and perform a package upgrade, the packages upgrade to the latest 5.11 release.
  • The package upgrade command might restart or stop services as part of the upgrade process.
  • The package upgrade command might disrupt core services by changing core service dependency packages.
  • After you upgrade, account UIDs and GIDs created by packages might be different on different switches, depending on the configuration and package installation history.
  • Cumulus Linux does not support the Linux sudo -E apt-get dist-upgrade command. Be sure to use sudo -E apt-get upgrade when upgrading packages.
  • To upgrade from Cumulus Linux 5.9 or 5.10 to Cumulus Linux 5.11, you need 0.8GB of free disk space. Before you upgrade, run the NVUE nv show system disk usage command or the Linux sudo df -h command to show how much disk space you are currently using on the switch.

Upgrade from Cumulus Linux 5.9

If you are running Cumulus Linux 5.9 (the current extended-support release), the default switch configuration allows you to upgrade to the latest Cumulus 5.9 maintenance release only.

To upgrade from Cumulus Linux 5.9 to the latest Cumulus Linux 5 release, perform the following procedure before you start the package upgrade:

  1. Edit the /etc/apt/sources.list file to include the following lines at the top of the file.

    cumulus@switch:~$ sudo nano /etc/apt/sources.list
deb      https://apt.cumulusnetworks.com/repo CumulusLinux-d12-latest cumulus upstream netq
deb-src  https://apt.cumulusnetworks.com/repo CumulusLinux-d12-latest cumulus upstream netq

  2. Remove or comment out the following lines in the /etc/apt/sources.list file:

    deb      https://apt.cumulusnetworks.com/repo CumulusLinux-5.9-latest cumulus upstream netq
deb-src  https://apt.cumulusnetworks.com/repo CumulusLinux-5.9-latest cumulus upstream netq

To upgrade the switch with package upgrade:

  1. Back up the configurations from the switch.

  2. Fetch the latest update metadata from the repository and review potential upgrade issues (in some cases, upgrading new packages might also upgrade additional existing packages due to dependencies).

    cumulus@switch:~$ nv action upgrade system packages to latest use-vrf default dry-run

    By default, the NVUE nv action upgrade system packages command runs in the management VRF. To run the command in a non-management VRF such as default, you must use the use-vrf <vrf> option.

  3. Upgrade all the packages to the latest distribution.

    cumulus@switch:~$ nv action upgrade system packages to latest use-vrf default

    By default, the NVUE nv action upgrade system packages command runs in the management VRF. To run the command in a non-management VRF such as default, you must use the use-vrf <vrf> option.

    If you see errors for expired GPG keys that prevent you from upgrading packages, follow the steps in Upgrading Expired GPG Keys.

  4. After the upgrade completes, check if you need to reboot the switch, then reboot the switch if required:

    cumulus@switch:~$ nv show system reboot required
yes
cumulus@switch:~$ nv action reboot system

  5. Verify correct operation with the old configurations on the new version.

  1. Back up the configurations from the switch.

  2. Fetch the latest update metadata from the repository.

    cumulus@switch:~$ sudo -E apt-get update

  3. Review potential upgrade issues (in some cases, upgrading new packages might also upgrade additional existing packages due to dependencies).

    cumulus@switch:~$ sudo -E apt-get upgrade --dry-run

  4. Upgrade all the packages to the latest distribution.

    cumulus@switch:~$ sudo -E apt-get upgrade

    If you do not need to reboot the switch after the upgrade completes, the upgrade ends, restarts all upgraded services, and logs messages in the /var/log/syslog file similar to the ones shown below. In the examples below, the process only upgrades the frr package.

    Policy: Service frr.service action stop postponed
Policy: Service frr.service action start postponed
Policy: Restarting services: frr.service
Policy: Finished restarting services
Policy: Removed /usr/sbin/policy-rc.d
Policy: Upgrade is finished

    If the upgrade process encounters changed configuration files that have new versions in the release to which you are upgrading, you see a message similar to this:

    Configuration file '/etc/frr/daemons'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** daemons (Y/I/N/O/D/Z) [default=N] ?
    • To see the differences between the currently installed version and the new version, type D.
    • To keep the currently installed version, type N. The new package version installs with the suffix .dpkg-dist (for example, /etc/frr/daemons.dpkg-dist). When the upgrade completes and before you reboot, merge your changes with the changes from the newly installed file.
    • To install the new version, type I. Your currently installed version has the suffix .dpkg-old.
    • Cumulus Linux includes /etc/apt/sources.list in the cumulus-archive-keyring package. During upgrade, you must select if you want the new version from the package or the existing file.

    When the upgrade is complete, you can search for the files with the sudo find / -mount -type f -name '*.dpkg-*' command.

    If you see errors for expired GPG keys that prevent you from upgrading packages, follow the steps in Upgrading Expired GPG Keys.

  5. Reboot the switch if the upgrade messages indicate that you need to perform a system restart.

```
cumulus@switch:~$ sudo -E apt-get upgrade
... upgrade messages here ...

*** Caution: Service restart prior to reboot could cause unpredictable behavior
*** System reboot required ***
cumulus@switch:~$ sudo reboot
```
  1. Verify correct operation with the old configurations on the new version.

Upgrade Switches in an MLAG Pair

If you are using MLAG to dual connect two switches in your environment, follow the steps below to upgrade the switches.

You must upgrade both switches in the MLAG pair to the same release of Cumulus Linux.

Only during the upgrade process does Cumulus Linux supports different software versions between MLAG peer switches. After you upgrade the first MLAG switch in the pair, run the clagctl showtimers command to monitor the init-delay timer. When the timer expires, make the upgraded MLAG switch the primary, then upgrade the peer to the same version of Cumulus Linux.

NVIDIA has not tested running different versions of Cumulus Linux on MLAG peer switches outside of the upgrade time period; you might see unexpected results.

  1. Verify the switch is in the secondary role:

    cumulus@switch:~$ nv show mlag

  2. Shut down the core uplink layer 3 interfaces. The following example shuts down swp1:

    cumulus@switch:~$ nv set interface swp1 link state down
cumulus@switch:~$ nv config apply

  3. Shut down the peer link:

    cumulus@switch:~$ nv set interface peerlink link state down
cumulus@switch:~$ nv config apply

  4. Upgrade the switch:

  5. If you installed a new image on the switch, restore the configuration files to the new release. If you performed an upgrade with apt, bring the uplink and peer link interfaces you shut down in steps 2-3 up:

    cumulus@switch:~$ nv set interface swp1 link state up
cumulus@switch:~$ nv set interface peerlink link state up
cumulus@switch:~$ nv config apply
cumulus@switch:~$ nv config save

  6. Verify STP convergence across both switches with the Linux mstpctl showall command. NVUE does not provide an equivalent command.

    cumulus@switch:~$ mstpctl showall

  7. Verify core uplinks and peer links are UP:

    cumulus@switch:~$ nv show interface

  8. Verify MLAG convergence:

    cumulus@switch:~$ nv show mlag

  9. Make this secondary switch the primary:

    cumulus@switch:~$ nv set mlag priority 2084

  10. Verify the other switch is now in the secondary role.

  11. Repeat steps 2-8 on the new secondary switch.

  12. Remove the priority 2048 and restore the priority back to 32768 on the current primary switch:

    cumulus@switch:~$ nv set mlag priority 32768

  1. Verify the switch is in the secondary role:

    cumulus@switch:~$ clagctl status

  2. Shut down the core uplink layer 3 interfaces:

    cumulus@switch:~$ sudo ip link set <switch-port> down

  3. Shut down the peer link:

    cumulus@switch:~$ sudo ip link set peerlink down

  4. Upgrade the switch:

  5. Reboot the switch:

    cumulus@switch:~$ sudo reboot

  6. If you installed a new image on the switch, restore the configuration files to the new release.

  7. Verify STP convergence across both switches:

    cumulus@switch:~$ mstpctl showall

  8. Verify that core uplinks and peer links are UP:

    cumulus@switch:~$ ip addr show

  9. Verify MLAG convergence:

    cumulus@switch:~$ clagctl status

  10. Make this secondary switch the primary:

    cumulus@switch:~$ clagctl priority 2048

  11. Verify the other switch is now in the secondary role.

  12. Repeat steps 2-9 on the new secondary switch.

  13. Remove the priority 2048 and restore the priority back to 32768 on the current primary switch:

    cumulus@switch:~$ clagctl priority 32768

Downgrade a Secure Boot Switch

The SN3700C-S, SN5400, and SN5600 secure boot switch running Cumulus Linux 5.11.0 boots with shim 15.8 that adds entries to the SBAT revocations to prevent the switch from booting shim 15.7 or earlier, which has security vulnerabilities.

After downgrading the switch from Cumulus Linux 5.11.0, follow the steps below to disable, then enable secure boot before the switch boots.

You can also follow the steps below to recover a downgraded secure boot switch that does not boot and that shows the following error:

Verifiying shim SBAT data failed: Security Policy Violation
Something has gone seriously wrong: SBAT self-check failed: Security Policy Violation

  1. On the switch, disable SecureBoot in BIOS:

    a. Press Ctrl B through the serial console during system boot while the BIOS version prints.

    b. When prompted, provide the BIOS password. The default password is admin.

    c. To disable secure boot, navigate to Security, and change Secure Boot to Disabled.

    d. Select Save & Exit.

  2. Boot into Cumulus Linux.

  3. Run the mokutil --set-sbat-policy delete command.

  4. Reboot the switch.

  5. Follow steps a through d above to enable secure boot in BIOS. In step c, change Secure Boot to Enabled.

Third Party Packages

If you install any third party applications on a Cumulus Linux switch, configuration data is typically installed in the /etc directory, but it is not guaranteed. It is your responsibility to understand the behavior and configuration file information of any third party packages installed on the switch.

After you upgrade using a full Cumulus Linux image install, you need to reinstall any third party packages or any Cumulus Linux add-on packages.

Considerations

  • The /etc/os-release and /etc/lsb-release files update to the currently installed Cumulus Linux release when you upgrade the switch using either package upgrade or Cumulus Linux image install. For example, if you perform a package upgrade and the latest Cumulus Linux release on the repository is 5.11, these two files display the release as 5.11 after the upgrade.
  • The /etc/image-release file updates only when you run a Cumulus Linux image install. Therefore, if you run a Cumulus Linux image install of Cumulus Linux 5.9.2, followed by a package upgrade to 5.11, the /etc/image-release file continues to display Cumulus Linux 5.9.2, which is the originally installed base image.

    Copyright © 2025  NVIDIA Corporation