Upgrading Cumulus Linux

This topic describes how to upgrade Cumulus Linux on your switch.

Consider deploying, provisioning, configuring, and upgrading switches using automation, even with small networks or test labs. During the upgrade process, you can upgrade dozens of devices in a repeatable manner. Using tools like Ansible, Chef, or Puppet for configuration management greatly increases the speed and accuracy of the next major upgrade; these tools also enable you to quickly swap failed switch hardware.

Before You Upgrade

Before upgrading Cumulus Linux, make sure to back up configuration files off the switch and create a cl-support file so that you can troubleshoot any issues that might occur during upgrade.

Back up Configuration Files

Understanding the location of configuration data is important for successful upgrades, migrations, and backup. As with other Linux distributions, the /etc directory is the primary location for all configuration data in Cumulus Linux. The following list contains the files you need to back up and migrate to a new release. Make sure you examine any changed files. Make the following files and directories part of a backup strategy.

The following commands verify which files have changed compared to the previous Cumulus Linux install. Be sure to back up any changed files.

• Run the sudo dpkg --verify command to show a list of changed files.
• Run the egrep -v '^$|^#|=""$' /etc/default/isc-dhcp-* command to see if any of the generated /etc/default/isc-* files have changed.

Back Up and Restore Configuration with NVUE

You can back up and restore the configuration file with NVUE only if you used NVUE commands to configure the switch you want to upgrade.

To back up and restore the configuration file:

1. Save the configuration to the /etc/nvue.d/startup.yaml file with the nv config save command:

   cumulus@switch:~$ nv config save
   saved
   

2. Copy the /etc/nvue.d/startup.yaml file off the switch to a different location.

3. After upgrade is complete, restore the configuration. Copy the /etc/nvue.d/startup.yaml file to the switch, run the nv config patch command, then run the nv config apply command. In the following example startup.yaml is in the /home/cumulus directory on the switch:

   cumulus@switch:~$ nv config patch /home/cumulus/startup.yaml
   cumulus@switch:~$ nv config apply
   

For information about the NVUE object model and commands, see NVIDIA User Experience - NVUE.

Create a cl-support File

Before and after you upgrade the switch, run the cl-support script to create a cl-support archive file. The file is a compressed archive of useful information for troubleshooting. If you experience any issues during upgrade, you can send this archive file to the Cumulus Linux support team to investigate.

1. Create the cl-support archive file with either the NVUE nv action generate system tech-support command or the Linux sudo cl-support command:

cumulus@switch:~$ nv action generate system tech-support

2. Copy the cl-support file off the switch to a different location.

3. After upgrade is complete, create a new archive file:

cumulus@switch:~$ nv action generate system tech-support

Upgrade Cumulus Linux

You can upgrade Cumulus Linux in one of two ways:

• Install a Cumulus Linux image of the new release, using ONIE.
• Upgrade only the changed packages using package upgrade.

Cumulus Linux also provides ISSU to upgrade an active switch with minimal disruption to the network. See In-Service-System-Upgrade-ISSU.

Install a Cumulus Linux Image or Upgrade Packages?

The decision to upgrade Cumulus Linux by either installing a Cumulus Linux image or upgrading packages depends on your environment and your preferences. The following section provides recommendations for each upgrade method.

Install a Cumulus Linux image if you are performing a rolling upgrade in a production environment and if you are using up-to-date and comprehensive automation scripts. This upgrade method enables you to choose the exact release to which you want to upgrade and is the only method available to upgrade your switch to a new release train (for example, from 4.4.3 to 5.11).

Be aware of the following when installing the Cumulus Linux image:

• Installing a Cumulus Linux image is destructive; any configuration files on the switch are not saved; copy them to a different server before you start the Cumulus Linux image install.
• You must move configuration data to the new network operating system using ZTP or automation while the operating system is first booted, or soon afterwards using out-of-band management.
• Moving a configuration file can cause issues.
• Identifying all the locations that include configuration data is not always an easy task. See Before You Upgrade Cumulus Linux above.
• Merge conflicts with configuration file changes in the new release sometimes go undetected.
• If configuration files do not restore correctly, you cannot ssh to the switch from in-band management. Use out-of-band connectivity (eth0 or the console).
• You must reinstall and reconfigure third-party applications after upgrade.

Run package upgrade if you are upgrading from one Cumulus Linux 5.11 release to a later 5.11 release, and if you use third-party applications (package upgrade does not replace or remove third-party applications, unlike the Cumulus Linux image install).

Be aware of the following when upgrading packages:

• You cannot upgrade the switch to a new release train. For example, you cannot upgrade the switch from 4.x to 5.x.
• Package upgrade only supports the current version plus two. For example, you can upgrade from Cumulus Linux 5.6 to 5.8 with package upgrade only if you installed the 5.6 binary image.
• The package upgrade command might restart or stop services as part of the upgrade process.
• The package upgrade command might disrupt core services by changing core service dependency packages.
• After you upgrade, account UIDs and GIDs created by packages might be different on different switches, depending on the configuration and package installation history.
• Cumulus Linux does not support the sudo -E apt-get dist-upgrade command. Be sure to use sudo -E apt-get upgrade when upgrading packages.

Cumulus Linux Image Install (ONIE)

ONIE is an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on a bare metal switch.

To upgrade the switch:

1. Back up the configurations off the switch.

2. Download the Cumulus Linux image.

3. Install the Cumulus Linux image with the onie-install -a -i <image-location> command, which boots the switch into ONIE. The following example command installs the image from a web server, then reboots the switch. There are additional ways to install the Cumulus Linux image, such as using FTP, a local file, or a USB drive. For more information, see Installing a New Cumulus Linux Image.

   cumulus@switch:~$ sudo onie-install -a -i http://10.0.1.251/cumulus-linux-5.11.0-mlx-amd64.bin && sudo reboot
   

4. Restore the configuration files to the new release (NVIDIA does not recommend restoring files with automation).

5. Verify correct operation with the old configurations on the new release.

6. Reinstall third party applications and associated configurations.

Package Upgrade

Cumulus Linux completely embraces the Linux and Debian upgrade workflow, where you use an installer to install a base image, then perform any package upgrades within that release train. Any packages that have changed after the base install get upgraded in place from the repository. All switch configuration files remain untouched, or in rare cases merged during the package upgrade.

When you use package upgrade to upgrade your switch, configuration data stays in place during the upgrade. If the new release updates a previously changed configuration file, the upgrade process prompts you to either specify the version you want to use or evaluate the differences.

Disk Space Requirements

Make sure you have enough disk space to perform a package upgrade. To upgrade from Cumulus Linux 5.10 to Cumulus Linux 5.11, you need 0.8GB of free disk space.

Before you upgrade, run the sudo df -h command to show how much disk space you are currently using on the switch.

cumulus@switch:~$ sudo df -h
Filesystem      Size   Used   Avail   Use%    Mounted on
udev            7.7G      0    7.7G     0%    /dev
tmpfs           1.6G    18M    1.6G     2%    /run
/dev/sda4        28G   7.9G     18G    31%    /
tmpfs           7.7G   277M    7.4G     4%    /dev/shm
tmpfs           5.0M      0    5.0M     0%    /run/lock
tmpfs           7.7G      0    7.7G     0%    /sys/fs/cgroup
tmpfs           7.7G    16K    7.7G     1%    /tmp
overlay          28G   7.9G     18G    31%   

Upgrade the Switch

To upgrade the switch using package upgrade:

```
cumulus@switch:~$ sudo -E apt-get upgrade
... upgrade messages here ...

*** Caution: Service restart prior to reboot could cause unpredictable behavior
*** System reboot required ***
cumulus@switch:~$ sudo reboot
```

Upgrade Notes

Package upgrade always updates to the latest available release in the Cumulus Linux repository. For example, if you are currently running Cumulus Linux 5.0.0 and perform a package upgrade, the packages upgrade to the latest releases in the latest 5.x release.

Cumulus Linux is a collection of different Debian Linux packages; be aware of the following:

• The /etc/os-release and /etc/lsb-release files update to the currently installed Cumulus Linux release when you upgrade the switch using either package upgrade or Cumulus Linux image install. For example, if you perform a package upgrade and the latest Cumulus Linux release on the repository is 5.9.1, these two files display the release as 5.9.1 after the upgrade.
• The /etc/image-release file updates only when you run a Cumulus Linux image install. Therefore, if you run a Cumulus Linux image install of Cumulus Linux 5.9.0, followed by a package upgrade to 5.9.1, the /etc/image-release file continues to display Cumulus Linux 5.9.0, which is the originally installed base image.

Upgrade Switches in an MLAG Pair

If you are using MLAG to dual connect two switches in your environment, follow the steps below to upgrade the switches.

Roll Back a Cumulus Linux Installation

Even the most well planned and tested upgrades can result in unforeseen problems and sometimes the best solution is to roll back to the previous state. These main strategies require detailed planning and execution:

• Flatten and rebuild. If the OS becomes unusable, you can use orchestration tools to reinstall the previous OS release from scratch and then rebuild the configuration automatically.
• Restore to a previous state using a backup configuration captured before the upgrade.

The method you employ is specific to your deployment strategy. Providing detailed steps for each scenario is outside the scope of this document.

Downgrade a Secure Boot Switch from Cumulus Linux 5.11.0

The SN3700C-S, SN5400, and SN5600 secure boot switch running Cumulus Linux 5.11.0 boots with shim 15.8 that adds entries to the SBAT revocations to prevent the switch from booting shim 15.7 or earlier, which has security vulnerabilities.

After downgrading the switch from Cumulus Linux 5.11.0, follow the steps below to disable, then enable secure boot before the switch boots.

You can also follow the steps below to recover a downgraded secure boot switch that does not boot and that shows the following error:

Verifiying shim SBAT data failed: Security Policy Violation
Something has gone seriously wrong: SBAT self-check failed: Security Policy Violation

1. On the switch, disable SecureBoot in BIOS:

   a. Press Ctrl B through the serial console during system boot while the BIOS version prints.

   b. When prompted, provide the BIOS password. The default password is admin.

   c. To disable secure boot, navigate to Security, and change Secure Boot to Disabled.

   d. Select Save & Exit.

2. Boot into Cumulus Linux.

3. Run the mokutil --set-sbat-policy delete command.

4. Reboot the switch.

5. Follow steps a through d above to enable secure boot in BIOS. In step c, change Secure Boot to Enabled.

Third Party Packages

If you install any third party applications on a Cumulus Linux switch, configuration data is typically installed in the /etc directory, but it is not guaranteed. It is your responsibility to understand the behavior and configuration file information of any third party packages installed on the switch.

After you upgrade using a full Cumulus Linux image install, you need to reinstall any third party packages or any Cumulus Linux add-on packages.

Related Information

• Upgrades: Network Device Worldview and Linux Host Worldview Comparison
• Automation Solutions
• ONIE Design Specification
• Multi-Chassis Link Aggregation - MLAG
• Zero Touch Provisioning - ZTP