layout-3 layout-6 layout-1 layout-top
NVIDIA Docs Hubchevron_right NVIDIA Networkingchevron_right Networking Softwarechevron_right Switch Softwarechevron_right

Cumulus Linux

layout-top layout-6 layout-1

Docker with Cumulus Linux

You can use Cumulus Linux to run the Docker container platform.

The Docker package installs as part of the Cumulus Linux installation or ONIE upgrade process and the service is running by default. The Docker package includes Docker Engine, and dependencies and configuration files required to run the Docker service.

Change the Docker Service VRF

By default, the Docker service runs in the management VRF. To run Docker in a different VRF, run the following commands.

Changing the Docker VRF restarts the Docker service, which disrupts all running containers.

Run the nv set system docker vrf <vrf-id> command:

cumulus@switch:~$ nv set system docker vrf RED
cumulus@switch:~$ nv config apply

This example assumes that the RED VRF is already configured with NVUE. To learn more about configuring VRFs, reference Virtual Routing and Forwarding.

To reset the Docker container to run in the management VRF (the default setting), run the nv unset system docker vrf command.

Disable the service in the mgmt VRF and run the systemctl start docker@<vrf-id>.service command. For example:

cumulus@switch:~$ sudo systemctl disable docker@mgmt.service
cumulus@switch:~$ sudo systemctl stop docker@mgmt.service
cumulus@switch:~$ systemctl enable docker@RED.service
cumulus@switch:~$ systemctl stop docker@RED.service

Container Management

NVUE provides commands to:

  • Download a Docker image from a registry.
  • Delete a Docker image from the switch.
  • Create and run a new container from an image.
  • Stop a container.
  • Delete a Docker container.

Before you download any container image onto the switch, check the available disk space. You must have enough disk space to account for the size of the container image plus 25 percent more space just to download the image. In addition, consider how much extra space you need to run the container.

Docker Images

To download a Docker image from a registry, import an image from an archive, or remove a Docker image from the switch, run the following commands.

To download a Docker image from a registry, run the nv action pull system docker image <image-id> tag <tag-name> command. If you do not specify a tag name, the name defaults to latest.

cumulus@switch:~$ nv action pull system docker image nginx tag latest

To import a Docker image from an archive, run the nv action import system docker image <image-url> repository <repository-name> [tag <tag-name>] command:

cumulus@switch:~$ nv action import system docker image /path/to/exampleimage.tgz repository xyz tag imported

Supported archive formats for nv action import system docker include .tar, .tar.gz, .tgz, .bzip, .tar.xz, and .txz.

To delete a Docker image from the switch, run the nv action remove system docker image <image-id> command:

cumulus@switch:~$ nv action remove system docker image nginx tag latest

To download a Docker image from a registry, run the docker pull <image-id> command:

cumulus@switch:~$ docker pull nginx

To import a Docker image from an archive run the docker load <image-path> command:

cumulus@switch:~$ docker load -i /path/to/tarball/filename.tgz

To delete a Docker image from the switch, run the docker rmi <image-id> command:

cumulus@switch:~$ docker rmi nginx

Docker Containers

To create and run a new container from an image, stop a container or delete a Docker container, run the following commands.

To create and run a new container from an image, run the nv action run system docker container <container-name> image <image-id> command. You can use Docker run options such as --pid, --network, and --storage-opt size. To define arguments for the container application, specify args.

You must escape special characters used in any Docker options and args specified in NVUE commands.

cumulus@switch:~$ nv action run system docker container nginx-demo image nginx:alpine option '\-\-hostname nginx-demo \-p 8080:80 \-\-restart unless-stopped \-e NGINX_ENTRYPOINT_QUIET_LOGS=1 \-v site:/usr/share/nginx/html:ro \-\-log-opt max-size=10m \-\-log-opt max-file=3' args "nginx -g 'daemon off; worker_processes auto; error_log /var/log/nginx/error.log warn;'"

To stop a container, run the nv action stop system docker container <container-name> command:

cumulus@switch:~$ nv action stop system docker container nginx-demo

To delete a Docker container from the switch, run the nv action remove system docker container <container-id> command:

cumulus@switch:~$ nv action remove system docker container nginx-demo

To create and run a new container from an image, run the sudo docker run -d <image-id> --name <container-name> command.

cumulus@switch:~$ sudo docker run -d nginx --name nginx-demo

To stop a container, run the sudo docker stop <container-name> command:

cumulus@switch:~$ sudo docker stop nginx-demo

To delete a Docker container from the switch, run the sudo docker rm <container-name> command:

cumulus@switch:~$ sudo docker rm nginx-demo

Show Docker Information

To show Docker information on the switch, run the nv show system docker command:

cumulus@switch:~$ nv show system docker
       operational  applied
-----  -----------  -------
vrf    mgmt         mgmt   
state  enabled      enabled

Docker Containers
====================
    Container Name      Image                            Container ID  Status               Ports  Summary
    ------------------  -------------------------------  ------------  -------------------  -----  -------
    repo                cumulus-linux-apt-mirror:5.17.0  a941e1e51c3e  Up 6 days (healthy)                
    what-just-happened  docker-wjh:latest                f834edf7fd3c  Up 7 days

To show Docker images present on the switch, run the nv show system docker image command. Add the -o native option to display additional data from Docker inspect.

cumulus@switch:~$ nv show system docker image
Image Id      Image Name                Tag     Size    Date                           Summary
------------  ------------------------  ------  ------  -----------------------------  -------
283e2bf92e80  docker-wjh                latest  716MB   2025-10-29 21:47:09 -0400 EDT         
d839322a5483  cumulus-linux-apt-mirror  5.17.0  3.47GB  2025-10-30 02:35:30 -0400 EDT

To list all containers and their status, including stopped containers, run the nv show system docker container command. Add the -o native option to display additional data from Docker inspect.

cumulus@switch:~$ nv show system docker container
Container Name      Image                            Container ID  Status               Ports  Summary
------------------  -------------------------------  ------------  -------------------  -----  -------
repo                cumulus-linux-apt-mirror:5.17.0  a941e1e51c3e  Up 6 days (healthy)                
what-just-happened  docker-wjh:latest                f834edf7fd3c  Up 7 days

To show details of container, run the nv show system docker container <container-id> command:

cumulus@switch:~$ nv show system docker container repo
               operational                    
-------------  -------------------------------
id             a941e1e51c3e                   
status         Up 6 days (healthy)            
image-name     cumulus-linux-apt-mirror:5.17.0
port                                          
stats                                         
  cpu          0.00%                          
  mem-usage    8.105MiB                       
  mem-limit    15.02GiB                       
  mem-percent  0.05%                          
  net-io       0B / 0B                        
  block-io     160kB / 41kB                   
  pids         9

To show all container statistics, run the nv show system docker container stats command:

cumulus@switch:~$ $ nv show system docker container stats
Container Name      CPU%   MEM USAGE  MEM LIMIT  MEM%   NET I/O  BLOCK I/O       PIDS
------------------  -----  ---------  ---------  -----  -------  --------------  ----
repo                0.00%  8.102MiB   15.02GiB   0.05%  0B / 0B  160kB / 41kB    9   
what-just-happened  0.05%  81.96MiB   15.02GiB   0.53%  0B / 0B  496kB / 16.4kB  9

To show statistics for a specific container, run the nv show system docker container <container-id-name> stats command:

cumulus@switch:~$ nv show system docker container repo stats 
             operational 
-----------  ------------
cpu          0.00%       
mem-usage    8.102MiB    
mem-limit    15.02GiB    
mem-percent  0.05%       
net-io       0B / 0B     
block-io     160kB / 41kB
pids         9

To show Docker engine configuration, run the nv show system docker engine command. Add the -o native option to display additional data from Docker inspect.

cumulus@switch:~$ nv show system docker engine
                operational                                                                                    
--------------  -----------------------------------------------------------------------------------------------
client                                                                                                         
  name          Docker Engine - Community                                                                      
  version       28.5.1                                                                                         
  context       default                                                                                        
server                                                                                                         
  containers    2                                                                                              
  running       2                                                                                              
  paused        0                                                                                              
  stopped       0                                                                                              
plugins                                                                                                        
  volume        ['local']                                                                                      
  network       ['bridge', 'host', 'ipvlan', 'macvlan', 'null', 'overlay']                                     
  log           ['awslogs', 'fluentd', 'gcplogs', 'gelf', 'journald', 'json-file', 'local', 'splunk', 'syslog']
images          2                                                                                              
server-version  28.5.1                                                                                         
id              ae4be5b9-6806-435d-80cb-5e4548a9c11a                                                           
init-binary     docker-init                                                                                    
data-root       /docker                                                                                        
debug-mode      False                                                                                          
log-level       json-file

Considerations

  • Be mindful of the types of applications you want to run in containers on a Cumulus Linux switch. Depending on the configuration of the container, DHCP servers, custom scripts, and other lightweight services run well. However, VPN, NAT and encryption-type services are CPU-intensive and lead to undesirable effects on critical applications.

  • NVUE manages the /etc/docker/daemon.json file and overwrites the file on every configuration. If you want to update this file, make sure to use a snippet. The following example shows a snippet that updates the /etc/docker/daemon.json file:

    ```
cumulus@switch:/etc/systemd/system$ nv config patch text.conf
created [rev_id: 35]
cumulus@switch:/etc/systemd/system$ nv config diff
- set:
   system:
      config:
      snippet:
         docker-daemon: |
            {
               "iptables": false,
               "ip6tables": false,
               "ip-forward": false,
               "ip-masq": false,
               "bridge": "none",
               "data-root": "/docker"
            }
```

Docker Resource Tiering System

Cumulus Linux includes a Docker Resource Tiering Governance system that protects the switch from noisy-neighbor CPU and memory exhaustion by classifying container workloads into governance tiers and placing them into the correct systemd slice at container creation time.

The Governance system is not a long-running background agent; the packaged docker wrapper applies governance at container birth. The cumulus-docker-resource-tier-governance.service systemd service programs the tier slice limits and reconciles existing governance-managed containers when you start, reload, restart, or stop the service.

When governance is enabled, containers use one of these tiers:

  • docker.slice is the Trusted tier. Containers in this tier are not subject to CPU or memory caps.
  • docker-limited.slice is the Limited tier. This tier is for workloads that need more headroom than the default tier but must still remain bounded.
  • docker-restricted.slice is the Restricted tier. This is the default tier for images that are not whitelisted.

By default, the Restricted and Limited tiers use aggregate limits of 10% and 50% of total host CPU and total host memory. These are shared per-tier limits, not per-container reservations. If multiple containers run in the same tier, they share the tier budget.

The packaged docker wrapper classifies supported docker container creation commands and places new containers directly into the correct parent slice with the appropriate cgroup-parent.

The cumulus-docker-resource-tier-governance.service service reads the current policy, programs the Restricted and Limited slice limits, and reconciles existing governance-managed containers when you apply policy. If you install the Docker Compose plugin, Docker compose workloads are also governed at creation time.

Governance does not move running container processes between control groups. If an existing governance-managed container needs to change tiers, the service recreates it in the correct slice. If the container is supervised by systemd, the service restarts the owning unit so the container is recreated through its normal service lifecycle.

Governance Configuration

Governance uses these configuration files:

  • /etc/cumulus/docker/resource-tier-governance/whitelist.json controls which images are classified as Trusted or Limited.
  • /etc/cumulus/docker/resource-tier-governance/quotas.conf controls the aggregate CPU and memory percentages for the Restricted and Limited tiers.

Images that are not listed in either the Trusted or Limited tier default to the Restricted tier.

The following example shows an /etc/cumulus/docker/resource-tier-governance/whitelist.json file:

{
  "trusted_images": [],
  "limited_images": [
    "nv-gnmi",
    "nv-umf",
    "nv-grpctunnel",
    "nv-otel-collector",
    "docker-wjh"
  ]
}

  • Whitelist entries can match repository names, tags, or digests as seen by Docker on the local system. Matching is case-insensitive.
  • Do not list the same image in both the Trusted and Limited tiers. If an image appears in both places, the Trusted tier takes precedence.

The following example shows an /etc/cumulus/docker/resource-tier-governance/quotas.conf file:

RESTRICTED_DOCKER_RESOURCE_QUOTA=10
LIMITED_DOCKER_RESOURCE_QUOTA=50

Enable and Disable Governance

The Docker Resource Tiering Governance system is installed and enabled by default. To disable governance immediately and prevent it from being enabled again at boot, stop and disable the service:

cumulus@switch:~$ sudo systemctl stop cumulus-docker-resource-tier-governance.service
cumulus@switch:~$ sudo systemctl disable cumulus-docker-resource-tier-governance.service

When you stop the service, Cumulus Linux:

  • Disables governance for future container births.
  • Promotes existing governance-managed containers to docker.slice.
  • Clears the Restricted and Limited slice caps.
  • Leaves containers with a custom parent control group (cgroup) unchanged.
  • Leaves unmanaged containers that do not carry governance metadata unchanged.

When you disable governance, new containers default to the Trusted tier unless you select another parent slice.

To re-enable governance:

cumulus@switch:~$ sudo systemctl enable cumulus-docker-resource-tier-governance.service
cumulus@switch:~$ sudo systemctl start cumulus-docker-resource-tier-governance.service

  • Enable and disable control boot-time activation only. Runtime policy changes occur when you start, reload, restart, or stop the service.
  • If the service is already active, systemctl start does not reapply policy. Use systemctl reload to apply updated governance policy to a running system.

Promote, Demote, and Whitelist Images

To move an image between tiers, edit the /etc/cumulus/docker/resource-tier-governance/whitelist.json file:

  • To promote an image to Trusted, add it to trusted_images and remove it from limited_images.
  • To move an image to Limited, add it to limited_images and remove it from trusted_images.
  • To demote an image back to Restricted, remove it from both limited_images and trusted_images.

After updating the whitelist, apply the change with the sudo systemctl reload cumulus-docker-resource-tier-governance.service command.

Use systemctl reload instead of systemctl restart for normal policy updates.

  • systemctl reload re-reads the whitelist and quota files, reprograms the Restricted and Limited slice limits, and reconciles existing governance-managed containers against the updated policy. This avoids the extra churn caused by a full stop-then-start cycle.
  • systemctl restart is more disruptive because it executes a full stop followed by start. The stop path disables governance and promotes governance-managed containers to docker.slice. The start path enables governance again and might recreate or restart the containers a second time to place them back into Limited or Restricted.

  • systemctl restart might cause unnecessary container churn during normal whitelist updates. Use systemctl reload when governance is already active. If the service is currently inactive, run the sudo systemctl start cumulus-docker-resource-tier-governance.service command.
  • If you update the /etc/cumulus/docker/resource-tier-governance/quotas.conf file, apply the change with the sudo systemctl reload cumulus-docker-resource-tier-governance.service command. For quota-only changes, reload updates the slice limits live and does not require recreating containers just to program the new slice cap values.

Considerations

  • Existing containers are reconsidered only when you apply governance with a service action such as start, reload, restart, or stop.
  • Editing the /etc/cumulus/docker/resource-tier-governance/whitelist.json file or the /etc/cumulus/docker/resource-tier-governance/quotas.conf file does not retier existing containers until you apply governance again.
  • Containers created before you install the governance wrapper, or other containers without governance metadata, are treated as unmanaged and are left untouched. To bring these containers under governance, stop and remove them, then recreate them.
  • Containers created with an explicit Docker --cgroup-parent or Compose cgroup_parent remain under user control and are not re-tiered by governance.
  • Retiering an existing container can require a stop, remove, and recreate operation or, for workloads that systemd supervises, a restart of the owning unit. Plan policy changes accordingly for stateful or production workloads.
  • Governance enforces aggregate ceilings per tier. It does not provide guaranteed minimum CPU or memory reservations for each container inside a tier.

Verify Container Placement and Tier Limits

To check the configuration of a container parent slice:

cumulus@switch:~$ docker inspect -f '{{.HostConfig.CgroupParent}}' <container_name>

To verify that the running process is actually under the expected slice:

cumulus@switch:~$ PID=$(docker inspect -f '{{.State.Pid}}' <container_name>)
awk -F: '$1=="0" {print $3}' /proc/$PID/cgroup

To inspect the slice limits programmed by governance:

cumulus@switch:~$ systemctl show docker-restricted.slice -p CPUQuotaPerSecUSec -p MemoryMax
cumulus@switch:~$ systemctl show docker-limited.slice -p CPUQuotaPerSecUSec -p MemoryMax

The authoritative CPU and memory limits are applied to the slice, not to the individual container scope. Verify both container placement and slice properties when checking governance behavior.

    Copyright © 2026  NVIDIA Corporation