Reserved VLAN Range and Limitations

Cumulus reserves a single contiguous set of VLAN IDs (the reserved VLAN range) for internal data plane operations. If you configure VLAN IDs that are within the reserved VLAN range, a scenario called reserved VLAN overlap occurs, which can result in unexpected traffic patterns.

To prevent a reserved VLAN overlap, you might need to change the reserved VLAN range.

Reserved VLAN Count

The minimum number of VLAN IDs required in the reserved VLAN range depends on the Cumulus Linux release and the feature you configure. You might need to change the reserved VLAN range to increase the number of VLANs reserved for certain features.

Default Reserved VLAN Ranges

Determine the Reserved VLAN Range

• NVIDIA Spectrum switches running Cumulus Linux 5.0 and later require a reserved VLAN for every bridge and QinQ bridge, plus two.
• NVIDIA Spectrum switches running Cumulus Linux 4.4 and earlier require a reserved VLAN for every bridge, physical interface, layer 3 subinterface, and QinQ bridge, plus one.
• Broadcom switches running Cumulus Linux 4.3 and earlier require a reserved VLAN for every bridge, physical interface, layer 3 subinterface, and QinQ bridge, plus one.

The example below provides Linux-type shell commands to help you determine the in-use and configured VLAN counts and values. These commands are only a guide. Follow the guidelines below to determine how to best calculate the values.

1. If your switch is running Cumulus Linux 5.0 or later, go to step 2. If your switch is running Cumulus Linux 4.4 or earlier, determine the total number of physical interfaces that your platform supports. For example, the NVIDIA SN2700 switch has a total of 32 QSFP ports that you can break out into 2x or 4x.

   At the maximum breakout (4x), there are 128 physical ports. Cumulus Linux always uses this number of reserved VLANs regardless of configuration:

   • 32 x 1 = 32 Reserved VLAN IDs on the switch with no breakout
   • 32 x 2 = 64 Reserved VLAN IDs on the switch with all ports broken out in 2x mode
   • 16 x 4 = 64 Reserved VLAN IDs on the switch with all ports broken out in 4x mode

2. Determine the total number of bridges in the configuration. The example below uses the bridge-ports statement as a counting key.

   cumulus@switch:~$ sudo cat /etc/network/interfaces | grep "bridge-ports" | wc -l
   cumulus@switch:~$ sudo cat /etc/network/interfaces.d/*.intf | grep "bridge-ports" | wc -l
   cumulus@switch:~$ sudo cat /etc/network/interfaces | grep "bridge-ports" | wc -l
   cumulus@switch:~$ sudo cat /etc/network/interfaces.d/*.intf | grep "bridge-ports" | wc -l
   

3. If you use double-tagged Q-in-Q, determine the total number of implicit bridges not defined in the configuration. The example below uses a unique interfaces+inner tag (swpX.Y) as a counting key.

   cumulus@switch:~$ sudo grep -o -h "iface swp.*\.[[:digit:]]*\." /etc/network/interfaces | sort -u |    wc -l
   cumulus@switch:~$ sudo grep -o -h "iface swp.*\.[[:digit:]]*\." /etc/network/interfaces.d/*.intf |    sort -u | wc -l
   

4. If your switch is running Cumulus Linux 5.0 or later, go to step 5. If your switch is running Cumulus Linux 4.4 or earlier, determine the total number of layer 3 subinterfaces in the configuration. The example below uses the vlan-raw-device statement as a counting key.

   cumulus@switch:~$ sudo cat /etc/network/interfaces | grep "vlan-raw-device" | wc -l
   cumulus@switch:~$ sudo cat /etc/network/interfaces.d/*.intf | grep "vlan-raw-device" | wc -l
   

5. Add the totals from step 1 through step 4.

   • For NVIDIA Spectrum switches running 5.0 and later: Bridges + (implied QinQ bridges) + 2 = MINIMUM for configuration.
   • For NVIDIA Spectrum switches running 4.4 and earlier: Interfaces + bridges + (implied QinQ bridges) + layer 3 sub interfaces + 1 = MINIMUM for configuration.
   • For Broadcom switches running 4.3 and earlier: Interfaces + bridges + (implied QinQ bridges) + layer 3 sub interfaces + 1 = MINIMUM for configuration.

Never exceed this count, even temporarily, so that you have room for future expansion, in-process operations, and automation. For example, an automation system can modify a bridge by adding, then removing the old bridge. Cumulus does not guarantee minimal usage of reserved VLANs when doing multi-step operations.

6. Select a contiguous range of VLANs that covers the desired count from step 5 and does not overlap with VLAN IDs in the configuration. The example below determines which VLAN IDs are in the configuration:

   cumulus@switch:~$ grep -h "iface swp" /etc/network/interfaces | grep -iv "\#" | cut -s -d '.' -f 2- |    tr '.' '\n' | sort -u | sort -n
   cumulus@switch:~$ grep -h "iface swp" /etc/network/interfaces.d/*.intf | grep -iv "\#" | cut -s -d '.   ' -f 2- | tr '.' '\n' | sort -u | sort -n
   

Change the Reserved VLAN Range

To change the reserved VLAN range, run the commands below.

• For Cumulus Linux 5.3 and later, you can run NVUE or Linux commands.
• For Cumulus Linux 5.2 and earlier, you must run Linux commands.

The following example changes the reserved VLAN range to be between 4064 and 4094.

cumulus@switch:~$ nv set system global reserved vlan internal range 4064-4094
cumulus@switch:~$ nv config apply

Reserved Layer 3 VNI VLANs

In addition to the internal reserved VLAN range, Cumulus Linux allocates a reserved VLAN range for layer 3 VNIs in EVPN symmetric routing deployments. Use this reserved VLAN range when you configure layer 3 VNIs in MLAG environments with NVUE commands. You can display the range with the nv show system global reserved vlan l3-vni-vlan command:

cumulus@switch:~$ nv show system global reserved vlan l3-vni-vlan
operational  applied
-----  -----------  -------
begin  4000         4000
end    4064         4064

Do not use this range of VLANs in the same bridge as your MLAG interfaces and layer 3 VNIs. You can configure the range with the nv set system global reserved vlan l3-vni-vlan begin <vlan> and nv set system global reserved vlan l3-vni-vlan end <vlan> commands. For more information, see symmetric routing.