If you are using the current version of Cumulus Linux, the content on this page may not be up to date. The current version of the documentation is available here. If you are redirected to the main page of the user guide, then this page may have been renamed; please search for it there.

Supported Route Table Entries

Cumulus Linux advertises the maximum number of route table entries supported on a given platform, including:

  • Layer 3 IPv4 LPM (longest prefix match) entries that have a mask less than /32
  • Layer 3 IPv6 LPM entries that have a mask of /64 or less
  • Layer 3 IPv6 LPM entries that have a mask greater than /64
  • Layer 3 IPv4 neighbor (or host) entries that are the next hops seen in ip neighbor
  • Layer 3 IPv6 neighbor entries that are the next hops seen in ip -6 neighbor
  • ECMP next hops, which are IP address entries in a router’s routing table that specify the next closest or most optimal router in its routing path
  • MAC addresses

In addition, Tomahawk, Trident II, Trident II+, and Trident3 switches are configured to manage route table entries using Algorithm Longest Prefix Match (ALPM). In ALPM mode, the hardware can store significantly more route entries.

To determine the current table sizes on a switch, use either the NCLU net show system asic command or cl-resource-query.

Forwarding Table Profiles

On Mellanox Spectrum and some Broadcom ASICs, you can configure the allocation of forwarding table resources and mechanisms. Cumulus Linux provides a number of generalized profiles for the platforms described below. These profiles work only with layer 2 and layer 3 unicast forwarding.

Choose the profile that best suits your network architecture and specify the profile name for the forwarding_table.profile variable in the /etc/cumulus/datapath/traffic.conf file; for example:

cumulus@switch:~$ cat /etc/cumulus/datapath/traffic.conf | grep forwarding_table -B 4
# Manage shared forwarding table allocations
# Valid profiles -
# default, l2-heavy, v4-lpm-heavy, v6-lpm-heavy
#
forwarding_table.profile = default

After you specify a different profile, restart switchd for the change to take effect. You can see the forwarding table profile when you run cl-resource-query.

  • Broadcom ASICs other than Maverick, Tomahawk/Tomahawk+, Trident II, Trident II+, and Trident3 support only the default profile.
  • For Broadcom ASICs, the maximum number of IP multicast entries is 8k.

Supported Route Entries

The following tables list the number of MAC addresses, layer 3 neighbors, and LPM routes validated for each forwarding table profile for supported platforms. If you do not specify any profiles as described above, the switch uses the default values.

The values provided in the profiles below are the maximum values that Cumulus Linux software allocates; the theoretical hardware limits might be higher. These limits refer to values that have been validated as part of the unidimensional scale validation. If you try to achieve maximum scalability with multiple features enabled, results might differ from the values listed in this guide.

Mellanox Spectrum Switches

Profile
MAC Addresses
L3 Neighbors
Longest Prefix Match (LPM)
default 40k 32k (IPv4) and 16k (IPv6) 64k (IPv4) and 28k (IPv6-long)
l2-heavy 88k 48k (IPv4) and 40k (IPv6) 8k (IPv4) and 8k (IPv6-long)
l2-heavy-1 180K 8k (IPv4) and 8k (IPv6) 8k (IPv4) and 8k (IPv6-long)
v4-lpm-heavy 8k 8k (IPv4) and 16k (IPv6) 80k (IPv4) and 16k (IPv6-long)
v4-lpm-heavy-1 8k 8k (IPv4) and 2k (IPv6) 176k (IPv4) and 2k (IPv6-long)
v6-lpm-heavy 40k 8k (IPv4) and 40k (IPv6) 8k (IPv4) and 32k (IPv6-long) and 32K (IPv6/64)
lpm-balanced 8k 8k (IPv4) and 8k (IPv6) 60k (IPv4) and 60k (IPv6-long)

Broadcom Tomahawk/Tomahawk+ Switches

Profile MAC Addresses L3 Neighbors Longest Prefix Match (LPM)
default 40k 40k 64k (IPv4) or 8k (IPv6-long)
l2-heavy 72k 72k 8k (IPv4) or 2k (IPv6-long)
v4-lpm-heavy
v6-lpm-heavy
8k 8k 128k (IPv4) or 20k (IPv6-long)

Broadcom Trident II/Trident II+/Trident3 Switches

Profile MAC Addresses L3 Neighbors Longest Prefix Match (LPM)
default 32k 16k 128k (IPv4) or 20k (IPv6-long)
l2-heavy 160k 96k 8k (IPv4) or 2k (IPv6-long)
v4-lpm-heavy
v6-lpm-heavy
32k 16k 128k (IPv4) or 20k (IPv6-long)

Broadcom Helix4 Switches

Helix4 switches do not have profiles.

MAC Addresses L3 Neighbors Longest Prefix Match (LPM)
24k 12k 7.8k (IPv4) or 2k (IPv6-long)

For Broadcom switches, IPv4 and IPv6 entries are not carved in separate spaces so it is not possible to define explicit numbers in the L3 Neighbors column of the tables above. An IPv6 entry takes up twice the space of an IPv4 entry.

TCAM Resource Profiles for Spectrum Switches

On the Mellanox Spectrum ASIC, you can configure TCAM resource allocation, which is shared between IP multicast forwarding entries and ACL tables. Cumulus Linux provides a number of general profiles for this platform. Choose the profile that best suits your network architecture and specify that profile name in the tcam_resource.profile variable in the /usr/lib/python2.7/dist-packages/cumulus/__chip_config/mlx/datapath.conf file; for example:

cumulus@switch:~$ cat /usr/lib/python2.7/dist-packages/cumulus/__chip_config/mlx/datapath.conf | grep -B3 "tcam_resource"
#TCAM resource forwarding profile

    1. Valid profiles -
    2. default, ipmc-heavy, acl-heavy, ipmc-max
       tcam_resource.profile = default

After you specify a different profile, restart switchd for the change to take effect.

When nonatomic updates are enabled (acl.non_atomic_update_mode is set to TRUE in the /etc/cumulus/switchd.conf file), the maximum number of mroute and ACL entries for each profile are:

Profile Mroute Entries ACL Entries
default 1000 500 (IPv6) or 1000 (IPv4)
ipmc-heavy 8500 1000 (IPv6) or 1500 (IPv4)
acl-heavy 450 2000 (IPv6) or 3500 (IPv4)
ipmc-max 13000 1000 (IPv6) or 2000 (IPv4)

When nonatomic updates are disabled (acl.non_atomic_update_mode is set to FALSE in the /etc/cumulus/switchd.conf file), the maximum number of mroute and ACL entries for each profile are:

Profile Mroute Entries ACL Entries
default 1000 250 (IPv6) or 500 (IPv4)
ipmc-heavy 8500 500 (IPv6) or 750 (IPv4)
acl-heavy 450 1000 (IPv6) or 1750 (IPv4)
ipmc-max 13000 500 (IPv6) or 1000 (IPv4)

Route Entry Takes Precedence Over Neighbor Entry

On Broadcom switches running Cumulus Linux 4.0 and later, when there is a /32 IPv4 or /128 IPv6 route and the same prefix is also a neighbor entry in the linux kernel, the route entry takes precedence over the neighbor entry in the forwarding lookup. To change this behavior, update the route_preferred_over_neigh variable to FALSE in the /etc/cumulus/switchd.conf file.