FRRouting
Cumulus Linux uses FRR to provide the routing protocols for dynamic routing and supports the following routing protocols:
- Open Shortest Path First (v2 and v3)
- Border Gateway Protocol (BGP)
- Protocol Independent Multicast (PIM)
- Policy-based Routing (PBR)
Architecture
The FRR suite consists of various protocol-specific daemons and a protocol-independent daemon called zebra. Each of the protocol-specific daemons are responsible for running the relevant protocol and building the routing table based on the information exchanged.
It is not uncommon to have more than one protocol daemon running at the same time. For example, at the edge of an enterprise, protocols internal to an enterprise such as OSPF run alongside the protocols that connect an enterprise to the rest of the world such as BGP.
zebra is the daemon that resolves the routes provided by multiple protocols (including the static routes you specify) and programs these routes in the Linux kernel using netlink (in Linux). The FRRouting documentation defines zebra as the IP routing manager for FRR that provides kernel routing table updates, interface lookups, and redistribution of routes between different routing protocols.
Configure FRR
The information in this section does not apply if you use NVUE to configure your switch. NVUE manages FRR daemons and configuration automatically. These instructions are only applicable for users managing FRR directly through linux flat file configurations.
If you do not configure your system using NVUE, FRR does not start by default in Cumulus Linux. Before you run FRR, make sure you have enabled the relevant daemons that you intend to use (bgpd, ospfd, ospf6d, pimd, or pbrd) in the /etc/frr/daemons file.
Cumulus Linux enables the zebra daemon by default. You can enable the other daemons according to how you plan to route your network.
Before you start FRR, edit the /etc/frr/daemons file to enable each daemon you want to use. For example, to enable BGP, set bgpd to yes:
...
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
fabricd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
vrrpd=no
...
Enable and Start FRR
The information in this section does not apply if you use NVUE to configure your switch. NVUE manages FRR daemons and configuration automatically. These instructions are only applicable for users managing FRR directly through linux flat file configurations.
After you enable the appropriate daemons, enable and start the FRR service:
cumulus@switch:~$ sudo systemctl enable frr.service
cumulus@switch:~$ sudo systemctl start frr.service
- All the routing protocol daemons (
bgpd,ospfd,ospf6d,ripd,ripngd,isisdandpimd) are dependent onzebra. When you start FRR,systemddetermines whether zebra is running; if zebra is not running,systemdstartszebra, then starts the dependent service, such asbgpd. - If you restart a service, its dependent services also restart. For example, running
systemctl restart frr.servicerestarts any of the enabled routing protocol daemons that are running. - For more information on the
systemctlcommand and changing the state of daemons, see Services and Daemons in Cumulus Linux.
Restore the Default Configuration
The information in this section does not apply if you use NVUE to configure your switch. NVUE manages FRR daemons and configuration automatically. These instructions are only applicable if you manage FRR directly with linux flat file configurations.
If you need to restore the FRR configuration to the default running configuration, delete the frr.conf file and restart the frr service.
Back up frr.conf (or any configuration files you want to remove) before proceeding.
-
Confirm that
service integrated-vtysh-configis running. -
Remove
/etc/frr/frr.conf:cumulus@switch:~$ sudo rm /etc/frr/frr.conf -
Restart FRR with this command:
cumulus@switch:~$ sudo systemctl restart frr.serviceRestarting FRR restarts all the routing protocol daemons that you enable and are running. NVIDIA recommends that you reboot the switch instead of restarting the FRR service to minimize traffic impact when redundant switches are present with MLAG.
Interface IP Addresses and VRFs
FRR inherits the IP addresses and any associated routing tables for the network interfaces from the /etc/network/interfaces file. This is the recommended way to define the addresses; do not create interfaces using FRR. For more information, see Configure IP Addresses and Virtual Routing and Forwarding - VRF.
vtysh Modal CLI
FRR provides a command-line interface (CLI) called vtysh for configuring and displaying protocol state. To start the CLI, run the sudo vtysh command:
cumulus@switch:~$ sudo vtysh
Hello, this is FRRouting (version 8.4.3).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
switch#
FRR provides different modes to the CLI and certain commands are only available within a specific mode. Configuration is available with the configure terminal command:
switch# configure terminal
switch(config)#
The prompt displays the current CLI mode. For example, when you run the interface-specific commands, the prompt changes to:
switch(config)# interface swp1
switch(config-if)#
When you run the routing protocol specific commands, the prompt changes:
switch(config)# router ospf
switch(config-router)#
? displays the list of available top-level commands:
switch(config-if)# ?
bandwidth Set bandwidth informational parameter
description Interface specific description
end End current mode and change to enable mode
exit Exit current mode and down to previous mode
ip IP Information
ipv6 IPv6 Information
isis IS-IS commands
link-detect Enable link detection on interface
list Print command list
mpls-te MPLS-TE specific commands
multicast Set multicast flag to interface
no Negate a command or set its defaults
ptm-enable Enable neighbor check with specified topology
quit Exit current mode and down to previous mode
shutdown Shutdown the selected interface
?-based completion is also available to see the parameters that a command takes:
switch(config-if)# bandwidth ?
<1-10000000> Bandwidth in kilobits
switch(config-if)# ip ?
address Set the IP address of an interface
irdp Alter ICMP Router discovery preference this interface
ospf OSPF interface commands
rip Routing Information Protocol
router IP router interface commands
In addition to ?-based completion, you can use tab completion to get help with the valid keywords or options as you enter commands. For example, using tab completion with router ospf shows the possible options for the command and returns you to the command prompt to complete the command.
switch(config)# router ospf vrf<<press tab>>
BLUE RED default mgmt
switch(config)# router ospf vrf
To search for specific vtysh commands so that you can identify the correct syntax to use, run the sudo vtysh -c 'find <term>' command. For example, to show only commands that include mlag:
cumulus@leaf01:mgmt:~$ sudo vtysh -c 'find mlag'
(view) show ip pim [mlag] vrf all interface [detail|WORD] [json]
(view) show ip pim [vrf NAME] interface [mlag] [detail|WORD] [json]
(view) show ip pim [vrf NAME] mlag upstream [A.B.C.D [A.B.C.D]] [json]
(view) show ip pim mlag summary [json]
(view) show ip pim vrf all mlag upstream [json]
(view) show zebra mlag
(enable) [no$no] debug zebra mlag
(enable) debug pim mlag
(enable) no debug pim mlag
(enable) test zebra mlag <none$none|primary$primary|secondary$secondary>
(enable) show ip pim [mlag] vrf all interface [detail|WORD] [json]
(enable) show ip pim [vrf NAME] interface [mlag] [detail|WORD] [json]
(enable) show ip pim [vrf NAME] mlag upstream [A.B.C.D [A.B.C.D]] [json]
(enable) show ip pim mlag summary [json]
(enable) show ip pim vrf all mlag upstream [json]
(enable) show zebra mlag
(config) [no$no] debug zebra mlag
(config) debug pim mlag
(config) ip pim mlag INTERFACE role [primary|secondary] state [up|down] addr A.B.C.D
(config) no debug pim mlag
(config) no ip pim mlag
You can display the state at any level, including the top level. For example, to see the routing table as seen by zebra:
switch# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, T - Table,
> - selected route, * - FIB route
B>* 0.0.0.0/0 [20/0] via fe80::4638:39ff:fe00:c, swp29, 00:11:57
* via fe80::4638:39ff:fe00:52, swp30, 00:11:57
B>* 10.0.0.1/32 [20/0] via fe80::4638:39ff:fe00:c, swp29, 00:11:57
* via fe80::4638:39ff:fe00:52, swp30, 00:11:57
B>* 10.0.0.11/32 [20/0] via fe80::4638:39ff:fe00:5b, swp1, 00:11:57
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:2e, swp2, 00:11:58
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:57, swp3, 00:11:59
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:43, swp4, 00:11:59
C>* 10.0.0.21/32 is directly connected, lo
B>* 10.0.0.51/32 [20/0] via fe80::4638:39ff:fe00:c, swp29, 00:11:57
* via fe80::4638:39ff:fe00:52, swp30, 00:11:57
B>* 172.16.1.0/24 [20/0] via fe80::4638:39ff:fe00:5b, swp1, 00:11:57
* via fe80::4638:39ff:fe00:2e, swp2, 00:11:57
B>* 172.16.3.0/24 [20/0] via fe80::4638:39ff:fe00:57, swp3, 00:11:59
* via fe80::4638:39ff:fe00:43, swp4, 00:11:59
To run the same command at a config level, prepend do:
switch(config-router)# do show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, T - Table,
> - selected route, * - FIB route
B>* 0.0.0.0/0 [20/0] via fe80::4638:39ff:fe00:c, swp29, 00:05:17
* via fe80::4638:39ff:fe00:52, swp30, 00:05:17
B>* 10.0.0.1/32 [20/0] via fe80::4638:39ff:fe00:c, swp29, 00:05:17
* via fe80::4638:39ff:fe00:52, swp30, 00:05:17
B>* 10.0.0.11/32 [20/0] via fe80::4638:39ff:fe00:5b, swp1, 00:05:17
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:2e, swp2, 00:05:18
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:57, swp3, 00:05:18
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:43, swp4, 00:05:18
C>* 10.0.0.21/32 is directly connected, lo
B>* 10.0.0.51/32 [20/0] via fe80::4638:39ff:fe00:c, swp29, 00:05:17
* via fe80::4638:39ff:fe00:52, swp30, 00:05:17
B>* 172.16.1.0/24 [20/0] via fe80::4638:39ff:fe00:5b, swp1, 00:05:17
* via fe80::4638:39ff:fe00:2e, swp2, 00:05:17
B>* 172.16.3.0/24 [20/0] via fe80::4638:39ff:fe00:57, swp3, 00:05:18
* via fe80::4638:39ff:fe00:43, swp4, 00:05:18
To run single commands with vtysh, use the -c option:
cumulus@switch:~$ sudo vtysh -c 'sh ip route'
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, A - Babel,
> - selected route, * - FIB route
K>* 0.0.0.0/0 via 192.168.0.2, eth0
C>* 192.0.2.11/24 is directly connected, swp1
C>* 192.0.2.12/24 is directly connected, swp2
B>* 203.0.113.30/24 [200/0] via 192.0.2.2, swp1, 11:05:10
B>* 203.0.113.31/24 [200/0] via 192.0.2.2, swp1, 11:05:10
B>* 203.0.113.32/24 [200/0] via 192.0.2.2, swp1, 11:05:10
C>* 127.0.0.0/8 is directly connected, lo
C>* 192.168.0.0/24 is directly connected, eth0
To run a command multiple levels down:
cumulus@switch:~$ sudo vtysh -c 'configure terminal' -c 'router ospf' -c 'area 0.0.0.1 range 10.10.10.0/24'
The commands also take a partial command name (for example, sh ip route) as long as the partial command name is not aliased:
cumulus@switch:~$ sudo vtysh -c 'sh ip r'
% Ambiguous command.
To disable a command or feature in FRR, prepend the command with no. For example:
cumulus@switch:~$ sudo vtysh
switch# configure terminal
switch(config)# router ospf
switch(config-router)# no area 0.0.0.1 range 10.10.10.0/24
switch(config-router)# exit
switch(config)# exit
switch# write memory
switch# exit
cumulus@switch:~$
To view the current state of the configuration, run the show running-config command:
If you try to configure a routing protocol that is not running, vtysh ignores those commands.
NVUE Show Commands and vtysh Output
NVUE provides the --output raw option for certain NVUE show commands to show vtysh native output.
Show Routes in the Routing Table
To show all the routes in the routing table, run the nv show vrf <vrf> router rib <address-family> route command:
cumulus@switch:~$ nv show vrf default router rib ipv4 route
Flags - * - selected, q - queued, o - offloaded, i - installed, S - fib-
selected, x - failed
Route Protocol Distance Uptime NHGId Metric Flags
--------------- --------- -------- -------------------- ----- ------ -----
10.0.1.12/32 connected 0 2024-10-22T18:36:01Z 15 0 *Sio
10.0.1.34/32 bgp 20 2024-10-22T18:42:22Z 125 0 *Si
10.0.1.255/32 bgp 20 2024-10-22T18:36:05Z 125 0 *Si
10.10.10.1/32 connected 0 2024-10-22T18:35:54Z 15 0 *Sio
10.10.10.2/32 bgp 20 2024-10-22T18:35:58Z 62 0 *Si
10.10.10.3/32 bgp 20 2024-10-22T18:42:16Z 125 0 *Si
10.10.10.4/32 bgp 20 2024-10-22T18:42:16Z 125 0 *Si
10.10.10.63/32 bgp 20 2024-10-22T18:36:05Z 125 0 *Si
10.10.10.64/32 bgp 20 2024-10-22T18:36:05Z 125 0 *Si
10.10.10.101/32 bgp 20 2024-10-22T18:36:05Z 115 0 *Si
10.10.10.102/32 bgp 20 2024-10-22T18:36:04Z 107 0 *Si
To show information about a specific route, run the nv show vrf <vrf> router rib <address-family> route <prefix> command:
cumulus@switch:~$ nv show vrf default router rib ipv4 route 10.0.1.34/32
route-entry
==============
Protocol - Protocol name, TblId - Table Id, NHGId - Nexthop group Id, Flags - u
- unreachable, r - recursive, o - onlink, i - installed, d - duplicate, c -
connected, A - active
EntryIdx Protocol TblId NHGId Distance Metric ResolvedVia ResolvedViaIntf Weight Flags
-------- -------- ----- ----- -------- ------ ------------------------- --------------- ------ -----
1 bgp 254 125 20 0 fe80::4ab0:2dff:fe32:2a3f swp52 1 iA
fe80::4ab0:2dff:fe41:6b79 swp51 1 iA
To show the total number of routes in the routing table, run the nv show vrf <vrf> router rib <address-family> route-count command:
cumulus@switch:~$ nv show vrf default router rib ipv4 route-count
operational
------------ -----------
total-routes 34
[protocol] bgp
[protocol] connected
For IPv6 run the nv show vrf <vrf> router rib ipv6 route-count command.
To show the total number of routes per protocol in the routing table, run the nv show vrf <vrf> router rib <address-family> route-count protocol command:
cumulus@switch:~$ nv show vrf default router rib ipv4 route-count protocol
Protocol Total
--------- -----
bgp 6
connected 3
ospf 8
static 3
For IPv6 run the nv show vrf <vrf> router rib ipv6 route-count protocol command.
Look Up the Route for a Destination
To look up the route in the routing table for a specific destination, run the nv action lookup vrf <vrf-id> router fib <address-family> <ip-address> command.
The following example looks up the route in the routing table for the destination with the IPv4 address 10.10.10.3:
cumulus@switch:~$ nv action lookup vrf default router fib ipv4 10.10.10.3
Action executing ...
[{"dst":"10.10.10.3","nhid":455,"table":"default","protocol":"bgp","metric":20,"flags":[]}]
Action succeeded
The following example shows the route in the routing table for the destination with the IPv6 address 228:35::5
cumulus@switch:~$ nv action lookup vrf RED router fib ipv6 228:35::5
[{"dst":"228:35::5","nhid":454,"table":"RED","protocol":"bgp","metric":20,"flags":[],"pref":"medium"}]
Action succeeded
Next Hop Tracking
Routing daemons track the validity of next hops through notifications from the zebra daemon. For example, FRR uninstalls BGP routes that resolve to a next hop over a connected route in zebra when bgpd receives a next hop tracking (NHT) notification after zebra removes the connected route if the associated interface goes down.
The zebra daemon does not consider next hops that resolve to a default route as valid by default. You can configure NHT to consider a longest prefix match lookup for next hop addresses resolving to the default route as a valid next hop. The following example configures the default route to be valid for NHT in the default VRF:
cumulus@leaf01:~$ nv set vrf default router nexthop-tracking ipv4 resolved-via-default on
cumulus@leaf01:~$ nv config apply
cumulus@leaf01:~$ sudo vtysh
leaf01# configure terminal
leaf01(config)# ip nht resolve-via-default
leaf01(config)# end
leaf01# write memory
leaf01# exit
cumulus@leaf01:~$
You can apply a route map to NHT for specific routing daemons to permit or deny routes from consideration as valid next hops. The following example applies ROUTEMAP1 to BGP, preventing NHT from considering next hops resolving to 10.0.0.0/8 in the default VRF as valid:
cumulus@leaf01:~$ nv set router policy prefix-list PREFIX1 type ipv4
cumulus@leaf01:~$ nv set router policy prefix-list PREFIX1 rule 1 match 10.0.0.0/8
cumulus@leaf01:~$ nv set router policy prefix-list PREFIX1 rule 1 action permit
cumulus@leaf01:~$ nv set router policy route-map ROUTEMAP1 rule 1 match ip-prefix-list PREFIX1
cumulus@leaf01:~$ nv set router policy route-map ROUTEMAP1 rule 1 action deny
cumulus@leaf01:~$ nv set router policy route-map ROUTEMAP1 rule 2 action permit
cumulus@leaf01:~$ nv set vrf default router nexthop-tracking ipv4 route-map ROUTEMAP1 protocol bgp
cumulus@leaf01:~$ nv config apply
cumulus@leaf01:~$ sudo vtysh
leaf02# configure terminal
leaf02(config)# ip prefix-list PREFIX1 seq 1 permit 10.0.0.0/8
leaf02(config)# route-map ROUTEMAP1 deny 1
leaf02(config-route-map)# match ip address prefix-list PREFIX1
leaf02(config-route-map)# route-map ROUTEMAP1 permit 2
leaf02(config-route-map)# ip nht bgp route-map ROUTEMAP1
leaf02(config)# end
leaf01# write memory
leaf01# exit
cumulus@leaf01:~$
You can show tracked next hops with the following NVUE commands:
nv show vrf <vrf> router nexthop-tracking ipv4nv show vrf <vrf> router nexthop-tracking ipv4 <ip-address>nv show vrf <vrf> router nexthop-tracking ipv6nv show vrf <vrf> router nexthop-tracking ipv6 <ip-address>
cumulus@leaf01:~$ nv show vrf default router nexthop-tracking ipv4
operational applied pending
-------------------- ----------- ------- -------
resolved-via-default on
route-map
============
No Data
ip-address
=============
DirectlyConnected - Indicates if nexthop is directly connected or not,
ResolvedProtocol - Resolved via protocol, Interface - Resolved via interface,
ProtocolFiltered - Indicates whether protocol filtered or not, Flags - o -
onlink, c - directly-connected, A - active
IPAddress DirectlyConnected ResolvedProtocol Interface VRF Weight ProtocolFiltered Flags
----------- ----------------- ---------------- ------------- ------- ------ ---------------- -----
10.0.1.34 off bgp swp52 default 1 off A
swp53 default 1 A
swp54 default 1 A
swp51 default 1 A
10.10.10.2 off bgp peerlink.4094 default 1 off A
10.10.10.3 off bgp swp52 default 1 off A
swp53 default 1 A
swp54 default 1 A
swp51 default 1 A
10.10.10.4 off bgp swp52 default 1 off A
swp53 default 1 A
swp54 default 1 A
swp51 default 1 A
10.10.10.63 off bgp swp52 default 1 off A
swp53 default 1 A
swp54 default 1 A
swp51 default 1 A
10.10.10.64 off bgp swp52 default 1 off A
swp53 default 1 A
swp54 default 1 A
swp51 default 1 A
You can also run the vtysh show ip nht vrf <vrf> <ip-address> command.
Reload the FRR Configuration
The information in this section does not apply if you use NVUE to configure your switch. NVUE manages FRR daemons and configuration automatically. These instructions are only applicable for users managing FRR directly through linux flat file configurations.
If you make a change to your routing configuration, you need to reload FRR so your changes take place. FRR reload enables you to apply only the modifications you make to your FRR configuration, synchronizing its running state with the configuration in /etc/frr/frr.conf. This is useful for optimizing FRR automation in your environment or to apply changes made at runtime.
To reload your FRR configuration after you modify /etc/frr/frr.conf, run:
cumulus@switch:~$ sudo systemctl reload frr.service
Examine the running configuration and verify that it matches the configuration in /etc/frr/frr.conf.
If the running configuration is not what you expect, submit a support request and supply the following information:
- The current running configuration (run
show running-configand output the contents to a file) - The contents of
/etc/frr/frr.conf - The contents of
/var/log/frr/frr-reload.log
FRR Logging
The information in this section does not apply if you use NVUE to configure your switch. NVUE manages FRR daemons and configuration automatically. These instructions are only applicable for users managing FRR directly through linux flat file configurations.
By default, Cumulus Linux configures FFR with syslog severity level 6 (informational). Log output writes to the /var/log/frr/frr.log file.
To write debug messages to the log file, you must run the log syslog debug command to configure FRR with syslog severity 7 (debug); otherwise, when you issue a debug command such as, debug bgp neighbor-events, no output goes to /var/log/frr/frr.log. However, when you manually define a log target with the log file /var/log/frr/debug.log command, FRR automatically defaults to severity 7 (debug) logging and the output logs to /var/log/frr/debug.log.
Considerations
Duplicate Hostnames
The switch can have two hostnames in the FRR configuration. For example:
cumulus@spine01:~$ sudo vtysh...
spine01# configure terminal
spine01(config)# hostname spine01-1
spine01-1(config)# do sh run
Building configuration...
Current configuration:
!
frr version 7.0+cl4u3
frr defaults datacenter
hostname spine01
hostname spine01-1
...
If you configure the same numbered BGP neighbor with both the neighbor x.x.x.x and neighbor swp# interface commands, two neighbor entries are present for the same IP address in the configuration. To correct this issue, update the configuration and restart the FRR service.
TCP Sockets and BGP Peering Sessions
The FRR startup configuration includes a setting for the maximum number of open files allowed. For BGP, open files include TCP sockets that BGP connections use. Either BGP speaker can start a BGP peering almost simultaneously; therefore, you can have two TCP sockets for a single BGP peer. These two sockets exist until the BGP protocol determines which socket to use, then the other socket closes.
The default setting of 1024 open files supports up to 512 BGP peering sessions. If you expect your network deployment to have more BGP peering sessions, you need to update this setting.
NVIDIA recommends you set the value to at least twice the maximum number of BGP peering sessions you expect.
To update the open files setting:
-
Edit the
/lib/systemd/system/frr.servicefile and change theLimitNOFILEparameter. The following example sets theLimitNOFILEparameter to 4096.cumulus@switch:~$ sudo cat /lib/systemd/system/frr.service [Unit] Description=FRRouting Documentation=https://frrouting.readthedocs.io/en/latest/setup.html After=networking.service csmgrd.service [Service] Nice=-5 Type=forking NotifyAccess=all StartLimitInterval=3m StartLimitBurst=3 TimeoutSec=2m WatchdogSec=60s RestartSec=5 Restart=on-abnormal LimitNOFILE=4096 ... -
Restart the FRR service.
cumulus@switch:~$ sudo systemctl restart frr.service