Troubleshooting
Use the following commands to troubleshoot BGP.
Basic Troubleshooting Commands
The following example commands run on a BGP unnumbered configuration and show IPv6 next hops or the interface name for any IPv4 prefix.
To show a summary of the BGP configuration on the switch, run the NCLU net show bgp summary command or the vtysh show ip bgp summary command. For example:
cumulus@switch:~$ net show bgp summary
how bgp ipv4 unicast summary
=============================
BGP router identifier 10.10.10.1, local AS number 65101 vrf-id 0
BGP table version 88
RIB entries 25, using 4800 bytes of memory
Peers 5, using 106 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
spine01(swp51) 4 65199 31122 31194 0 0 0 1d01h44m 7
spine02(swp52) 4 65199 31060 31151 0 0 0 01:47:13 7
spine03(swp53) 4 65199 31150 31207 0 0 0 01:48:31 7
spine04(swp54) 4 65199 31042 31098 0 0 0 01:46:57 7
leaf02(peerlink.4094) 4 65101 30919 30913 0 0 0 01:47:43 12
Total number of neighbors 5
show bgp ipv6 unicast summary
=============================
% No BGP neighbors found
To determine if the sessions above are iBGP or eBGP sessions, look at the ASNs.
To view the routing table as defined by BGP, run the NCLU net show bgp ipv4 unicast command or the vtysh show ip bgp command. For example:
cumulus@leaf01:~$ net show bgp ipv4 unicast
GP table version is 88, local router ID is 10.10.10.1, vrf id 0
Default local pref 100, local AS 65101
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* i10.0.1.1/32 peerlink.4094 0 100 0 ?
*> 0.0.0.0 0 32768 ?
*= 10.0.1.2/32 swp54 0 65199 65102 ?
*= swp52 0 65199 65102 ?
* i peerlink.4094 100 0 65199 65102 ?
*= swp53 0 65199 65102 ?
*> swp51 0 65199 65102 ?
*= 10.0.1.254/32 swp54 0 65199 65132 ?
*= swp52 0 65199 65132 ?
* i peerlink.4094 100 0 65199 65132 ?
*= swp53 0 65199 65132 ?
*> swp51 0 65199 65132 ?
*> 10.10.10.1/32 0.0.0.0 0 32768 ?
*>i10.10.10.2/32 peerlink.4094 0 100 0 ?
*= 10.10.10.3/32 swp54 0 65199 65102 ?
*= swp52 0 65199 65102 ?
* i peerlink.4094 100 0 65199 65102 ?
*= swp53 0 65199 65102 ?
*> swp51 0 65199 65102 ?
...
Displayed 13 routes and 42 total paths
To show a more detailed breakdown of a specific neighbor, run the NCLU net show bgp neighbor <neighbor> command or the vtysh show ip bgp neighbor <neighbor> command:
cumulus@switch:~$ net show bgp neighbor swp51
GP neighbor on swp51: fe80::7c41:fff:fe93:b711, remote AS 65199, local AS 65101, external link
Hostname: spine01
Member of peer-group underlay for session parameters
BGP version 4, remote router ID 10.10.10.101, local router ID 10.10.10.1
BGP state = Established, up for 1d01h47m
Last read 00:00:00, Last write 00:00:00
Hold time is 9, keepalive interval is 3 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
AddPath:
IPv4 Unicast: RX advertised IPv4 Unicast and received
Extended nexthop: advertised and received
Address families by peer:
IPv4 Unicast
Route refresh: advertised and received(old & new)
Address Family IPv4 Unicast: advertised and received
Hostname Capability: advertised (name: leaf01,domain name: n/a) received (name: spine01,domain name: n/a)
Graceful Restart Capability: advertised
Graceful restart information:
Local GR Mode: Helper*
Remote GR Mode: Disable
R bit: False
Timers:
Configured Restart Time(sec): 120
Received Restart Time(sec): 0
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 2 1
Notifications: 0 0
Updates: 309 237
Keepalives: 30942 30943
Route Refresh: 0 0
Capability: 0 0
Total: 31253 31181
Minimum time between advertisement runs is 0 seconds
For address family: IPv4 Unicast
underlay peer-group member
Update group 2, subgroup 2
Packet Queue length 0
Community attribute sent to this neighbor(all)
7 accepted prefixes
Connections established 1; dropped 0
Last reset 1d01h47m, No AFI/SAFI activated for peer
Local host: fe80::2294:15ff:fe02:7bbf, Local port: 179
Foreign host: fe80::7c41:fff:fe93:b711, Foreign port: 45548
Nexthop: 10.10.10.1
Nexthop global: fe80::2294:15ff:fe02:7bbf
Nexthop local: fe80::2294:15ff:fe02:7bbf
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Read thread: on Write thread: on FD used: 30
To see details of a specific route, such as its source and destination, run the NCLU net show bgp <route> command or the vtysh show ip bgp <route> command.
cumulus@switch:~$ net show bgp 10.10.10.3/32
GP routing table entry for 10.10.10.3/32
Paths: (5 available, best #5, table default)
Advertised to non peer-group peers:
spine01(swp51) spine02(swp52) spine03(swp53) spine04(swp54) leaf02(peerlink.4094)
65199 65102
fe80::8e24:2bff:fe79:7d46 from spine04(swp54) (10.10.10.104)
(fe80::8e24:2bff:fe79:7d46) (used)
Origin incomplete, valid, external, multipath
Last update: Wed Oct 7 13:13:13 2020
65199 65102
fe80::841:43ff:fe27:caf from spine02(swp52) (10.10.10.102)
(fe80::841:43ff:fe27:caf) (used)
Origin incomplete, valid, external, multipath
Last update: Wed Oct 7 13:13:14 2020
65199 65102
fe80::90b1:7aff:fe00:3121 from leaf02(peerlink.4094) (10.10.10.2)
Origin incomplete, localpref 100, valid, internal
Last update: Wed Oct 7 13:13:08 2020
65199 65102
fe80::48e7:fbff:fee9:5bcf from spine03(swp53) (10.10.10.103)
(fe80::48e7:fbff:fee9:5bcf) (used)
Origin incomplete, valid, external, multipath
Last update: Wed Oct 7 13:13:13 2020
65199 65102
fe80::7c41:fff:fe93:b711 from spine01(swp51) (10.10.10.101)
(fe80::7c41:fff:fe93:b711) (used)
Origin incomplete, valid, external, multipath, bestpath-from-AS 65199, best (Older Path)
Last update: Wed Oct 7 13:13:13 2020
Troubleshoot BGP Unnumbered
To verify that FRR learns the neighboring link-local IPv6 address through the IPv6 neighbor discovery router advertisements on a given interface, run the NCLU net show interface <interface> command or the vtysh show interface <interface> command.
If you do not enable ipv6 nd suppress-ra on both ends of the interface, Neighbor address(s): shows the link-local address of the other end (the address that BGP uses when that interface uses BGP).
IPv6 route advertisements (RAs) are automatically enabled on an interface with IPv6 addresses. The no ipv6 nd suppress-ra command is not needed for BGP unnumbered.
cumulus@switch:~$ net show interface swp51
Name MAC Speed MTU Mode
-- ----- ----------------- ----- ---- -------
UP swp51 10:d8:68:d4:a6:81 1G 9216 Default
Alias
-----
leaf to spine
cl-netstat counters
-------------------
RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR
----- ------ ------ ------ ----- ------ ------ ------
1874 0 0 0 1252 0 0 0
LLDP Details
------------
LocalPort RemotePort(RemoteHost)
--------- ----------------------
swp51 swp1(spine01)
Routing
-------
Interface swp51 is up, line protocol is up
Link ups: 0 last: (never)
Link downs: 0 last: (never)
PTM status: disabled
vrf: default
OS Description: leaf to spine
index 8 metric 0 mtu 9216 speed 1000
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 10:d8:68:d4:a6:81
inet6 fe80::12d8:68ff:fed4:a681/64
Interface Type Other
protodown: off
ND advertised reachable time is 0 milliseconds
ND advertised retransmit interval is 0 milliseconds
ND advertised hop-count limit is 64 hops
ND router advertisements sent: 217 rcvd: 216
ND router advertisements are sent every 10 seconds
ND router advertisements lifetime tracks ra-interval
ND router advertisement default router preference is medium
Hosts use stateless autoconfig for addresses.
Neighbor address(s):
inet6 fe80::f208:5fff:fe12:cc8c/128
Troubleshoot IPv4 Prefixes Learned with IPv6 Next Hops
To show IPv4 prefixes learned with IPv6 next hops, run the following commands.
The following examples show an IPv4 prefix learned from a BGP peer over an IPv6 session using IPv6 global addresses, but where the next hop installed by BGP is a link-local IPv6 address. This occurs when the session is directly between peers, and the BGP update for the prefix includes both link-local and global IPv6 addresses as next hops. If both global and link-local next hops exist, BGP prefers the link-local address for route installation.
cumulus@spine01:mgmt:~$ net show bgp ipv4 unicast summary
BGP router identifier 10.10.10.101, local AS number 65199 vrf-id 0
BGP table version 3
RIB entries 3, using 576 bytes of memory
Peers 1, using 21 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
leaf01(2001:db8:2::a00:1) 4 65101 22 22 0 0 0 00:01:00 0
Total number of neighbors 1
cumulus@spine01:mgmt:~$ net show bgp ipv4 unicast
BGP table version is 3, local router ID is 10.10.10.101, vrf id 0
Default local pref 100, local AS 65199
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
10.10.10.101/32 fe80::a00:27ff:fea6:b9fe 0 0 32768 i
Displayed 1 routes and 1 total paths
cumulus@spine01:~$ net show bgp ipv4 unicast 10.10.10.101/32
BGP routing table entry for 10.10.10.101/32
Paths: (1 available, best #1, table default)
Advertised to non peer-group peers:
Leaf01(2001:db8:0002::0a00:1)
3
2001:db8:0002::0a00:1 from Leaf01(2001:db8:0002::0a00:1) (10.10.10.101)
(fe80::a00:27ff:fea6:b9fe) (used)
Origin IGP, metric 0, valid, external, bestpath-from-AS 3, best (First path received)
AddPath ID: RX 0, TX 3
Last update: Mon Oct 22 08:09:22 2018
The example output below shows the results of installing the route in the FRR RIB as well as the kernel FIB. The next hop installed in the FRR RIB is the link-local IPv6 address, which Cumulus Linux converts into an IPv4 link-local address, as required for installation into the kernel FIB.
cumulus@spine01:~$ net show route 10.10.10.101/32
RIB entry for 10.10.10.101/32
===========================
Routing entry for 10.10.10.101/32
Known via "bgp", distance 20, metric 0, best
Last update 2d17h05m ago
* fe80::a00:27ff:fea6:b9fe, via swp1
FIB entry for 10.10.10.101/32
===========================
10.10.10.101/32 via 10.0.1.0 dev swp1 proto bgp metric 20 onlink
If BGP learns an IPv4 prefix with only an IPv6 global next hop address (when it learns the route through a route reflector), the command output shows the IPv6 global address as the next hop value. The command also shows that it learns recursively through the link-local address of the route reflector. When you use a global IPv6 address as a next hop for route installation in the FRR RIB, the switch still converts it into an IPv4 link-local address for installation into the kernel.
cumulus@leaf01:~$ net show bgp ipv4 unicast summary
BGP router identifier 10.10.10.1, local AS number 65101 vrf-id 0
BGP table version 1
RIB entries 1, using 152 bytes of memory
Peers 1, using 19 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
Spine01(2001:db8:0002::0a00:2) 4 1 74 68 0 0 0 00:00:45 1
Total number of neighbors 1
cumulus@leaf01:~$ net show bgp ipv4 unicast
BGP table version is 1, local router ID is 10.10.10.1
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i10.1.10.0/24 2001:2:2::4 0 100 0 i
Displayed 1 routes and 1 total paths
cumulus@leaf01:~$ net show bgp ipv4 unicast 10.10.10.101/32
BGP routing table entry for 10.10.10.101/32
Paths: (1 available, best #1, table default)
Not advertised to any peer
Local
2001:2:2::4 from Spine01(2001:1:1::1) (10.10.10.104)
Origin IGP, metric 0, localpref 100, valid, internal, bestpath-from-AS Local, best (First path received)
Originator: 10.0.0.14, Cluster list: 10.10.10.111
AddPath ID: RX 0, TX 5
Last update: Mon Oct 22 14:25:30 2018
cumulus@leaf01:~$ net show route 10.10.10.1/32
RIB entry for 10.10.10.1/32
===========================
Routing entry for 10.10.10.1/32
Known via "bgp", distance 200, metric 0, best
Last update 00:01:13 ago
2001:2:2::4 (recursive)
* fe80::a00:27ff:fe5a:84ae, via swp1
FIB entry for 10.10.10.1/32
===========================
10.10.10.1/32 via 10.0.1.1 dev swp1 proto bgp metric 20 onlink
To only use IPv6 global addresses for route installation into the FRR RIB, you must add an additional route map to the neighbor or peer group statement in the appropriate address family. When the route map command set ipv6 next-hop prefer-global applies to a neighbor, if both a link-local and global IPv6 address are in the BGP update for a prefix, BGP uses the IPv6 global address for route installation.
With this additional configuration, the output in the FRR RIB changes in the direct neighbor case as shown below:
router bgp 65101
bgp router-id 10.10.10.1
neighbor 2001:db8:2::a00:1 remote-as internal
neighbor 2001:db8:2::a00:1 capability extended-nexthop
!
address-family ipv4 unicast
neighbor 2001:db8:2::a00:1 route-map GLOBAL in
exit-address-family
!
route-map GLOBAL permit 20
set ipv6 next-hop prefer-global
!
The resulting FRR RIB output is as follows:
cumulus@leaf01:~$ net show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR,
> - selected route, * - FIB route
B 0.0.0.0/0 [200/0] via 2001:2:2::4, swp2, 00:01:00
K 0.0.0.0/0 [0/0] via 10.0.2.2, eth0, 1d02h29m
C>* 10.0.0.9/32 is directly connected, lo, 5d18h32m
C>* 10.0.2.0/24 is directly connected, eth0, 03:51:31
B>* 172.16.4.0/24 [200/0] via 2001:2:2::4, swp2, 00:01:00ß
C>* 172.16.10.0/24 is directly connected, swp3, 5d18h32m
When the switch learns the route through a route reflector, it appears like this:
router bgp 65101
bgp router-id 10.10.10.1
neighbor 2001:db8:2::a00:2 remote-as internal
neighbor 2001:db8:2::a00:2 capability extended-nexthop
!
address-family ipv6 unicast
neighbor 2001:db8:2::a00:2 activate
neighbor 2001:db8:2::a00:2 route-map GLOBAL in
exit-address-family
!
route-map GLOBAL permit 10
set ipv6 next-hop prefer-global
cumulus@leaf01:~$ net show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR,
> - selected route, * - FIB route
B 0.0.0.0/0 [200/0] via 2001:2:2::4, 00:00:01
K 0.0.0.0/0 [0/0] via 10.0.2.2, eth0, 3d00h26m
C>* 10.0.0.8/32 is directly connected, lo, 3d00h26m
C>* 10.0.2.0/24 is directly connected, eth0, 03:39:18
C>* 172.16.3.0/24 is directly connected, swp2, 3d00h26m
B> 172.16.4.0/24 [200/0] via 2001:2:2::4 (recursive), 00:00:01
* via 2001:1:1::1, swp1, 00:00:01
C>* 172.16.10.0/24 is directly connected, swp3, 3d00h26m
Check BGP Timer Settings
To check BGP timers, such as the BGP keepalive interval, hold time, and advertisement interval, run the NCLU net show bgp neighbor <peer> command or the vtysh show ip bgp neighbor <peer> command. For example:
cumulus@leaf01:~$ net show bgp neighbor swp51
GP neighbor on swp51: fe80::f208:5fff:fe12:cc8c, remote AS 65199, local AS 65101, external link
Hostname: spine01
Member of peer-group underlay for session parameters
BGP version 4, remote router ID 10.10.10.101, local router ID 10.10.10.1
BGP state = Established, up for 06:50:58
Last read 00:00:03, Last write 00:00:03
Hold time is 9, keepalive interval is 3 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
AddPath:
IPv4 Unicast: RX advertised IPv4 Unicast and received
Extended nexthop: advertised and received
Address families by peer:
IPv4 Unicast
Route refresh: advertised and received(old & new)
Address Family IPv4 Unicast: advertised and received
Hostname Capability: advertised (name: leaf01,domain name: n/a) received (name: spine01,domain name: n/a)
Graceful Restart Capability: advertised and received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart information:
End-of-RIB send: IPv4 Unicast
End-of-RIB received: IPv4 Unicast
Local GR Mode: Helper*
Remote GR Mode: Helper
R bit: True
Timers:
Configured Restart Time(sec): 120
Received Restart Time(sec): 120
IPv4 Unicast:
F bit: False
End-of-RIB sent: Yes
End-of-RIB sent after update: No
End-of-RIB received: Yes
Timers:
Configured Stale Path Time(sec): 360
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 2 1
Notifications: 0 0
Updates: 54 59
Keepalives: 8219 8219
Route Refresh: 0 0
Capability: 0 0
Total: 8275 8279
Minimum time between advertisement runs is 0 seconds
Neighbor State Change Log
Cumulus Linux records the changes that a neighbor goes through in syslog and in the /var/log/frr/frr.log file. For example:
020-10-05T15:51:32.621773-07:00 leaf01 bgpd[10104]: %NOTIFICATION: sent to neighbor peerlink.4094 6/7 (Cease/Connection collision resolution) 0 bytes
2020-10-05T15:51:32.623023-07:00 leaf01 bgpd[10104]: %ADJCHANGE: neighbor peerlink.4094(leaf02) in vrf default Up
2020-10-05T15:51:32.623156-07:00 leaf01 bgpd[10104]: %NOTIFICATION: sent to neighbor peerlink.4094 6/7 (Cease/Connection collision resolution) 0 bytes
2020-10-05T15:51:32.623496-07:00 leaf01 bgpd[10104]: %ADJCHANGE: neighbor peerlink.4094(leaf02) in vrf default Down No AFI/SAFI activated for peer
2020-10-05T15:51:33.040332-07:00 leaf01 bgpd[10104]: [EC 33554454] swp53 [Error] bgp_read_packet error: Connection reset by peer
2020-10-05T15:51:33.279468-07:00 leaf01 bgpd[10104]: [EC 33554454] swp52 [Error] bgp_read_packet error: Connection reset by peer
2020-10-05T15:51:33.339487-07:00 leaf01 bgpd[10104]: %ADJCHANGE: neighbor swp54(spine04) in vrf default Up
2020-10-05T15:51:33.340893-07:00 leaf01 bgpd[10104]: %ADJCHANGE: neighbor swp53(spine03) in vrf default Up
2020-10-05T15:51:33.341648-07:00 leaf01 bgpd[10104]: %ADJCHANGE: neighbor swp52(spine02) in vrf default Up
2020-10-05T15:51:33.342369-07:00 leaf01 bgpd[10104]: %ADJCHANGE: neighbor swp51(spine01) in vrf default Up
2020-10-05T15:51:33.627958-07:00 leaf01 bgpd[10104]: %ADJCHANGE: neighbor peerlink.4094(leaf02) in vrf default Up