FRR not able to establish a BGP session with the host itself

[enkelprifti98]

Describe the bug

FRR fails to connect to other BGP speakers running on the same host. An example is running MetalLB inside a kubernetes cluster and trying to peer with FRR running as a host process. This setup works with BIRD but FRR doesn't seem to be able to establish a connection:

Jun 08 12:20:38 dc-worker-2 bgpd[28366]: [VX6SM-8YE5W][EC 33554460] 127.0.0.1: nexthop_set failed, resetting connection - intf 0x0
Jun 08 12:20:38 dc-worker-2 bgpd[28366]: [NQGZV-Y3W62][EC 100663299] bgp_connect_success: bgp_getsockname(): failed for peer 127.0.0.1, fd 26

• [ x ] Did you check if this is a duplicate issue?
• [ x ] Did you test it on the latest FRRouting/frr master branch?

To Reproduce

Describe the steps to reproduce the behavior.
Be as descriptive as possible.

1. Install FRR and another BGP speaker running as a container.
2. Add BGP peers on both speakers with the same host IP and ASNs

Expected behavior

FRR should be able to establish a BGP session to localhost/self host IPs.

Versions

• OS Version: Ubuntu 20.04

• Kernel: 5.13.0-44-generic

• FRR Version: frr version 8.2.2

Additional context

Does FRR support peering to the host itself? If not, is it possible to add support for this use case since other BGP speakers like BIRD support it?

[ton31337]

What are your /etc/frr/daemons and the configuration?

[enkelprifti98]

FRR daemons: (basically only enabling bgpd)

# This file tells the frr package which daemons to start.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activating a daemon for the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
#
# The watchfrr, zebra and staticd daemons are always started.
#
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no
fabricd=no
vrrpd=no
pathd=no

#
# If this option is set the /etc/init.d/frr script automatically loads
# the config via "vtysh -b" when the servers are started.
# Check /etc/pam.d/frr if you intend to use "vtysh"!
#
vtysh_enable=yes
zebra_options="  -A 127.0.0.1 -s 90000000"
bgpd_options="   -A 127.0.0.1"
ospfd_options="  -A 127.0.0.1"
ospf6d_options=" -A ::1"
ripd_options="   -A 127.0.0.1"
ripngd_options=" -A ::1"
isisd_options="  -A 127.0.0.1"
pimd_options="   -A 127.0.0.1"
ldpd_options="   -A 127.0.0.1"
nhrpd_options="  -A 127.0.0.1"
eigrpd_options=" -A 127.0.0.1"
babeld_options=" -A 127.0.0.1"
sharpd_options=" -A 127.0.0.1"
pbrd_options="   -A 127.0.0.1"
staticd_options="-A 127.0.0.1"
bfdd_options="   -A 127.0.0.1"
fabricd_options="-A 127.0.0.1"
vrrpd_options="  -A 127.0.0.1"
pathd_options="  -A 127.0.0.1"

# configuration profile
#
#frr_profile="traditional"
#frr_profile="datacenter"

#
# This is the maximum number of FD's that will be available.
# Upon startup this is read by the control files and ulimit
# is called.  Uncomment and use a reasonable value for your
# setup if you are expecting a large number of peers in
# say BGP.
#MAX_FDS=1024

# The list of daemons to watch is automatically generated by the init script.
#watchfrr_options=""

# To make watchfrr create/join the specified netns, use the following option:
#watchfrr_options="--netns"
# This only has an effect in /etc/frr/<somename>/daemons, and you need to
# start FRR with "/usr/lib/frr/frrinit.sh start <somename>".

# for debugging purposes, you can specify a "wrap" command to start instead
# of starting the daemon directly, e.g. to use valgrind on ospfd:
#   ospfd_wrap="/usr/bin/valgrind"
# or you can use "all_wrap" for all daemons, e.g. to use perf record:
#   all_wrap="/usr/bin/perf record --call-graph -"
# the normal daemon command is added to this at the end.

FRR configuration:

frr defaults traditional
log syslog informational
ipv6 forwarding
service integrated-vtysh-config
!
!
router bgp 65000
 bgp ebgp-requires-policy
 neighbor METALLB peer-group
 neighbor METALLB remote-as 65000
 neighbor 127.0.0.1 peer-group METALLB
 !
 address-family ipv4 unicast
  redistribute connected
  neighbor METALLB route-map IMPORT in
 exit-address-family
 !
route-map EXPORT deny 100
!
route-map EXPORT permit 1
 match interface lo
!
route-map IMPORT permit 1
!
line vty
!

[enkelprifti98]

MetalLB config:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 127.0.0.1
      peer-asn: 65000
      my-asn: 65000

[ton31337]

Are you trying to establish between 127.0.0.1 on the same 179 (default) port?

[enkelprifti98]

Yep

[klausenbusk]

Did you figure out a workaround?

[enkelprifti98]

No workaround for FRR but the way I did it was to have MetalLB peer with BIRD on a different TCP port instead of TCP port 179, BIRD would add the learned routes to the host kernel route table. Then FRR running on the same host picks up the routes on the kernel route table and announces them upstream.

[klausenbusk]

The solution I came up with, is peering with the ToR switches on a different subnets, this way you avoid BGP's Connection Collision Detection.

So FRR peers with 10.4.0.1/28 (src 10.4.0.2) and Cilium peers with 10.4.0.17/28 (src 10.4.0.18). Cilium is basically doing the same thing as MetalLB.

[ton31337]

At the moment we don't allow peering with 0.0.0.0/127.0.0.0/240.0.0.0 as they are treated as invalid ranges. We are in progress relaxing this in the future with an additional knob, but for now, it's not possible normally.

Even more, connecting to self it's not allowed at all, unless you configure multiple instances binding for the specific address.

http://docs.frrouting.org/en/stable-8.2/bgp.html#cmdoption-bgpd-l

[klausenbusk]

At the moment we don't allow peering with 0.0.0.0/127.0.0.0/240.0.0.0 as they are treated as invalid ranges. We are in progress relaxing this in the future with an additional knob, but for now, it's not possible normally.

Is there a issue for this?

[klausenbusk]

I have opened a issue: #12767.

[azalio]

Оh.
I'm found out this after 3 hour of debugging bird && frr.
(Yes, I have metallb and bird on the same host)