Degradation of IPaddr2/findif: more than 1 matching routes error for multiple addresses on the same IPv4 interface

[tverdoff]

Hi Team,

Same error as in #1980 but a different case

Recently after update from resource-agents-4.10.0-52.el9_4.4.x86_64 to resource-agents-4.10.0-64.el9.x86_64 in scope of Oracle Linux system update I have encountered the issue with autodetecting the interface name for VIP address.

ISSUE: vip-10.30.40.44 cannot be started due to more than one routes matched

ROOT:
On my system there are more than one IP address configured on the same network interface, thus network manager creates several routes automatically.

> sudo crm configure show vip-10.30.40.44
primitive vip-10.30.40.44 IPaddr2 \
        params ip=10.30.40.44 iflabel=web cidr_netmask=32 \
        op monitor interval=10s timeout=20s \
        op start interval=0 timeout=30s \
        op stop interval=0 timeout=30s \
        meta failure-timeout=120s migration-threshold=3

> ip a
...
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether bc:30:5b:ed:ea:41 brd ff:ff:ff:ff:ff:ff
    altname eno2
    altname enp1s0f1
    inet 10.30.40.48/24 brd 10.30.40.255 scope global noprefixroute em2 <<<<<<<<<<<<<<<<
       valid_lft forever preferred_lft forever
    inet 10.30.40.47/24 brd 10.30.40.255 scope global secondary noprefixroute em2 <<<<<<<<<<<<
       valid_lft forever preferred_lft forever
    inet6 fe80::be30:5bff:feed:ea41/64 scope link 
       valid_lft forever preferred_lft forever

> ip route 
default via 192.168.192.2 dev em1 proto static metric 100 
10.30.40.0/24 dev em2 proto kernel scope link src 10.30.40.48 metric 101 <<<<<<<<<<
10.30.40.0/24 dev em2 proto kernel scope link src 10.30.40.47 metric 101 <<<<<<<<<<
192.168.192.0/19 dev em1 proto kernel scope link src 192.168.196.177 metric 100

All worked fine before commit f717b4a
btw, to be honest it is not clear from commit message what exactly corner cases were the reason for this commit.

-    set -- $(ip -o -f $family route list match $match $scope | awk 'BEGIN{best=0} /\// { mask=$1; sub(".*/", "", mask); if( int(mask)>=best ) { best=int(mask); best_ln=$0; } } END{print best_ln}')
+    set -- $(ip -o -f $family route list match $match $scope | sed -e 's,^\([0-9.]\+\) ,\1/32 ,;s,^\([0-9a-f:]\+\) ,\1/128 ,' | sort -t/ -k2,2nr)

So before the commit only one route was returned:

> family=inet; match=10.30.40.44; netmask=32; scope='scope link'; ip -o -f $family route list match $match $scope | awk 'BEGIN{best=0} /\// { mask=$1; sub(".*/", "", mask); if( int(mask)>=best ) { best=int(mask); best_ln=$0; } } END{print best_ln}'
10.30.40.0/24 dev em2 proto kernel src 10.30.40.47 metric 101

After commit both routes are returned:

> family=inet; match=10.30.40.44; netmask=32; scope='scope link'; ip -o -f $family route list match $match $scope | sed -e 's,^\([0-9.]\+\) ,\1/32 ,;s,^\([0-9a-f:]\+\) ,\1/128 ,' | sort -t/ -k2,2nr
10.30.40.0/24 dev em2 proto kernel src 10.30.40.47 metric 101 
10.30.40.0/24 dev em2 proto kernel src 10.30.40.48 metric 101

++ findif
++ local match=10.30.40.44
++ local family
++ local proto
++ local scope
++ local nic=
++ local netmask=32
++ local brdcast=
++ local metric
++ local routematch
++ echo 10.30.40.44
++ grep -qs :
++ '[' 1 = 0 ']'
++ family=inet
++ scope='scope link'
++ findif_check_params inet
++ local family=inet
++ local match=10.30.40.44
++ local nic=
++ netmask=32
++ local brdcast=
++ local errmsg
++ maybe_convert_dotted_quad_to_cidr
++ case $netmask in
++ return
++ case $__OCF_ACTION in
++ true
++ '[' -n '' ']'
++ '[' inet = inet6 ']'
++ ipcheck_ipv4 10.30.40.44
++ local 'r1_to_255=([1-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])'
++ local 'r0_to_255=([0-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])'
++ local 'r_ipv4=^([1-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$'
++ echo 10.30.40.44
++ grep -q -Ee '^([1-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.([0-9][0-9]?|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$'
++ '[' 0 = 1 ']'
++ '[' -n 32 ']'
++ prefixcheck 32 32
++ local prefix=32
++ local prefix_length=2
++ local prefix_check=32
++ '[' 2 -gt 3 -o 2 -eq 0 ']'
++ echo 32
++ grep -qs '[^0-9]'
++ '[' 1 = 0 ']'
++ '[' 32 -lt 1 -o 32 -gt 32 ']'
++ return 0
++ '[' 0 = 1 ']'
++ '[' -n '' ']'
++ return 0
++ '[' -n 32 ']'
++ match=10.30.40.44/32
++ '[' -n '' ']'
+++ ip -o -f inet route list match 10.30.40.44/32 scope link
+++ grep -v '^\(unreachable\|prohibit\|blackhole\)'
+++ sed -e 's,^\([0-9.]\+\) ,\1/32 ,;s,^\([0-9a-f:]\+\) ,\1/128 ,'
+++ sort -t/ -k2,2nr
++ routematch='10.30.40.0/24 dev em2 proto kernel src 10.30.40.47 metric 101 
10.30.40.0/24 dev em2 proto kernel src 10.30.40.48 metric 101 '
++ '[' inet = inet6 ']'
+++ echo '10.30.40.0/24 dev em2 proto kernel src 10.30.40.47 metric 101 
10.30.40.0/24 dev em2 proto kernel src 10.30.40.48 metric 101 '
+++ wc -l
++ '[' 2 -gt 1 ']'
++ ocf_exit_reason 'More than 1 routes match 10.30.40.44/32. Unable to decide which route to use.'

Considering that those routes are created automatically during IP address adding, I find this use case as valid when there are more than one route for the same network. Could you please return support of the multiple addresses per interface?

Thanks for your assistance!