Problem integrating E2 interface with external nearRT-RIC

[EspiMCF]

Hello,

I am facing issues while I am integrating the srsRAN solution with a nearRT-RIC running on a docker compose.

My current setup is the srsRAN ( commit e73b461 ) on a baremetal server, with Intel E810 and a Broadcom Quad 25Gb Ethernet. The solution is able to connect to the AMF correctly, a full Core Simulator and also to a O-RU FHI simulator. The nearRT-RIC I am trying to integrate is a comercial solution. But when I am adding the E2 interface setup configuration section to the same yaml file I was using previously, it seems it can not reach the nearRT-RIC somehow. But from the nearRT-RIC I can ping the srsRAN server E2 interface and from the server I can ping the nearRT-RIC E2 address. The core is connected to the Intel card, and the RU and nearRT-RIC to the Broadcom card. As far as I know, my nearRT-RIC is running on docker and it is exposing the port 30000:

dric-service-e2node                        NodePort    <none>        5000:30000/SCTP 

And this IP:

dric_e2_intf Link encap:Ethernet 
          inet addr:10.10.20.162

When I am executing the config file without E2 section, I am able to see that is attached to the AMF and the gNB keep working and reporting logs. And I can even attach UEs and make some testing.
But when I add the E2 configuration described here:

e2:
  enable_du_e2: true
  e2sm_kpm_enabled: true
  e2sm_rc_enabled: true
  addr: 10.10.20.162
  bind_addr: 10.10.20.171          
  port: 30000

It reports the following message:

--== srsRAN gNB (commit e73b46182) ==--
2024-11-19T10:05:18.146254 [GNB     ] [I] Built in Release mode using commit e73b46182 on branch main
Connecting to NearRT-RIC on 10.10.20.162:30000
Failed to connect SCTP socket to 10.10.20.162:30000. error="Connection refused" timeout=0ms
2024-11-19T10:05:19.447524 [SCTP-GW ] [E] Failed to connect SCTP socket to 10.10.20.162:30000. error="Connection refused" timeout=0ms
srsRAN ERROR: Failed to create SCTP gateway.

It seems it is not able to configure the interface somehow. Also, what I am expecting in the nearRT-RIC is some message from the gNB in order to reject in a first try but with the values in order to admit it in the database, something like this:

 ERROR         E2T_E2AP     E2T_DATA_01 fd 8 E2node Global Id Not Provisioned: {
   gNB {
      global_gNB_ID {
         plmn_id = 0xXXXXXX
         gnb_id {
            gnb-ID = { 32, 0xXX 0xXX 0xXX 0xXX }
         }
      }
   }
}

ERROR         E2T_E2AP     E2T_DATA_01 Provisioned E2node missing in e2_htab table
INFO         E2T_E2AP     E2T_DATA_01 E2node fd 8 sent invalid e2setup

But even this message does not appear in the nearRT-RIC. Also something I am missing is the bind_interface variable for this interface, as it is not the same as the AMF or RU FHI.

Here I attach my configuration file I am currently using and as I said, I am running on commit e73b461.
gnb-2x2-50-fdd-nearRT.txt

Thanks and regards.

[pgawlowicz]

could you share logs and e2ap pcap file?

[EspiMCF]

Hi, do you need any specific level/layers of log?
Here I have attached a zip file where you can find the pcap of the E2 interface from the srsRAN solution.
DUinterfaceE2.zip

[s21sm]

@EspiMCF is the port number correct? I am not sure, but for my case, if Near-RT RIC either in the same machine or different machine it works for following setup

e2:
  enable_du_e2: true                # Enable DU E2 agent (one for each DU instance)
  e2sm_kpm_enabled: true    # Enable KPM service module
  #addr: 10.224.32.125                        #  When Near-RT RIC is in different machine
  #bind_addr: 10.228.132.238            # When Near-RT RIC is in different machine
  addr: 10.0.2.10                        # When Near-RT RIC is in the same machine
  bind_addr: 10.0.2.1                 # When Near-RT RIC is in the same machine
  port: 36421                             # RIC port
  e2sm_rc_enabled: true

[EspiMCF]

@s21sm As far as I know, the port that the container of the Near-RT RIC is exposing is 30000, I have tried also the default value port 36421 but it is the same behavior.

[pgawlowicz]

Can you ping the srsgnb host machine from the dric-service-e2node contanier?

Is this interface (and IP) part of the docker container or the server running docker?

dric_e2_intf Link encap:Ethernet 
          inet addr:10.10.20.162

[EspiMCF]

Hello,

Yes, the dric_e2_intf is the interface of the NearRT-RIC and running that IP address.

I have executed 3 sniffing tests I attach here:
PingAndRunningTests.zip

1. A ping from the DU to the NearRT-RIC server that replies and the ICMP packets seems to be ok
2. A ping from the Near-RT RIC to the DU that replies properly as you can see with the tcpdump here, but the problem is that the reply is not reaching the NearRT-RIC:

listening on eno8403np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:55:01.128111 IP 10.10.20.162 > DUserver: ICMP echo request, id 239, seq 22, length 64
12:55:01.128117 IP DUserver > 10.10.20.162: ICMP echo reply, id 239, seq 22, length 64
12:55:01.141564 IP DUserver.41692 > dns.google.domain: 10813+ [1au] PTR? 171.20.10.10.in-addr.arpa. (54)
12:55:01.154700 IP dns.google.domain > DUserver.41692: 10813 NXDomain 0/0/1 (54)
12:55:01.154731 IP DUserver.41692 > dns.google.domain: 10813+ PTR? 171.20.10.10.in-addr.arpa. (43)
12:55:01.168209 IP dns.google.domain > DUserver.41692: 10813 NXDomain 0/0/0 (43)
12:55:01.168460 IP DUserver.41062 > dns.google.domain: 44312+ [1au] PTR? 162.20.10.10.in-addr.arpa. (54)
12:55:01.182188 IP dns.google.domain > DUserver.41062: 44312 NXDomain 0/0/1 (54)
12:55:01.182215 IP DUserver.41062 > dns.google.domain: 44312+ PTR? 162.20.10.10.in-addr.arpa. (43)
12:55:01.195918 IP dns.google.domain > DUserver.41062: 44312 NXDomain 0/0/0 (43)
12:55:01.853324 STP 802.1w, Rapid STP, Flags [Proposal, Learn, Forward, Agreement], bridge-id 80ce.c4:5a:b1:81:06:05.8090, length 36
12:55:01.853339 STP 802.1w, Rapid STP, Flags [Proposal, Learn, Forward, Agreement], bridge-id 80ce.c4:5a:b1:81:06:05.8090, length 36
12:55:02.152125 IP 10.10.20.162 > DUserver: ICMP echo request, id 239, seq 23, length 64
12:55:02.152130 IP DUserver > 10.10.20.162: ICMP echo reply, id 239, seq 23, length 64

3. Started again the srsRAN gnb configuration and INIT messages are able to execute without any ABORT now but still not connecting and reporting this logs:

--== srsRAN gNB (commit e73b46182) ==--

2024-11-22T12:59:46.268459 [GNB     ] [I] Built in Release mode using commit e73b46182 on branch main
Connecting to NearRT-RIC on 10.10.20.162:30000
2024-11-22T12:59:47.592544 [E2AP    ] [I] Connecting to E2 Agent RIC (10.10.20.162)..
Failed to connect SCTP socket to 10.10.20.162:30000. error="Connection timed out" timeout=1364ms
2024-11-22T12:59:48.957338 [SCTP-GW ] [E] Failed to connect SCTP socket to 10.10.20.162:30000. error="Connection timed out" timeout=1364ms
srsRAN ERROR: Failed to create SCTP gateway.

Thanks and regards.

[pgawlowicz]

could you show the routing table of Near-RT RIC ?

[EspiMCF]

$ ip route show
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link

$ ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope: Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          

eth0      Link encap:Ethernet  HWaddr fa:35:7e:51:6e:fa
          inet addr:100.106.143.159  Bcast:0.0.0.0  Mask:255.255.255.255
          inet6 addr: fe80::f835:7eff:fe51:6efa/64 Scope: Link
          UP BROADCAST RUNNING MULTICAST  MTU:1480  Metric:1

dric_e2_intf Link encap:Ethernet  HWaddr 00:50:56:ae:1f:c2  Driver iavf
          inet addr:10.10.20.162  Bcast:10.10.20.191  Mask:255.255.255.224
          inet6 addr: fe80::250:56ff:feae:1fc2/64 Scope: Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1

[pgawlowicz]

and also the routing table of the DU server?

[EspiMCF]

$ ip route show
default via 10.10.1.1 dev enp81s0f3 proto static
10.10.1.0/24 dev enp81s0f3 proto kernel scope link src 10.10.1.215
10.10.2.0/24 dev eno8303np0.1002 proto kernel scope link src 10.10.2.50
10.10.20.160/27 dev eno8403np1 proto kernel scope link src 10.10.20.171

[EspiMCF]

Update: I have already ping between both nodes without any lost packet (problem was related with an internal firewall in the docker compose environment denying the echo reply).
Problem now, it is that I have checked with nmap -sY -p 30000 10.10.20.162 and that port (30000) for SCTP connections are closed and the port 5000 is open:

$nmap -sY -p 5000 10.10.20.162
Starting Nmap 7.80 ( https://nmap.org ) at 2024-11-25 11:28 UTC
Nmap scan report for 10.10.20.162
Host is up (0.000037s latency).
PORT      STATE SERVICE
5000/sctp open  unknown
Nmap done: 1 IP address (1 host up) scanned in 3.74 seconds

So, what I need now is to force the E2 service from the srsRAN to attack the port 5000 because when I am changing to that port and run the configuration file, it says:

--== srsRAN gNB (commit e73b46182) ==--
--port: Value 5000 not in range [20000 - 40000]
Run with --help for more information.

I know that currently is not supported and maybe is easier to change the port in the NearRT-RIC but for the moment, I am not able to do that, so any suggest?

Thanks and regards.

[pgawlowicz]

you might try to translate the port using iptables.

[EspiMCF]

I have tried that and finally yes!! I got the request and I have added the DUServer in the admission control table of the NearRT-RIC, but still I am not able to get connected the srsRAN because I would expect a reconnection but the DU seems that just try to connect once and this is the logs I am getting now:

Running on 0000:51:00.0 = enp81s0f0

--== srsRAN gNB (commit e73b46182) ==--

2024-11-28T12:27:23.101131 [GNB     ] [I] Built in Release mode using commit e73b46182 on branch main
Connecting to NearRT-RIC on 10.10.20.162:36421
2024-11-28T12:27:24.404954 [E2AP    ] [I] Connecting to E2 Agent RIC (10.10.20.162)..
2024-11-28T12:27:24.405222 [E2AP    ] [I] E2 connection established
Initializing the Open Fronthaul Interface for sector#0: ul_compr=[BFP,9], dl_compr=[BFP,9], prach_compr=[BFP,9], prach_cp_enabled=true, downlink_broadcast=false
Cell pci=3, bw=50 MHz, 2T2R, dl_arfcn=536020 (n7), dl_freq=2680.1 MHz, dl_ssb_arfcn=531610, ul_freq=2560.1 MHz

N2: Connection to AMF on 10.10.2.10:38412 completed
==== gNB started ===
Type <h> to view help
2024-11-28T12:27:24.467388 [E2      ] [I] Generate RAN function definition for OID: 1.3.6.1.4.1.53148.1.2.2.2
2024-11-28T12:27:24.467501 [E2      ] [I] Generate RAN function definition for OID: 1.3.6.1.4.1.53148.1.1.2.3
2024-11-28T12:27:24.467540 [E2      ] [I] Added RAN function OID 1.3.6.1.4.1.53148.1.2.2.2 to candidate list under RAN Function ID 2
2024-11-28T12:27:24.467542 [E2      ] [I] Added RAN function OID 1.3.6.1.4.1.53148.1.1.2.3 to candidate list under RAN Function ID 3
2024-11-28T12:27:24.467858 [E2-ASN1-PCK] [I] Received PDU of 18 bytes
2024-11-28T12:27:24.467897 [E2      ] [I] Handling E2 PDU of type unsuccessfulOutcome
2024-11-28T12:27:24.467898 [E2      ] [I] E2AP msg, "unsuccessfulOutcome.E2setupFailure", transaction id=0
2024-11-28T12:27:24.467901 [E2      ] [E] E2 Setup procedure failed.
2024-11-28T12:27:24.467909 [E2      ] [I] Transmitting E2 Setup Failure message

And in the part of the NearRT-RIC I am able to see the gnb data requesting access but even I have already added that info, I am getting this:

Nov 28 14:27:23 847717    ERROR         E2T_E2AP     E2T_DATA_01 Provisioned E2node missing in e2_htab table
Nov 28 14:27:23 847731     INFO         E2T_E2AP     E2T_DATA_01 E2node fd 10 sent invalid e2setup
Nov 28 14:27:23 858472     INFO           E2T_IO     E2T_DATA_01 new conn fdtype 2 ip b3140a0affff0000 0000000000000000 port 61925 fd 11
Nov 28 14:27:23 858478     INFO           E2T_IO     E2T_DATA_01 dric/dric_data.c:902 E2node IP table not provisioned
Nov 28 14:27:23 858487     INFO        E2T_E2NIO     E2T_DATA_01 yippee! new e2node connection family 2 ip 0xb3140a0affff0000 0x0000000000000000 port 61925 fd 11
Nov 28 14:27:23 858491     INFO         CLIENT_1     E2T_DATA_01 SCTP status for fd 11 assoc_id 49 state 0x4 local rwnd 16777216 unackdata 0 penddata 0 instrms 2 outstrms 2 frag_point 1432
Nov 28 14:27:23 858494     INFO         CLIENT_1     E2T_DATA_01 SCTP assoc for fd 11 assoc_id 0 maxrxt 10 peer dst 1 peer_rwnd 16777216 local_rwnd 212992 cookie_life 60000
Nov 28 14:27:23 921073    ERROR         E2T_E2AP     E2T_DATA_01 fd 11 E2node Global Id Not Provisioned: {
   gNB {
      global_gNB_ID {
         plmn_id = 0x12f410
         gnb_id {
            gnb-ID = { 32, 0x00 0x00 0x00 0x03 }
         }
      }
      gNB_DU_ID = 0
   }
}