struggling to transmit packets on time?

[spencersevilla]

Hi all,

I am struggling to get srsRAN up and running and communicating with the RU. I think it might be an issue related to my system (specifically timing?) but it's hard for me to tell or figure out what to do from here. I know system-related stuff is a little bit outside the purview of this specific project, but it seems to be where a lot of the O-RAN complexity lies, so I was hoping someone might be able to point me in the right direction or help me find some tools or ways I could confirm where the issue is and when I've got it fixed.

System Specs:
https://protectli.com/product/vp6650/
Debian 12 realtime kernel 6.1.0-21-rt-amd64 (but I've also tried with the stock kernel)
Intel® i5-1235U 10 Core / 12 Thread
32Gb ram

I have my system connected directly to our RU over a 10Gbps SFP+ fiber port. I am running ptp4l and phc2sys in grandmaster mode, so that the RU and system both sync themselves to my NIC's PHC. ptp4l looks happy, the RU reports that it has a good time sync lock, and phc2sys shows me numbers between -10 and 10. When I start up the gnb, I can hear my machine start working harder, and top shows me about 60-200% CPU utilization, but I still have about 10 idle cores.

The big symptom I see is that as soon as the gnb starts, I start seeing an incredible amount of "late downlink C/U-Plane messages in the transmitter queue" errors for literally every slot. As in, for example I'll see the error in a row for slot 978.1 symbol 0, 1, 2, ... 13 then slot 978.2 symbol 0, 1, 2, ... 13, and so on. Also, tcpdump/wireshark makes it look like none of these packets are being sent out over the interface at all, so I assume they're too late and are just being discarded - is that how srsRAN works here?

Aside from the srsperformance script (which I've already tried running) I'm wondering if there are other things I should try, or something obvious I've missed, or some way to sanity-check my machine? From everything I've read in the guides/forums I think this machine should be performant enough, but it's hard to tell from my vantage point. Thanks a ton in advance!!!

[herlesupreeth]

Hey @spencersevilla Intel® i5-1235U seems to be a low powered Intel processor for Notebook. I am not sure what is the BW and MIMO configuration you are running the RU with. It could be that the processor is not strong enough. However, you can try setting as mentioned in this link to get maximum performance - #425 (reply in thread)

[spencersevilla]

@herlesupreeth Thanks a ton for this context and thread pointer! I will apply all the optimizations you've listed. The hardware platform we're using is intended for field/network deployments, thought it should be strong enough but maybe you're right.

In the meantime I have one more question for you: in the interest of getting anything working at all, am I correct in assuming that reducing MIMO and BW will reduce the load on the processor? And are there other features I could change or turn off in srsRAN to reduce the load? Thanks!

[herlesupreeth]

in the interest of getting anything working at all, am I correct in assuming that reducing MIMO and BW will reduce the load on the processor?

yes, it should reduce the load. I think the processor you are using can easily handle 20Mhz BW and SISO configuration.

And are there other features I could change or turn off in srsRAN to reduce the load?

I think there are some configurations (nof. PUSCH threads etc.) which reduces the load but I haven't used them.

For reference, I have used AMD Ryzen 7 5825U processor to easily run 40Mhz BW and 2x2 MIMO. So, i believe your processor can do the same

[spencersevilla]

Sincerely thanks so much for this information! Super super encouraging and helpful.

One more simple question for you if you don't mind: I'm not 100% sure that our RU is operating correctly, and am trying to get a handle on what I should expect to be seeing over tcpdump. I can successfully get a ptp lock between the RU/DU and can see lots of PTP messages being exchanged. From there, as soon as I start srsRAN should I see it start sending lots of eCPRI packets to the RU, or does it expect to receive some packets from the RU first? I know the srsRAN guide says to start the RU before the DU, and the issue is I'm not quite sure if the RU is actually started or not, so I figured I'd figure out which side to pin down first. In the past I was able to successfully get both sides running before, but I don't know who started comms because my pcap file just shows oodles of IQ packets being sent in both directions.

Thanks again so much I sincerely appreciate it.

[herlesupreeth]

I dont have that much experience working with an RU as I majorly work with a SDR. I will let other in the community to jump in on this.
@ninjab3s Can you please comment on the above question when you have time?

[ninjab3s]

Hey @spencersevilla,

If I understood correctly you have set up an LLS-C1 [1] configuration (DU server is PTP GM and drives RU). We have made some experiments with this configuration and we figured out that usually the quartz of the NICs we have used were not good enough. Do you mind sharing the exact NIC model you are using?

Also, could you share more insights on your setup if you get a chance?

• What RU model are you using?
• Can you share your PTP GM config?
• Can you share some output of ptp4l on the RU?
• Can you check the KPIs of the RU to see if there are early/late packets?
• What SFP adapters and cables are you using?

Thanks!

[1] https://www.techplayon.com/o-ran-fronthaul-transport-synchronization-configurations/

[spencersevilla]

Thanks so much for this insight! Really appreciate you taking the time; a lot of these specs are new to me. A big part of the challenge here is that our RU manufacturer is not familiar with srsRAN so I'm trying to tease out whether issues exist on their side or with srsRAN or with the system setup/integration. And yes you're correct, we're doing an LLS-C1 configuration.

The strangest symptom I'm hoping I can get some help with is the following: In the past, I was able to get eCPRI packets running between the DU and the RU, and collected some good pcaps, but it looked like the RU was rejecting/ignoring them for some reason. I restarted the RU to get a fresh start and since then, I haven't been able to get the gnb to send any eCPRI packets at all. Instead, now when I run ./gnb I see two errors for each message slot/symbol: "Failed to dispatch new symbol task in the message transmitter" and "Late downlink C/U-Plane messages in the transmitter queue".

RU is a G06RRH-46-01B from MTI
NIC is a Intel X710-BM2 SFP+
PTP GM config is attached below as ptp.txt
srsRAN gnb config attached below as gnb.txt
Cable and SFP adapters came from the RU manufacturer. The RU doesn't really provide/expose a lot of KPIs and uses its own serial interface for communication. The equivalent output to ptp4l is copy pasted below. "Offset to Master" field tends to range between +20ns and -20ns:

PTP Status Information:  
Role:                              Slave 
Transport Protocol:                802.3 Ethernet Multicast 
Master Id:                         Foreign 646266.fffe.228673 
Master Clock Class:                248 
Clock Domain:                      24 
Grand Master Unicast IP Addr:      0.0.0.0 
Announcement Timeout:              3 
Master Sync Interval:              -4 
Control Interface:                 1 
VLAN             :                 none 
Clock Lock Status:                 Locked 
Offset To Master:                  -6 ns 

ptp.txt
gnb.txt

Thanks again I really appreciate your time looking at this!

[andrepuschmann]

Instead, now when I run ./gnb I see two errors for each message slot/symbol: "Failed to dispatch new symbol task in the message transmitter" and "Late downlink C/U-Plane messages in the transmitter queue".

The issue is fixed now. Please update main to latest commit, build and test again.

[spencersevilla]

@andrepuschmann that fixed these errors. Thank you!!! I might have different questions later on but the issue is resolved for now.