Extremely slow RNDIS throughput with esp-tinyusb (IEC-137)

[wuyuanyi135]

Answers checklist.

• I have read the component documentation ESP-IDF Components and the issue is not addressed there.
• I am using target and esp-idf version as defined in component's idf_component.yml
• I have searched the issue tracker for a similar issue and not found any related issue.

Which component are you using? If you choose Other, provide details in More Information.

device/esp_tinyusb

ESP-IDF version.

v5.2

Development Kit.

ESP32-S3-WROOM-1-N8R8

Used Component version.

1.4.4 (Dirty)

More Information.

In my project, a USB dongle made of ESP32-S3 are used as communication gateway that creates an RNDIS interface on the PC for connecting with the other devices.

Observations

Current setup

I have been using the tinyusb and additions from esp-iot-bridge and the throughtput was satisfactory (max ~7Mbps). Because of the USB refactoring work, the esp-iot-bridge solution does not compile on esp-idf > 5.2.

Code difference

The iot-bridge's [tinyusb_install_driver](https://github.com/espressif/esp-iot-bridge/blob/master/components/usb/usb_device/additions/src/tinyusb.c) uses tons of hal code but the esp-tinyusb counterpart uses new API that is compatible with esp-idf>5.2.

However, from my observation, with esp_tinyusb, the throughtput was only about 400 Kbps, which is much slower than what I could get from esp-iot-bridge. Looking at the implementation of ECM_RNDIS in tinyusb, it seems there isn't any significant change from the versions.

Benchmark

idf	esp-tinyusb	tinyusb	net	speed (Mbps)	comment
5.2	esp-iot-bridge	0.11.0	RNDIS	~7	baseline
5.2	esp-tinyusb	0.15.0~10	RNDIS	~0.4
5.2	esp-tinyusb	0.15.0~10	NCM	~0.4	unstable, hang after some time. (upstream bug)
5.2	esp-tinyusb	master	RNDIS	~0.4
5.2	esp-tinyusb	master	NCM	~0.4	new driver no longer hang but still very slow
5.3-rc1	esp-tinyusb	0.15.0~10	RNDIS	~0.4
5.3-rc1	esp-tinyusb	0.15.0~10	NCM	~0.4	unstable, hang after some time. (upstream bug)
5.3-rc1	esp-tinyusb	master	RNDIS	~0.4
5.3-rc1	esp-tinyusb	master	NCM	~0.4	new driver no longer hang but still very slow

Note:

1. to make RNDIS work, the fix has to be applied.
2. upstream NCM driver is being rewritten

Conclusions

From my analysis, I found that

1. Using the legacy NCM driver in current esp-tinyusb will lead to very chopy network (ping vary from <1ms to seconds or timeout) and will no longer respond after heavy traffic.
2. The legacy RNDIS driver in esp-tinyusb does not show unstable traffic but the throughtput is very slow compared to the one in esp-iot-solution
3. Different upstream versions of tinyusb does not affect RNDIS. In fact, the codebase of RNDIS did not change significantly.
4. Different idf.py version does not affect the performance, but newer version of idf prevents the esp-iot-solution to compile, thus no comparison can be made

I guess the slowdown was not caused by tinyusb. Most probably the way how tinyusb driver are installed (see Code difference) caused the issue. But these code involves too much implementation details of USB peripheral so I am not able to further track down.

Can you please compare the network throughtput when using this project and see if the speed confirms my observations?

[leeebo]

@wuyuanyi135 The main difference between tinyusb 0.11.0 and 0.15.0~* is the latest one switch the dcd layer from espressif/esp32sx/dcd_esp32sx.c to synopsys/dwc2/dcd_dwc2.c

Would you please try again with esp-tinyusb 0.15.0~10 with this line https://github.com/espressif/tinyusb/blob/release/v0.15/CMakeLists.txt#L49 change to src/portable/espressif/esp32sx/dcd_esp32sx.c

[wuyuanyi135]

@leeebo Hello, Thanks for your tips! I have tried your suggestion by commenting out "src/portable/synopsys/dwc2/dcd_dwc2.c" and add "src/portable/espressif/esp32sx/dcd_esp32sx.c" in CMakeLists.txt. It does not help improving the speed -- same performance @ ~0.4 Mbps

This is way slower than the implementation in esp-iot-bridge

[lijunru-hub]

I discovered that the new version of esp_tinyusb has low processing efficiency when handling tud_network_xmit in RNDIS. There's no effective notification mechanism to quickly send out data from the user layer. After replacing tinyusb_net.c in esp_tinyusb with the old version, the data transmission rate returned to normal.

[chegewara]

Hi,
i have not tested with previous versions, but on P4 i have this setup:

• esp-idf - master
• tinyusb - 0.15.0~10
• esp_tinyusb - 1.4.5
• CDC-NCM
• iperf-cmd with iperf on esp32-p4

results are satysfying, over 34mbit/s when esp32 is a server and 26mbit/s when its a client.
On esp32-s3-otg devkit it was over 5.5mbit/s when esp32 is a server

[wuyuanyi135]

Hi, i have not tested with previous versions, but on P4 i have this setup:

• esp-idf - master
• tinyusb - 0.15.0~10
• esp_tinyusb - 1.4.5
• CDC-NCM
• iperf-cmd with iperf on esp32-p4

results are satysfying, over 34mbit/s when esp32 is a server and 26mbit/s when its a client. On esp32-s3-otg devkit it was over 5.5mbit/s when esp32 is a server

IIRC P4 has high speed USB 2.0, while S3 only has full speed (12mbps) interface right?

[chegewara]

Thats correct

[emaayan]

hi, is still an issue in 5.3.1 ? is it because S3 has full speed USB?
what's the difference between esp_iot_bridge and esp_tinyusb?

[lijunru-hub]

The esp_tinyusb library has not yet resolved this issue. If you want to use the virtual network adapter functionality, you can refer to this example:

https://github.com/espressif/esp-iot-solution/tree/master/examples/usb/device/usb_dongle

[chegewara]

To be honest it is hard to say if library or esp32 performance is bad.
For example, NCM speed test with esp32-p4 i did can be find here
https://esp32.com/viewtopic.php?f=12&t=41620&sid=665db6a4ae28a5832f23d197b2edfaba&start=10#p140002

50-ish mbit/s seems to be very slow on USB HS, which is 480mbit/s, but it should be satisfactory for low price chip.
In other hand specialized USB to eth dongle connected over hub to slow it down from USB SS to USB HS have slightly better performance

The question is, are our expectation matching reality and can be achieved with freertos?
Can we have esp32 having performance closer to specialized devices?

[emaayan]

To be honest it is hard to say if library or esp32 performance is bad. For example, NCM speed test with esp32-p4 i did can be find here esp32.com/viewtopic.php?f=12&t=41620&sid=665db6a4ae28a5832f23d197b2edfaba&start=10#p140002

50-ish mbit/s seems to be very slow on USB HS, which is 480mbit/s, but it should be satisfactory for low price chip. In other hand specialized USB to eth dongle connected over hub to slow it down from USB SS to USB HS have slightly better performance

The question is, are our expectation matching reality and can be achieved with freertos? Can we have esp32 having performance closer to specialized devices?

i'm interested in this since i'm trying to implement a sniffer that will transmit to wireshark with tcp over usb, where wireshark would be the tcp client calling the sniffer, getting trafic over serial connection can quickly clog up the connection, so i'm trying to figure out something faster.

btw i'm also having issues to control the dhcp server behavior in the sample sta2eth where i'm trying to prevent it from issuing it a dns ip as i don't want it receiving any dns queries from the network (because performance)

`
esp_err_t wired_netif_init(void)
{
const tinyusb_config_t tusb_cfg = {
.external_phy = false,

};
ESP_ERROR_CHECK(tinyusb_driver_install(&tusb_cfg));


const tinyusb_net_config_t net_config = {
    // locally administrated address for the ncm device as it's going to be used internally
    // for configuration only
    .mac_addr = {0x00, 0x02, 0x11, 0x22, 0x33, 0x01},
    .on_recv_callback = netif_recv_callback,       
};

esp_err_t ret = tinyusb_net_init(TINYUSB_USBDEV_0, &net_config);
if (ret != ESP_OK) {
    ESP_LOGE(TAG, "Cannot initialize USB Net device");
    return ret;
}

// with OUI range MAC to create a virtual netif running http server
// this needs to be different to usb_interface_mac (==client)
uint8_t lwip_addr[6] =  {0x00, 0x02, 0x11, 0x22, 0x33, 0x02};

// Definition of
// 1) Derive the base config (very similar to IDF's default WiFi AP with DHCP server)
esp_netif_inherent_config_t base_cfg =  {
    .flags = //ESP_NETIF_DHCP_SERVER |
     ESP_NETIF_FLAG_AUTOUP, // Run DHCP server; set the netif "ip" immediately
    .ip_info = &_g_esp_netif_soft_ap_ip,                    // Use the same IP ranges as IDF's soft AP
    .if_key = "wired",                                      // Set mame, key, priority
    .if_desc = "usb ncm config device",       
    .route_prio = 10
    
};
// 2) Use static config for driver's config pointing only to static transmit and free functions
esp_netif_driver_ifconfig_t driver_cfg = {
    .handle = (void *)1,                // not using an instance, USB-NCM is a static singleton (must be != NULL)
    .transmit = netif_transmit,         // point to static Tx function
    .driver_free_rx_buffer = l2_free    // point to Free Rx buffer function
};

// 3) USB-NCM is an Ethernet netif from lwip perspective, we already have IO definitions for that:
struct esp_netif_netstack_config lwip_netif_config = {
    .lwip = {
        .init_fn = ethernetif_init,
        .input_fn = ethernetif_input
    }
};

// Config the esp-netif with:
//   1) inherent config (behavioural settings of an interface)
//   2) driver's config (connection to IO functions -- usb)
//   3) stack config (using lwip IO functions -- derive from eth)
esp_netif_config_t cfg = {
    .base = &base_cfg,
    .driver = &driver_cfg,
    .stack = &lwip_netif_config
};

s_netif = esp_netif_new(&cfg);
if (s_netif == NULL) {
    return ESP_FAIL;
}
esp_netif_set_mac(s_netif, lwip_addr);

/*
// set the minimum lease time
uint32_t  lease_opt = 1000;
esp_netif_dhcps_option(s_netif, ESP_NETIF_OP_SET, IP_ADDRESS_LEASE_TIME, &lease_opt, sizeof(lease_opt));


dhcps_offer_t dhcps_dns_value = OFFER_DNS;   
ESP_ERROR_CHECK(esp_netif_dhcps_option(s_netif, ESP_NETIF_OP_SET, ESP_NETIF_DOMAIN_NAME_SERVER, &dhcps_dns_value, sizeof(dhcps_dns_value)));    
esp_netif_dns_info_t dns={
	.ip.u_addr.ip4.addr=ESP_IP4TOADDR( 192, 168, 9, 2),	
	dns.ip.type = IPADDR_TYPE_V4
};    

ESP_ERROR_CHECK(esp_netif_set_dns_info(s_netif, ESP_NETIF_DNS_MAIN, &dns));

*/

    

// start the interface manually (as the driver has been started already)
esp_netif_action_start(s_netif, 0, 0, 0);
return ESP_OK;

}
`

[wuyuanyi135]

@emaayan Greetings. I also tried disabling DNS providing but without luck. I didn't try further since it doesn't really matter anyway. As long as the metrics of the USB RNDIS interface has lower priority than the internet interface it won't cause connectivity issue of the PC. I was looking for some configuration bytes to provide a large metrics number by the DHCP server but I couldn't find any. Why did you want to disable DNS on the DHCP?

[emaayan]

@emaayan Greetings. I also tried disabling DNS providing but without luck. I didn't try further since it doesn't really matter anyway. As long as the metrics of the USB RNDIS interface has lower priority than the internet interface it won't cause connectivity issue of the PC. I was looking for some configuration bytes to provide a large metrics number by the DHCP server but I couldn't find any. Why did you want to disable DNS on the DHCP?

I'm trying to reduce the amount of traffic this card has as much as possible , because I've noticed it was starting to get DNS queries from the machine, and I'm having performance issues when trying to push traffic from the card to the machine. and i'm not sure why that is. i've also noticed it's processing MDNS traffic so i'm trying to eliminate that as well.