Cannot reenumarate devices in same python process #114

Teslafly · 2021-03-11T21:45:12Z

It seems that once hidapi has been enumerated, it cannot detect / connect to new or replugged devices.

I have 2 examples.
First, using the hid.enumerate() example code, I have an mcp2221 device disconnected and run

>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

Then I connect the mcp2221 and it shows up in lsusb:

$ lsusb -d 04d8:00dd 
Bus 001 Device 107: ID 04d8:00dd Microchip Technology, Inc.

rerunning hid.enumerate() and the mcp2221 is not found:

>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

and I can't connect to the device either:

>>> #connect mcp2221 here
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "hid.pyx", line 113, in hid.device.open
    raise IOError('open failed')
OSError: open failed

but if I restart python I can see the device:

root@0b5c447f6028:/power-test/temp/cython-hidapi# python3
Python 3.7.10 (default, Feb 16 2021, 19:28:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hid
  keys = list(device_dict.keys())
    keys.sort()
    for key in keys:
        print("%s : %s" % (ke>>> 
y, device_dict[key]))
    print()>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 2
manufacturer_string : Power Engineering
path : b'0001:006b:02'
product_id : 221
product_string : Test Board
release_number : 256
serial_number : 01234567
usage : 0
usage_page : 0
vendor_id : 1240

interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

and I can connect to it:

>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering
>>> print("Product: %s" % h.get_product_string())
Product: Test Board
>>> print("Serial No: %s" % h.get_serial_number_string())
Serial No: 01234567

But if I unplug it and replug it, and close and reopen, I cannot reconnect to the same device:

...
>>> print("Serial No: %s" % h.get_serial_number_string())
Serial No: 01234567
>>> #replug here:
>>> h.close()
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "hid.pyx", line 113, in hid.device.open
    raise IOError('open failed')
OSError: open failed

It requires restarting python (or spawning a new python process) to get the device connected again.
Though if you start python, import hid but do not use any functions, then connect your device and run an hid function it will connect/find your device just fine. Just after the first hid function has been run your device list is locked in.

Is this a limitation of libusb hidapi itself? or something this library can fix by properly reinitialising the hidapi library? It seems like the list of usb devices may be being cached and not updated.

This is running in a docker container (but works the same on a native ubuntu system) and I have built the hidapi library from source because it seemed like #91 might fix it. (Though I have not verified this source build library is the actual one being used vs pypi)

The text was updated successfully, but these errors were encountered:

todbot · 2021-03-11T22:09:16Z

Another thing to check on Linux is the output of dmesg to see if the HID device in question is being grabbed by a device-specific driver and removing it from the HID subsystem so hidapi can't see it.

Teslafly · 2021-03-11T22:32:29Z

Disconnecting and reconnecting the mcp2221 to the system produces the below dmesg:

[944446.637428] usb 1-5.3.1: USB disconnect, device number 113
[944448.881642] usb 1-5.3.1: new full-speed USB device number 114 using xhci_hcd
[944448.977729] usb 1-5.3.1: New USB device found, idVendor=04d8, idProduct=00dd
[944448.977751] usb 1-5.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[944448.977762] usb 1-5.3.1: Product: Test Board
[944448.977774] usb 1-5.3.1: Manufacturer: Power Engineering
[944448.977785] usb 1-5.3.1: SerialNumber: 01234567
[944448.979421] cdc_acm 1-5.3.1:1.0: ttyACM0: USB ACM device
[944448.982831] hid-generic 0003:04D8:00DD.0057: hiddev0,hidraw2: USB HID v1.11 Device [Power Engineering Test Board] on usb-0000:00:14.0-5.3.1/input2

It does not seem to be getting grabbed by another driver.
If I only restart python

$Ctrl - D
$python3
>>> #hid stuff

I can connect to it fine.

todbot · 2021-03-11T22:38:04Z

Yup, agreed. I think you're right that the patch to add hid_ext() could help.

Teslafly · 2021-03-11T22:46:37Z

except I'm pretty sure I compiled the version with that patch, and it still wasn't working.

root@0b5c447f6028:/power-test/temp/cython-hidapi# pip3 show hidapi
Name: hidapi
Version: 0.10.1
Summary: A Cython interface to the hidapi from https://github.com/libusb/hidapi
Home-page: https://github.com/trezor/cython-hidapi
Author: Pavol Rusnak
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib/python3.7/site-packages/hidapi-0.10.1-py3.7-linux-x86_64.egg
Requires: setuptools
Required-by: power-mcp2221a

Has the version with the hid_ext() been pushed to pypi yet?

prusnak · 2021-03-11T22:54:25Z

except I'm pretty sure I compiled the version with that patch, and it still wasn't working.

You are not exiting the process, so hid_exit is not being called.

But anyways, enumerate should work just fine. You should not need to run hid_exit to be able to enumerate.

Has the version with the hid_ext() been pushed to pypi yet?

Not yet

Teslafly · 2021-03-11T23:19:25Z

So the thing I want to do is re enumerate a reconnected device without spawning a new python process. Ideally without disconnecting other devices as well (but can work around that).

I bumped the source version number and ensured that it had the hid_ext patch and that it was installed correctly.
I reran the tests, it still doesn't automatically detect new devices, but calling hid_exit does seem to reinitialize things.

>>> import hid
>>> 
>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

>>> #plug in mcp2221
>>> hid.hidapi_exit()
>>> import hid
>>> 
>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 2
manufacturer_string : Power Engineering
path : b'0001:0076:02'
product_id : 221
product_string : Test Board
release_number : 256
serial_number : 01234567
usage : 0
usage_page : 0
vendor_id : 1240

interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering

So it is possible to reconnect to devices now, but you have to nuke everything else connected via hidapi to do so.

Also, calling hidapi_exit() with a device still connected produces a segmentation fault, or at least a core dump

$ python3
Python 3.7.10 (default, Feb 16 2021, 19:28:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hid
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering
>>> 
>>> h.close()
>>> hid.hidapi_exit()

$ python3
Python 3.7.10 (default, Feb 16 2021, 19:28:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hid
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering
>>>
>>> # h.close()
>>> hid.hidapi_exit()
python3: ../../libusb/io.c:2116: handle_events: Assertion `ctx->pollfds_cnt >= internal_nfds' failed.
Aborted (core dumped)

# I have also gotten "Segmentation fault (core dumped)"

jonasmalacofilho · 2021-03-11T23:25:07Z

I cannot reproduce this with the hidraw backend, it could be a bug in hidapi-libusb.

Teslafly · 2021-03-11T23:47:56Z

yeah, I am using libusb.

jonasmalacofilho · 2021-03-12T00:40:02Z

Actually, I still couldn't reproduce the enumeration issue with the libusb backend (using cython-hidapi 0.10.1, hidapi 0.10.1 and libusb 1.0.24; linux 5.11.4).

Plugging the device in and out should while running the program bellow should have showed me the problem, right?

import hid
import time


def enumerate_hids():
    print("HID devices found:")

    for x in hid.enumerate():
        print(
            "{:04x}:{:04x}  {} {}  at: {}".format(
                x["vendor_id"],
                x["product_id"],
                x["manufacturer_string"] or "(None)",
                x["product_string"] or "(None)",
                x["path"],
            )
        )


if __name__ == "__main__":
    while True:
        enumerate_hids()
        print("---")
        time.sleep(1)

Version directly using hidapi from C:

/* repro.c */

#include <errno.h>
#include <stdio.h>
#include <unistd.h>

#include "hidapi/hidapi.h"

int enumerate() {
	struct hid_device_info *head = hid_enumerate(0, 0);
	struct hid_device_info *cur = head;

	if (!head)
		return -EIO;

	printf("HID devices found:\n");

	do {
		printf("- %04x:%04x  %ls %ls  at: %s\n", cur->vendor_id,
		       cur->product_id, cur->manufacturer_string,
		       cur->product_string, cur->path);
		cur = cur->next;

	} while (cur != NULL);

	hid_free_enumeration(head);

	return 0;
}

int main()
{
	int ret;

	while (1) {
		ret = enumerate();
		if (-ret)
			return -ret;
		printf("---\n");
		sleep(1);
	}

	return 0;
}

$ gcc -o repro-libusb repro.c -lhidapi-libusb -g
$ ./repro-libusb

Teslafly · 2021-03-12T01:53:16Z

yeah, that program looks like it should work.
... and it does, on the base os, but not in the docker container with the same setup. (removed devices never reappear until you restart the script/python, but not the container)
so looks like this might be related to using a docker container.

The a dockerfile that should work to reproduce the error:

FROM python:3.8-buster

RUN apt-get update \
    && apt-get -y install --no-install-recommends \
    libudev-dev \
    libusb-1.0-0-dev \
    udev

RUN pip3 install hidapi
CMD /bin/bash

# docker build -t test_usbhidapi:local -f Dockerfile .
# docker run --rm -it --privileged -v /dev:/dev test_usbhidapi:local

Given that this now appears to potentially be related to passing devices through docker, and there is a workaround. It's possible you want to close this issue unless you feel like debugging that mess.

If I find out more useful information I will likely post it here.

It is interesting that restarting Python or hidapi_exit() works to refresh devices though.

bearsh · 2021-03-12T07:37:48Z

@Teslafly regarding Docker, you shouldn't need a privileged container or full access to /dev to have it working, all you need is something like this:

echo 'c 189:* rwm' > /sys/fs/cgroup/devices/docker/CONTAINER_HASH/devices.allow
docker run --rm -it -v /dev/bus:/dev/bus:ro test_usbhidapi:local

this works for me (I access multiple hid devices, which get plugged/unplugged by the user multiple times over the lifetime of the application, from a container using libusb). for more info see Marc Merlin's post http://marc.merlins.org/perso/linux/post_2018-12-20_Accessing-USB-Devices-In-Docker-_ttyUSB0_-dev-bus-usb-_-for-fastboot_-adb_-without-using-privileged.html

amendelzon · 2021-05-27T23:59:37Z

yeah, that program looks like it should work.
... and it does, on the base os, but not in the docker container with the same setup. (removed devices never reappear until you restart the script/python, but not the container)
so looks like this might be related to using a docker container.

The a dockerfile that should work to reproduce the error:
FROM python:3.8-buster

RUN apt-get update \
    && apt-get -y install --no-install-recommends \
    libudev-dev \
    libusb-1.0-0-dev \
    udev

RUN pip3 install hidapi
CMD /bin/bash

# docker build -t test_usbhidapi:local -f Dockerfile .
# docker run --rm -it --privileged -v /dev:/dev test_usbhidapi:local
Given that this now appears to potentially be related to passing devices through docker, and there is a workaround. It's possible you want to close this issue unless you feel like debugging that mess.

If I find out more useful information I will likely post it here.

It is interesting that restarting Python or hidapi_exit() works to refresh devices though.

Hi @Teslafly , I'm having the same device reconnection issues only within a docker container. Did you manage to find a workaround? I haven't tried the hidapi_exit patch yet, but if possible I'd like to avoid it. Thanks, any help is greatly appreciated!

It was not supposed to be used explicitly in the first place (see comments on the PR this commit is a part of). The HIDAPI library will still be cleanly finalized before the program terminates, through another – automatic – mechanism. Additionally, the only currently known caller is using it as a workaround for another issue (trezor#114), and that use case would not be supported by the no-op compatibility shim anyway. Related: trezor#114 ("Cannot reenumarate devices in same python process") Related: trezor#128 ("Fatal Python error: Segmentation fault on 0.11.0.post2")

It was not supposed to be used explicitly in the first place (see comments on the PR this commit is a part of). The HIDAPI library will still be cleanly finalized before the program terminates, through another – automatic – mechanism. Additionally, the only currently known caller is using it as a workaround for another issue (#114), and that use case would not be supported by the no-op compatibility shim anyway. Related: #114 ("Cannot reenumarate devices in same python process") Related: #128 ("Fatal Python error: Segmentation fault on 0.11.0.post2")

mcuee · 2023-05-10T11:35:06Z

I guess this is no longer an issue with cython-hidapi but rather docker container. So this can be closed, right?

jonasmalacofilho mentioned this issue Nov 2, 2021

Prevent memory corruption when finalizing HIDAPI #130

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot reenumarate devices in same python process #114

Cannot reenumarate devices in same python process #114

Teslafly commented Mar 11, 2021

todbot commented Mar 11, 2021

Teslafly commented Mar 11, 2021 •

edited

Loading

todbot commented Mar 11, 2021

Teslafly commented Mar 11, 2021 •

edited

Loading

prusnak commented Mar 11, 2021

Teslafly commented Mar 11, 2021

jonasmalacofilho commented Mar 11, 2021

Teslafly commented Mar 11, 2021

jonasmalacofilho commented Mar 12, 2021

Teslafly commented Mar 12, 2021

bearsh commented Mar 12, 2021

amendelzon commented May 27, 2021

mcuee commented May 10, 2023

Cannot reenumarate devices in same python process #114

Cannot reenumarate devices in same python process #114

Comments

Teslafly commented Mar 11, 2021

todbot commented Mar 11, 2021

Teslafly commented Mar 11, 2021 • edited Loading

todbot commented Mar 11, 2021

Teslafly commented Mar 11, 2021 • edited Loading

prusnak commented Mar 11, 2021

Teslafly commented Mar 11, 2021

jonasmalacofilho commented Mar 11, 2021

Teslafly commented Mar 11, 2021

jonasmalacofilho commented Mar 12, 2021

Teslafly commented Mar 12, 2021

bearsh commented Mar 12, 2021

amendelzon commented May 27, 2021

mcuee commented May 10, 2023

Teslafly commented Mar 11, 2021 •

edited

Loading

Teslafly commented Mar 11, 2021 •

edited

Loading