[Bug]: recoll's python module does not work out of the box

[Grimler91]

Problem description

In #12497 recoll's python module was added to the package. It installs to $PREFIX/lib/python3/dist-packages/recoll/, but this directory is not part of PYTHONPATH in termux per default, giving an error unless the PYTHONPATH is modified:

$ python3
Python 3.10.8 (main, Oct 12 2022, 12:40:52) [Clang 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0 on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import recoll
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'recoll'
>>> 
$ PYTHONPATH=$PREFIX/lib/python3/dist-packages/ python3
Python 3.10.8 (main, Oct 12 2022, 12:40:52) [Clang 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0 on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import recoll
>>> 

Either we should make our python package recognise the lib/python3/dist-packages folder (by setting PYTHONPATH in a profile script?), or we should make recoll install to $PREFIX/lib/python3.X/site-packages instead.

Some related discussion was had in the PR to add termux-gui-pm by @tareksander.

What steps will reproduce the bug?

$ pkg install recoll
$ python3
>>> import recoll

What is the expected behavior?

recoll should import without errors out of the box.

System information

termux-info:

Termux Variables:
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_MAIN_PACKAGE_FORMAT=debian
TERMUX_VERSION=0.118.0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://mirror.termux.dev/termux-main stable main
# x11-repo (sources.list.d/x11.list)
deb https://mirror.termux.dev/termux-x11 x11 main
# root-repo (sources.list.d/root.list)
deb https://mirror.termux.dev/termux-root root stable
Updatable packages:
flac/stable 1.4.2 aarch64 [upgradable from: 1.4.1]
glib-bin/stable 2.74.1 aarch64 [upgradable from: 2.74.0]
glib/stable 2.74.1 aarch64 [upgradable from: 2.74.0]
graphviz/stable 7.0.0 aarch64 [upgradable from: 6.0.1]
libcairo/stable 1.16.0-10 aarch64 [upgradable from: 1.16.0-9]
libexpat/stable 2.5.0 aarch64 [upgradable from: 2.4.9]
libffi/stable 3.4.4 aarch64 [upgradable from: 3.4.3]
libflac/stable 1.4.2 aarch64 [upgradable from: 1.4.1]
libidn2/stable 2.3.4 aarch64 [upgradable from: 2.3.3]
libxrender/stable 0.9.11 aarch64 [upgradable from: 0.9.10-14]
ragel/stable 7.0.4 aarch64 [upgradable from: 6.10-4]
rsync/stable 3.2.7 aarch64 [upgradable from: 3.2.6-1]
swig/stable 4.1.0 aarch64 [upgradable from: 4.0.2-2]
webkit2gtk/x11 2.38.1 aarch64 [upgradable from: 2.38.0-1]
termux-tools version:
1.31.0
Android version:
11
Kernel build information:
Linux localhost 4.4.111-ge6e6b41f98d8 #1 SMP PREEMPT Sat Aug 21 13:35:28 CEST 2021 aarch64 Android
Device manufacturer:
samsung
Device model:
SM-G955F

[Grimler91]

Ping @JCGoran, @landfillbaby maybe you have some suggestions on how to handle this

[JCGoran]

I think there is a slightly bigger issue here, which I've only realized afterwards (sorry about that, haven't made packages for termux before): namely, if you look at the manifest via apt-file list recoll, you'll see:

recoll: /data/data/com.termux/files/usr/lib/python3/dist-packages/recoll/_recoll.cpython-310-x86_64-linux-gnu.so

which, if I'm not mistaken, indicates that the Python lib is actually built for x86_64, and not aarch64, so the bindings probably don't function properly at the moment. I've naively looked at other packages which have Python bindings, and maybe something like this would work?

termux-packages/packages/unbound/build.sh

Lines 32 to 44 in e0232d9

	termux_step_pre_configure() {
	_PYTHON_VERSION=$(. $TERMUX_SCRIPTDIR/packages/python/build.sh; echo $_MAJOR_VERSION)
	termux_setup_python_crossenv
	pushd $TERMUX_PYTHON_CROSSENV_SRCDIR
	_CROSSENV_PREFIX=$TERMUX_PKG_BUILDDIR/python-crossenv-prefix
	python${_PYTHON_VERSION} -m crossenv \
	$TERMUX_PREFIX/bin/python${_PYTHON_VERSION} \
	${_CROSSENV_PREFIX}
	popd
	. ${_CROSSENV_PREFIX}/bin/activate
	export PYTHON_SITE_PKG=$TERMUX_PREFIX/lib/python${_PYTHON_VERSION}/site-packages
	}

[Grimler91]

Missed that, the name is wrong indeed, file seems to be correct architecture though:

$ file /data/data/com.termux/files/usr/lib/python3/dist-packages/recoll/_recoll.cpython-310-x86_64-linux-gnu.so
/data/data/com.termux/files/usr/lib/python3/dist-packages/recoll/_recoll.cpython-310-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped

using termux_setup_python_crossenv as above would probably fix both the path and the file name issue

[xtkoba]

Please make sure to always use termux_setup_python_crossenv when installing a Python module, even if it does not contain any C extensions.

[xtkoba]

The following is an explanation of commit 2c40a8f.

In this specific case it turns out that the culprit is in Makefile.in:

termux-packages/packages/recoll/Makefile.in.patch

Lines 7 to 8 in 2c40a8f

	-OPTSFORPYTHON = $(shell test -f /etc/debian_version && echo --install-layout=deb)
	+OPTSFORPYTHON =

Also a workaround is needed for android/ndk#201:

termux-packages/packages/recoll/python-recoll-setup.py.in.diff

Lines 7 to 12 in 2c40a8f

	-if "CYGWIN" in os.environ:
	- libraries = ['recoll', 'xapian', 'iconv', 'z']
	+if True:
	+ libraries = ['recoll', 'python@PYTHON_VERSION@', 'xslt', 'xml2', 'xapian', 'iconv', 'z']
	else:
	libraries = ['recoll']

(Commonly-used LDFLAGS+=" -lpython${_PYTHON_VERSION}" is insufficient.)

And finally it needs to disable Python 2:

termux-packages/packages/recoll/Makefile.in.patch

Lines 16 to 17 in 2c40a8f

	-@MAKEPYTHON_TRUE@ for v in 2 3;do test -n "`which python$${v}`" && \
	+@MAKEPYTHON_TRUE@ for v in 3;do test -n "`command -v python$${v}`" && \

termux-packages/packages/recoll/Makefile.in.patch

Lines 23 to 24 in 2c40a8f

	-@MAKEPYTHON_TRUE@ for v in 2 3;do test -n "`which python$${v}`" && \
	+@MAKEPYTHON_TRUE@ for v in 3;do test -n "`command -v python$${v}`" && \

[JCGoran]

With the current revision (1.33.1-2), the module can't seem to be found again:

$ python -c 'import recoll'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'recoll'

From what I can tell, the following issues remain:

1. the files are installed in $PREFIX/lib, which is not in the Python interpreter's path
2. the files conftree.py, __init__.py, qresultstore.py, rclconfig.py, rclextract.py, recoll.py should be included in the distribution
3. the lib file should be called _recoll[SOMETHING].so (since the .py files above use from ._recoll import [OBJECT])

Note that even if I do all the above manually (fix path, copy .py files there, rename library), there still seem to be issues:

$ PYTHONPATH=$PREFIX/lib/ python -c 'from recoll import recoll'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/recoll/recoll.py", line 22, in <module>
    from ._recoll import *
ImportError: dynamic module does not define module export function (PyInit__recoll)

[JCGoran]

Could this be reopened? The Python module still doesn't work OOTB.

[xtkoba]

Not reproducible for me. The command

python -c "from recoll import recoll"

exits normally.

[xtkoba]

Would you please share the strace output of the command?

IMO that should have a new issue, instead of reusing once closed one.

[xtkoba]

@JCGoran I believe that is another issue. Please open a new one. Thanks.