Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eti homologs --num_procs with more than 1 results in a run-time error #171

Open
khiron opened this issue Dec 15, 2024 · 0 comments
Open

Comments

@khiron
Copy link

khiron commented Dec 15, 2024

Running eti homologs --installed mammals_install_112 --outpath homology100 --ref human --limit 100 --force_overwrite --verbose works as expected

Running the same command with --num_procs 2 results in 2 python3 tasks being scheduled and allocated to processors as expected

image

However the Extracting 🧬 display is not updated during extraction and upon apparent completion of extraction the following run-time error occurs trying to serialize the sqllite connection

eti homologs --installed mammals_install_112 --outpath homology100 --ref human --num_procs 2 --limit 100 --force_overwrite --verbose
Loaded genome for 'human'
Found 63,140 gene IDs for 'human'
Found 100 homolog groups
Homolog search ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00
Extracting  🧬 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:-- 0:04:07
loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/loky/process_executor.py", line 370, in _sendback_result
    result_queue.put(
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/loky/backend/queues.py", line 230, in put
    obj = dumps(obj, reducers=self._reducers)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/loky/backend/reduction.py", line 215, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/loky/backend/reduction.py", line 208, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1295, in dump
    return super().dump(obj)
           ^^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/cogent3/core/annotation_db.py", line 560, in __getstate__
    return {"data": self._db.serialize(), "source": self.source}
                    ^^^^^^^^^^^^^^^^^^
AttributeError: 'sqlite3.Connection' object has no attribute 'serialize'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/richard/source/ensembl/.venv/bin/eti", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/ensembl_tui/cli.py", line 578, in homologs
    for seqs in get_seqs.as_completed(
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/cogent3/util/progress_display.py", line 112, in series
    for i, item in enumerate(items):
                   ^^^^^^^^^^^^^^^^
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/cogent3/util/parallel.py", line 228, in as_completed
    yield from _as_completed_mproc(f, s, max_workers)
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/cogent3/util/parallel.py", line 216, in _as_completed_mproc
    yield result.result()
          ^^^^^^^^^^^^^^^
  File "/home/richard/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
AttributeError: 'sqlite3.Connection' object has no attribute 'serialize'
Exception ignored in: <function Hdf5Mixin.__del__ at 0x7cf39bf3f600>
Traceback (most recent call last):
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/ensembl_tui/_storage_mixin.py", line 182, in __del__
  File "/home/richard/source/ensembl/.venv/lib/python3.12/site-packages/ensembl_tui/_storage_mixin.py", line 188, in close
NameError: name 'open' is not defined
@khiron khiron changed the title --num_procs with more than 1 results in a run-time error eti homologs --num_procs with more than 1 results in a run-time error Dec 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant