fasta2speclib hangs when multiprocessing is enabled

[RalfG]

Hi Ralf,

I tried the new features today and they seem to work. However, when I set more than one cpu in the config file, the process seems to get stuck in the multiprocessing step at the very beginning. For the tests I used a VM-Server with win10 and python 3.7 (I can not change the version there due to access restrictions). Even with only one cpu the process seems to stop at a step "generated new fontManager".
When I run the very same config file from my desktop computer (python 3.10) multiprocessing works as expected but I have to reduce the chunk-size.

On the computer where the process does not finish, I also have to use a virtual environment in order to call fasta2speclib while on my desktop I installed ms2pip globally and could run both commands directly.

Best,
Juergen

Originally posted by @JB91451 in #188 (comment)

[RalfG]

Hi @JB91451,

Thanks for testing and reporting back! The generated new fontManager message could indicate an issue with matplotlib. Is the Windows server a headless environment with only the CLI active and now graphical environment?
Could you provide a list of all package versions used in the environment (e.g. with pip freeze or conda list? A full log might also help me in figuring out the source of the issue.

Thanks!
Ralf

[JB91451]

Hi Ralf,

No, the Windows server has a gui and I connect via the remote-desktop tool.

The output from pip freeze is as follows:

E:\JB>E:\JB\ms2pip\venv\Scripts\activate.bat
Parameterformat falsch - 437.                                   # this is an error message related to the usage of _venv_ in German Windows versions but should not cause any issues according to https://bugs.python.org/issue35148

(venv) E:\JB>pip freeze
absl-py==1.4.0
astunparse==1.6.3
biopython==1.81
cachetools==5.3.0
certifi==2022.12.7
charset-normalizer==3.1.0
click==8.1.3
colorama==0.4.6
cycler==0.11.0
deeplc==1.2.1
deeplcretrainer==0.1.13
flatbuffers==23.3.3
fonttools==4.38.0
future==0.18.3
gast==0.4.0
google-auth==2.16.2
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
greenlet==2.0.2
grpcio==1.51.3
h5py==3.8.0
idna==3.4
importlib-metadata==6.0.0
joblib==1.2.0
keras==2.11.0
kiwisolver==1.4.4
libclang==15.0.6.1
llvmlite==0.39.1
lxml==4.9.2
Markdown==3.4.1
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.5.3
mdurl==0.1.2
ms2pip==3.11.0
numba==0.56.4
numexpr==2.8.4
numpy==1.21.6
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==23.0
pandas==1.3.5
Pillow==9.4.0
progressbar2==4.2.0
protobuf==3.19.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydantic==1.10.6
pygam==0.8.0
Pygments==2.14.0
pyparsing==3.0.9
pyteomics==4.5.6
python-dateutil==2.8.2
python-utils==3.5.2
pytz==2022.7.1
requests==2.28.2
requests-oauthlib==1.3.1
rich==13.3.2
rsa==4.9
scikit-learn==1.0.2
scipy==1.7.3
six==1.16.0
spectrum-utils==0.3.5
SQLAlchemy==1.4.46
tables==3.7.0
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-intel==2.11.0
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.2.0
threadpoolctl==3.1.0
tomlkit==0.11.6
tqdm==4.65.0
typing_extensions==4.5.0
urllib3==1.26.14
Werkzeug==2.2.3
wrapt==1.15.0
xgboost==1.6.2
zipp==3.15.0

(venv) E:\JB>python -v
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# C:\Program Files\Python3\lib\encodings\__pycache__\__init__.cpython-37.pyc matches C:\Program Files\Python3\lib\encodings\__init__.py
# code object from 'C:\\Program Files\\Python3\\lib\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# C:\Program Files\Python3\lib\__pycache__\codecs.cpython-37.pyc matches C:\Program Files\Python3\lib\codecs.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1B52E8>
# C:\Program Files\Python3\lib\encodings\__pycache__\aliases.cpython-37.pyc matches C:\Program Files\Python3\lib\encodings\aliases.py
# code object from 'C:\\Program Files\\Python3\\lib\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1CAE10>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1986A0>
# C:\Program Files\Python3\lib\encodings\__pycache__\utf_8.cpython-37.pyc matches C:\Program Files\Python3\lib\encodings\utf_8.py
# code object from 'C:\\Program Files\\Python3\\lib\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1DAEB8>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# C:\Program Files\Python3\lib\encodings\__pycache__\latin_1.cpython-37.pyc matches C:\Program Files\Python3\lib\encodings\latin_1.py
# code object from 'C:\\Program Files\\Python3\\lib\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1DE320>
# C:\Program Files\Python3\lib\__pycache__\io.cpython-37.pyc matches C:\Program Files\Python3\lib\io.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\io.cpython-37.pyc'
# C:\Program Files\Python3\lib\__pycache__\abc.cpython-37.pyc matches C:\Program Files\Python3\lib\abc.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1DE8D0>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1DE518>
# C:\Program Files\Python3\lib\__pycache__\site.cpython-37.pyc matches C:\Program Files\Python3\lib\site.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\site.cpython-37.pyc'
# C:\Program Files\Python3\lib\__pycache__\os.cpython-37.pyc matches C:\Program Files\Python3\lib\os.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\os.cpython-37.pyc'
# C:\Program Files\Python3\lib\__pycache__\stat.cpython-37.pyc matches C:\Program Files\Python3\lib\stat.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\stat.cpython-37.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B597B38>
# C:\Program Files\Python3\lib\__pycache__\ntpath.cpython-37.pyc matches C:\Program Files\Python3\lib\ntpath.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\ntpath.cpython-37.pyc'
# C:\Program Files\Python3\lib\__pycache__\genericpath.cpython-37.pyc matches C:\Program Files\Python3\lib\genericpath.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\genericpath.cpython-37.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B5A59E8>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B5A1208>
# C:\Program Files\Python3\lib\__pycache__\_collections_abc.cpython-37.pyc matches C:\Program Files\Python3\lib\_collections_abc.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\_collections_abc.cpython-37.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B5A5FD0>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1EE748>
# C:\Program Files\Python3\lib\__pycache__\_sitebuiltins.cpython-37.pyc matches C:\Program Files\Python3\lib\_sitebuiltins.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\_sitebuiltins.cpython-37.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1EEB38>
# C:\Program Files\Python3\lib\__pycache__\_bootlocale.cpython-37.pyc matches C:\Program Files\Python3\lib\_bootlocale.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\_bootlocale.cpython-37.pyc'
import '_locale' # <class '_frozen_importlib.BuiltinImporter'>
import '_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B592E80>
# C:\Program Files\Python3\lib\encodings\__pycache__\cp1252.cpython-37.pyc matches C:\Program Files\Python3\lib\encodings\cp1252.py
# code object from 'C:\\Program Files\\Python3\\lib\\encodings\\__pycache__\\cp1252.cpython-37.pyc'
import 'encodings.cp1252' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B6EE160>
# E:\JB\ms2pip\venv\lib\site-packages\_distutils_hack\__pycache__\__init__.cpython-37.pyc matches E:\JB\ms2pip\venv\lib\site-packages\_distutils_hack\__init__.py
# code object from 'E:\\JB\\ms2pip\\venv\\lib\\site-packages\\_distutils_hack\\__pycache__\\__init__.cpython-37.pyc'
import '_distutils_hack' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B6F5358>
# C:\Program Files\Python3\lib\__pycache__\types.cpython-37.pyc matches C:\Program Files\Python3\lib\types.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\types.cpython-37.pyc'
import 'types' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B6FCBE0>
# C:\Program Files\Python3\lib\importlib\__pycache__\__init__.cpython-37.pyc matches C:\Program Files\Python3\lib\importlib\__init__.py
# code object from 'C:\\Program Files\\Python3\\lib\\importlib\\__pycache__\\__init__.cpython-37.pyc'
# C:\Program Files\Python3\lib\__pycache__\warnings.cpython-37.pyc matches C:\Program Files\Python3\lib\warnings.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\warnings.cpython-37.pyc'
import 'warnings' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B701C50>
import 'importlib' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B701860>
# C:\Program Files\Python3\lib\importlib\__pycache__\util.cpython-37.pyc matches C:\Program Files\Python3\lib\importlib\util.py
# code object from 'C:\\Program Files\\Python3\\lib\\importlib\\__pycache__\\util.cpython-37.pyc'
# C:\Program Files\Python3\lib\importlib\__pycache__\abc.cpython-37.pyc matches C:\Program Files\Python3\lib\importlib\abc.py
# code object from 'C:\\Program Files\\Python3\\lib\\importlib\\__pycache__\\abc.cpython-37.pyc'
# C:\Program Files\Python3\lib\importlib\__pycache__\machinery.cpython-37.pyc matches C:\Program Files\Python3\lib\importlib\machinery.py
# code object from 'C:\\Program Files\\Python3\\lib\\importlib\\__pycache__\\machinery.cpython-37.pyc'
import 'importlib.machinery' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B715828>
import 'importlib.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B70BF98>
# C:\Program Files\Python3\lib\__pycache__\contextlib.cpython-37.pyc matches C:\Program Files\Python3\lib\contextlib.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\contextlib.cpython-37.pyc'
# C:\Program Files\Python3\lib\collections\__pycache__\__init__.cpython-37.pyc matches C:\Program Files\Python3\lib\collections\__init__.py
# code object from 'C:\\Program Files\\Python3\\lib\\collections\\__pycache__\\__init__.cpython-37.pyc'
# C:\Program Files\Python3\lib\__pycache__\operator.cpython-37.pyc matches C:\Program Files\Python3\lib\operator.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\operator.cpython-37.pyc'
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B747DA0>
# C:\Program Files\Python3\lib\__pycache__\keyword.cpython-37.pyc matches C:\Program Files\Python3\lib\keyword.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\keyword.cpython-37.pyc'
import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B751F60>
# C:\Program Files\Python3\lib\__pycache__\heapq.cpython-37.pyc matches C:\Program Files\Python3\lib\heapq.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\heapq.cpython-37.pyc'
import '_heapq' # <class '_frozen_importlib.BuiltinImporter'>
import 'heapq' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B758978>
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
# C:\Program Files\Python3\lib\__pycache__\reprlib.cpython-37.pyc matches C:\Program Files\Python3\lib\reprlib.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\reprlib.cpython-37.pyc'
import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B758F60>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B72AFD0>
# C:\Program Files\Python3\lib\__pycache__\functools.cpython-37.pyc matches C:\Program Files\Python3\lib\functools.py
# code object from 'C:\\Program Files\\Python3\\lib\\__pycache__\\functools.cpython-37.pyc'
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B72E400>
import 'contextlib' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B715400>
import 'importlib.util' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B70B080>
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\google
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\mpl_toolkits
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\google
# destroy google
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\pyteomics
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\google
# destroy google
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\mpl_toolkits
# destroy mpl_toolkits
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\google
# destroy google
# possible namespace for E:\JB\ms2pip\venv\lib\site-packages\pyteomics
# destroy pyteomics
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x000002016B1E43C8>
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import 'atexit' # <class '_frozen_importlib.BuiltinImporter'>
exit()
...

(venv) E:\JB>systeminfo

Hostname:                                      REMOVED
Betriebssystemname:                            Microsoft Windows 10 Enterprise N 2016 LTSB
Betriebssystemversion:                         10.0.14393 Nicht zutreffend Build 14393
Betriebssystemhersteller:                      Microsoft Corporation
Betriebssystemkonfiguration:                   Eigenständige Arbeitsstation
Betriebssystem-Buildtyp:                       Multiprocessor Free
Registrierter Benutzer:                        Windows-Benutzer
Registrierte Organisation:
Produkt-ID:                                    REMOVED
Ursprüngliches Installationsdatum:             3/7/2019, 11:16:02 AM
Systemstartzeit:                               3/21/2023, 12:44:36 PM
Systemhersteller:                              QEMU
Systemmodell:                                  Standard PC (Q35 + ICH9, 2009)
Systemtyp:                                     x64-based PC
Prozessor(en):                                 1 Prozessor(en) installiert.
                                               [01]: Intel64 Family 15 Model 6 Stepping 1 GenuineIntel ~2000 MHz
BIOS-Version:                                  REMOVED
Windows-Verzeichnis:                           C:\Windows
System-Verzeichnis:                            C:\Windows\system32
Startgerät:                                    \Device\HarddiskVolume2
Systemgebietsschema:                           en-us;Englisch (Vereinigte Staaten)
Eingabegebietsschema:                          en-us;Englisch (Vereinigte Staaten)
Zeitzone:                                      (UTC+01:00) Belgrad, Bratislava (Pressburg), Budapest, Ljubljana, Prag
Gesamter physischer Speicher:                  127,994 MB
Verfügbarer physischer Speicher:               123,445 MB
Virtueller Arbeitsspeicher: Maximale Größe:    146,426 MB
Virtueller Arbeitsspeicher: Verfügbar:         141,934 MB
Virtueller Arbeitsspeicher: Zurzeit verwendet: 4,492 MB
Auslagerungsdateipfad(e):                      E:\pagefile.sys
Domäne:                                        REMOVED
Anmeldeserver:                                 REMOVED
Hotfix(es):                                    29 Hotfix(e) installiert.
                                               [01]: KB4033631
                                               [02]: KB4054590
                                               [03]: KB4485447
                                               [04]: KB4498947
                                               [05]: KB4503537
                                               [06]: KB4509091
                                               [07]: KB4512574
                                               [08]: KB4520724
                                               [09]: KB4521858
                                               [10]: KB4524244
                                               [11]: KB4535680
                                               [12]: KB4540723
                                               [13]: KB4550994
                                               [14]: KB4562561
                                               [15]: KB4565912
                                               [16]: KB4576750
                                               [17]: KB4589210
                                               [18]: KB4601392
                                               [19]: KB5001078
                                               [20]: KB5001402
                                               [21]: KB5005698
                                               [22]: KB5011570
                                               [23]: KB5012170
                                               [24]: KB5014026
                                               [25]: KB5016058
                                               [26]: KB5017095
                                               [27]: KB5017396
                                               [28]: KB5023788
                                               [29]: KB5023697
Netzwerkkarte(n):                              1 Netzwerkadapter installiert.
                                               [01]: Intel(R) PRO/1000 MT Network Connection
                                                     Verbindungsname: Ethernet
                                                     DHCP aktiviert:  Ja
                                                     DHCP-Server:     REMOVED
                                                     IP-Adresse(n)
                                                     [01]: REMOVED
Anforderungen für Hyper-V:                     Es wurde ein Hypervisor erkannt. Features, die für Hyper-V erforderlich sind, werden nicht angezeigt.

(venv) E:\JB>cd E:\JB\ms2pip\semiTry

(venv) E:\JB\ms2pip\semiTry>E:\JB\ms2pip\venv\Scripts\fasta2speclib.exe -o E:\JB\ms2pip\semiTry\ms2pipTest -c E:\JB\ms2pip\semiTry\MS2PIP_Test_cpuCount10.json E:\JB\ms2pip\semiTry\Test.fasta
2023-03-21 13:21:25 INFO     Starting library generation pipeline...
                    INFO     Preparing search space...
Digesting proteins... ----------------------------------------   0% -:--:--

At this step the prediction stops without any notice. However, I waited several hours and the computer seems to be idle during the complete time as there is no cpu, I/O or RAM activity from any of the python processes that were started. When I repeat the process with with an altered config file with only one cpu, the digestion and modification assignment works, but ms2pip does not proceed after retention time prediction (or spectrum prediction if DeepLC is disabled):

(venv) E:\JB\ms2pip\semiTry>E:\JB\ms2pip\venv\Scripts\fasta2speclib.exe -o E:\JB\ms2pip\semiTry\ms2pipTest -c E:\JB\ms2pip\semiTry\MS2PIP_Test_cpuCount10.json E:\JB\ms2pip\semiTry\Test.fasta
2023-03-21 15:04:09 INFO     Starting library generation pipeline...
                    INFO     Preparing search space...
2023-03-21 15:09:38 INFO     Search space contains 2722016 peptides.
2023-03-21 15:09:39 INFO     Processing batch 1/273...
                    INFO     Generating MS²PIP input...
2023-03-21 15:09:45 INFO     Chunk contains 128924 peptidoforms.
2023-03-21 15:09:49 INFO     Filtered batch on precursor m/z: 128924 -> 128340
                    INFO     Predicting retention times with DeepLC...

Please note that the prediction for this first chunk seems to work as the counter that displays ETA, etc. runs down and disapears before the computer turns idle.

Best,
Juergen

[RalfG]

Hi Juergen,

There has been some additional refactoring for library generation in MS²PIP v4.0.0. The same functionality can now be called with ms2pip predict-library, with a slightly modified JSON config file .

If you can share the FASTA, I can also try to debug the issue myself.

Thanks!
Ralf

[JB91451]

Hi Ralf,

Thanks for coming back to that issue. We have deleted that server (including my test files) meanwhile and use a local koina-server with the ms2pip models for those predictions. So, unfortunately, I can not test this anymore.

However, the issue was there with any .fasta file I tried.

Best regards,
Juergen