Skip to content

Commit

Permalink
return to parallel processing with new logging
Browse files Browse the repository at this point in the history
  • Loading branch information
mzappitello committed Oct 16, 2023
1 parent 599e343 commit 68d3d7e
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions python_src/src/lamp_py/ingestion/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,5 +93,7 @@ def ingest_files(files: List[str], metadata_queue: Queue) -> None:
# "spawn" some of the behavior described above only occurs when using
# "fork". On OSX (and Windows?) to force this behavior, run
# multiprocessing.set_start_method("fork") when starting the script.
for converter in converters.values():
converter.convert()
with Pool(processes=2) as pool:
pool.map_async(run_converter, converters.values())
pool.close()
pool.join()

0 comments on commit 68d3d7e

Please sign in to comment.