Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel checks and debugging for empty filament cases #88

Merged
merged 4 commits into from
Sep 30, 2024

Conversation

e-koch
Copy link
Owner

@e-koch e-koch commented Sep 30, 2024

  • Additional debugging for longest path failures
  • Check threading behaviour with 2D parallelization

Occasional edge cases continue to crop up in FIlament2D:

  File "/project//FilFinder/fil_finder/filament.py", line 274, in skeleton_analysis
    longest_path(edge_list, nodes,
  File "/project//FilFinder/fil_finder/length.py", line 446, in longest_path
    max_path_length = max(values)
                      ^^^^^^^^^^^
ValueError: max() arg is an empty sequence
  • added debug=True mode for FilFinder2D.analyze_skeletons that enables additional printing to help identify these problem cases
  • Always impose a minimum >1 pixel distance minimum. Otherwise the definition of a "filament" doesn't really make sense.
  • Add a more clear error message for these cases.

@e-koch
Copy link
Owner Author

e-koch commented Sep 30, 2024

Changed the parallelization to adopt a single pool for processing, making it easier to change arguments for the pool and switch to using MPI:

e.g., examples added to the tutorial notebook:

from concurrent.futures import ProcessPoolExecutor

mypool = ProcessPoolExecutor(max_workers=2)
fil = FilFinder2D(hdu, save_name="FilFinder_Output", pool=mypool)

# is equivalent to

fil = FilFinder2D(hdu, save_name="FilFinder_Output", pool=None, nthreads=2)

# Using mpi4py:
# from mpi4py.futures import MPIPoolExecutor

# mypool = MPIPoolExecutor(max_workers=None)
# fil = FilFinder2D(hdu, save_name="FilFinder_Output", pool=mypool)

@e-koch
Copy link
Owner Author

e-koch commented Sep 30, 2024

Merging here. Tests and CI updates remain out-of-date and wil be fixed elsewhere.

@e-koch e-koch merged commit a5d2598 into master Sep 30, 2024
2 of 20 checks passed
@e-koch e-koch deleted the parallel_checks branch September 30, 2024 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant