Expose number of jobs to organize

[CodyCBakerPhD]

Recent usage of the DANDI API in the NWB GUIDE revealed a sporadic joblib issue that results in segmentation faults - no clue what actually causes it for some users, but the best solution we have for now is to just expose job usage directly

Where should tests for this go?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e99ee62) 88.88% compared to head (579d108) 89.03%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1341      +/-   ##
==========================================
+ Coverage   88.88%   89.03%   +0.14%     
==========================================
  Files          76       76              
  Lines       10240    10243       +3     
==========================================
+ Hits         9102     9120      +18     
+ Misses       1138     1123      -15     

Flag	Coverage Δ
unittests	89.03% <100.00%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
dandi/cli/cmd_organize.py	100.00% <100.00%> (ø)
dandi/organize.py	85.29% <100.00%> (+1.71%)	⬆️
dandi/tests/test_organize.py	95.53% <100.00%> (+0.02%)	⬆️

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yarikoptic]

more info on segfault? I usually try to run those through valgrind to get a sense where/why might be happening although IIRC it was tricky with python stuff.

re tests - just make one of the invocations in dandi/tests/test_organize.py with jobs=1 to cover that codepath, not sure if we should sweat too much on really testing beyond smoke test on that aspect

[CodyCBakerPhD]

more info on segfault? I usually try to run those through valgrind to get a sense where/why might be happening although IIRC it was tricky with python stuff.

Sadly no. It was just something random that occurred in certain user environments during first round of alpha testing - the local traceback led back to joblib called by DANDI organize - we've been unable to reproduce it generally - so the best workaround we can think on is to disable joblib usage in those situations

[github-actions]

🚀 PR was released in 0.58.0 🚀