-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pyprind with joblib #21
Comments
Hi, So, I could think of 2 possible scenarios here:
which would already work I guess.
Is this what you have in mind? I think in theory this should be easily possible; all the processes would have to do is to call the |
The second option is what I was looking for but doesnt seem to work with your suggestion of letting all processes update pbar
|
Hm, I think the problem is that the standard output is blocked during the computation which is why the pogressbar appears after everything has finished. I think this is something to investigate further after the "double progressbar" support had been added (see #18) In any case, another problem is that multiprocessing created copies of the objects that are send to the different processors (in contrast to threading). So basically, there are 4 progressbars then that are running from 0% to 25% each if you use 4 processors. Honest question: What's the advantage of joblib over multiprocessing? I saw it in certain libraries (e.g., scikit-learn) but never really understood why joblib instead of multiprocessing. E.g.,
vs.
|
well i am kinda new to the python ecosystem and I recently came across joblib. I noticed sklearn is using it, so kinda assumed it must be solving some issues that multiprocessing might have. Honestly didnt evaluate the 2 yet. |
I think there could be a way around that ... but it'll require some tweaks. Btw. if you use the "threading" backend, it should give you the 100% correctly but the problem is still how to print to stdout while the processing are still running...
Unfortunately, there are too many things on my to do list, currently. But I will leave this issue open, maybe someone has a good idea how to implement it, or maybe there will be a boring weekend for me some day ... ;) |
Is it possible to use the callback from joblib parallel to make it work with pyprind for Parallel processing tasks ?
The text was updated successfully, but these errors were encountered: