Multiprocessing

[nisach02]

When running GELATO on multiple objects with multiprocessing option, the fitting time per object increases quite a lot.
For example, if I run 1 object on a single CPU, it would take ~1 minute to fit the spectra. But if I run 4 objects with 4 CPUs on multiprocessing, it takes more than 4-5 minutes to fit each spectra.
This behaviour was found when running GELATO on a cluster.

[TheSkyentist]

Thanks for bringing this to my attention! Were the single-core and multi-core tests performed on the same machine? Are you able to reproduce this on a laptop/desktop?

…

On Sat, Aug 31, 2024 at 00:54 nisach02 ***@***.***> wrote: When running GELATO on multiple objects with multiprocessing option, the fitting time per object increases quite a lot. For example, if I run 1 object on a single CPU, it would take ~1 minute to fit the spectra. But if I run 4 objects with 4 CPUs on multiprocessing, it takes more than 4-5 minutes to fit each spectra. This behaviour was found when running GELATO on a cluster. — Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEB6FNEMOSY3VXPENFXT2X3ZUDZ4BAVCNFSM6AAAAABNNIFARKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ4TQMZUG42DIMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[nisach02]

Were the single-core and multi-core tests performed on the same machine? Are you able to reproduce this on a laptop/desktop?

They were performed on the same machine. I could not reproduce this issue in my personal computer though (MacbookPro i7 2020 version). i.e, no issues at all. The issues could be something to do with the EDR machines running the code in parallel.

[TheSkyentist]

Interesting. There are definitely more hurdles to getting multiprocessing with Python to work on cluster environments. Can you perform a more basic test with Python multiprocessing but not with GELATO but just with a basic example to see if you get a speed improvement or if the same issue persists?

…

On Mon, Sep 2, 2024 at 15:49 nisach02 ***@***.***> wrote: Were the single-core and multi-core tests performed on the same machine? Are you able to reproduce this on a laptop/desktop? They were performed on the same machine. I could not reproduce this issue in my personal computer though (MacbookPro i7 2020 version). i.e, no issues at all. The issues could be something to do with the EDR machines running the code in parallel. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEB6FNEGJY2WAMCFEEI6BSDZURUG7AVCNFSM6AAAAABNNIFARKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRUHAYDMMRSGA> . You are receiving this because you commented.Message ID: ***@***.***>