Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multi config analysis #229

Conversation

sahiljhawar
Copy link
Member

@sahiljhawar sahiljhawar commented Sep 10, 2023

The aim of this PR is to allow multiple model/configuration analysis with the use config/yaml file, follows from #214, #215 and b4fd564. However there are several pitfalls when using MPI:

  • With or without mpiexec the injection.log for the first configuration is populated by all the logs. And no log files are created for subsequent analysis.
  • With mpiexec, while individual configurations are running parallely but the multiple configurations are executed sequentially. To solve this, this PR is created.
  • In an attempt to solve the above issue, a --parallel flag is introduced which upon using distributes the given number of processess (specified using mpiexec -np) amongst all the configurations and try to execute them in parallel. However this does not seem to work. Here are two differnent cases now.
    • Firstly, if comm.Barrier() and MPI.Finalize() are used then the program gets stuck with file not found errors (for the 2nd configuration)
    • If these are not used then the analysis of first configuration is completed and the execution terminates.
  • Even when --parallel is not used (but mpiexec is), the beaviour is somewhat erratic; analysis of first configuration is completed and gets stuck while executing the later.

For now I would recommended not to use yaml file for multiple configuration analysis, since it might break somewhere in between.

Use the given commands to duplicate the issues:

  • mpiexec -np 10 light_curve_analysis --config config.yaml
  • mpiexec -np 10 light_curve_analysis --config config.yaml --parallel
  • uncommenting comm.Barrier() and MPI.Finalize() and again mpiexec -np 10 light_curve_analysis --config config.yaml --parallel

I would like everyone who knows MPI (possibly much more than me) to have a look on this.
Also, DO NOT MERGE :')

@sahiljhawar sahiljhawar added enhancement New feature or request help wanted Extra attention is needed labels Sep 10, 2023
@mcoughlin
Copy link
Member

@tsunhopang Any chance you can help @sahiljhawar ?

@sahiljhawar
Copy link
Member Author

Continued in #230

@sahiljhawar sahiljhawar deleted the multi_model_analysis branch January 25, 2024 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants