Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve combine_scorefiles #111

Closed
smlmbrt opened this issue Jun 15, 2023 Discussed in #110 · 0 comments · Fixed by #244
Closed

Improve combine_scorefiles #111

smlmbrt opened this issue Jun 15, 2023 Discussed in #110 · 0 comments · Fixed by #244
Labels
enhancement New feature or request

Comments

@smlmbrt
Copy link
Member

smlmbrt commented Jun 15, 2023

I agree, we should consider how to make this faster. Potential solutions (thinking out loud):

  • Not appending to a file, writing to an open file connection
  • Processing multiple files in parallel (multiple threads)

Also related to #64

Discussed in #110

Originally posted by mglev1n June 15, 2023
When using pgsc_calc for more than a few scores, the pipeline seems to take a very long time. Despite using the --parallel argument and having access to HPC resources, the step that combines scores into a single scoring file seems to run serially (reading score1, performing QC, appending to scoring file; reading score2, performing QC, appending, ...). Is there a more efficient approach for creating a combined scoring file (either using the pgsc_calc pipeline, or preparing the combined scoring file prior to running the pipeline)?

@smlmbrt smlmbrt added the enhancement New feature or request label Jun 15, 2023
@nebfield nebfield added this to the v2.1.0 milestone Jul 14, 2023
@nebfield nebfield mentioned this issue Mar 19, 2024
7 tasks
@nebfield nebfield linked a pull request Mar 19, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants