Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preparing for veto file handling in PyGRB #4929

Merged
merged 14 commits into from
Nov 11, 2024

Conversation

pannarale
Copy link
Contributor

@pannarale pannarale commented Nov 7, 2024

This is the first PR of a series that will enable PyGRB to handle an xml veto file when producing final results (plots, p-values, tables, etc.)

Standard information about the request (and the following ones that will be linked to this)

This is a: a new feature in PyGRB. As an aside, some utilities and scripts will be streamlined.

This change affects: PyGRB

This change changes: result presentation / plotting and scientific output.

This change will probably break current functionalities at the moment. If the standard automated test running --help on all scripts fails on PyGRB plotting scripts, I will add some workarounds to avoid this. If needed, these will likely be empty functions: the plotting scripts will be progressively renovated in PRs following this one.

Motivation

PyGRB in its current version is not using data vetoes.

Contents

  • PyGRB used to handle veto definer files "collating" CAT1, CAT2, etc. xml veto definer files. Now this can go through the PyCBC +/-flag syntax and a single file. This is reflected in the post-processing utilities file, where a single veto vile is handled and a bunch of functions to combine multiple files were removed.
  • While upgrading the post-processing utilities file, I cleaned up other minor things, included more logging messages, inserted a few additional checks, and tried increasing robustness in general.
  • At workflow level the switch to a single veto file is reflected in grb_utils.py and the veto file is handled in parallel with the segments files throughout the functions (as noted in Combine PyGRB segments files #4887 the segments files will need to become a single json file eventually).
  • I did not work on the json file yet, but I did remove some hackiness in file handling from jobsetup.py.

Testing performed

The totality of the changes that will be broken down in multiple PRs was tested on GRB 170817A data by producing a full results webpage (see here).

  • The author of this pull request confirms they will adhere to the code of conduct

@pannarale pannarale added work in progress PyGRB PyGRB development labels Nov 7, 2024
@pannarale pannarale self-assigned this Nov 7, 2024
@pannarale
Copy link
Contributor Author

pannarale commented Nov 7, 2024

@spxiwh, @titodalcanton I added both of you as reviewers since there will be a combination of workflow and "pure PyGRB" content in this chain of PRs.
I will now watch the --help tests breaking and add the fixes to avoid them, but I will not push the full set of changes I made all at once because it is too much for one PR.
As for the previous PRs, black will come in at the very end.

Copy link
Contributor

@spxiwh spxiwh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only touches PyGRB code, so if it does what you want, then no issues from me. I leave a couple of optional comments. Should also wait to hear approval from Tito.

bin/pygrb/pycbc_make_offline_grb_workflow Show resolved Hide resolved
pycbc/results/pygrb_postprocessing_utils.py Outdated Show resolved Hide resolved
Copy link
Contributor

@titodalcanton titodalcanton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good apart from several minor comments and questions.

bin/pygrb/pycbc_make_offline_grb_workflow Show resolved Hide resolved
bin/pygrb/pycbc_make_offline_grb_workflow Show resolved Hide resolved
pycbc/results/pygrb_postprocessing_utils.py Outdated Show resolved Hide resolved
pycbc/results/pygrb_postprocessing_utils.py Outdated Show resolved Hide resolved
pycbc/results/pygrb_postprocessing_utils.py Outdated Show resolved Hide resolved
pycbc/results/pygrb_postprocessing_utils.py Outdated Show resolved Hide resolved
vetoes.coalesce()
for ifo in ifos:
for v in vetoes[ifo]:
v_span = v[1] - v[0]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can just do abs(v) here, though I am not sure it is more readable :)

pycbc/results/pygrb_postprocessing_utils.py Outdated Show resolved Hide resolved
pycbc/results/pygrb_postprocessing_utils.py Outdated Show resolved Hide resolved
Comment on lines +677 to +682
coinc_snr = numpy.array([])
if 'network/coherent_snr' in trigs_or_injs.keys() and \
'network/null_snr' in trigs_or_injs.keys():
coh_snr_sq = numpy.square(trigs_or_injs['network/coherent_snr'][:])
null_snr_sq = numpy.square(trigs_or_injs['network/null_snr'][:])
coinc_snr = numpy.sqrt(coh_snr_sq + null_snr_sq)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily a request for change as this is probably coupled with downstream code, but would it make more sense to return either None or an array of NaNs, rather than an empty array?

@pannarale pannarale merged commit 1ab428e into gwastro:master Nov 11, 2024
29 checks passed
@pannarale pannarale deleted the pygrb_vetoes branch November 11, 2024 19:56
prayush pushed a commit to prayush/pycbc that referenced this pull request Nov 21, 2024
* Missing empty line

* grb_utils.py preparation to handle veto-files and segment files

* pycbc_make_offline_grb_workflow update based on grb_utils.py upgrade

* Updated hacky pygrb stuff in jobsetup.py

* First batch of updates to handle the VDF in PyGRB and streamline pygrb_postprocessing_utils.py

* Removing build_veto_filelist from pycbc_pygrb_minifollowups

* Better variable naming and string search in jobsetup.py

* Removing redundant variables

* Corrected call to segments plots

* Better argparse syntax

* No f-string in logging

* assert instead of if
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PyGRB PyGRB development
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants