fix : restructure correction files, cleanup workflows for JERC (#20)

* fix : restructure correction files, cleanup workflows for JERC

-  restructure correction files

-  add compiled JEC files, code to produce compiled files

-  modify the runner.py to scale out jobs

-  update README.md

* fix:clean up unused files

* feat: make it a python package

* feat : update flexible campaigns

- add  `metadata/AK4_parameters.py` for different campaign, fix things in `runner.py` and `workflows`

- correct swapped CvL, CvB definitions in histograms, defined the value in the beginning

- condense the systematics into additional histogram axis

- simplified arrays filling

- use `dense_lookup` for `cTagSFReader.py`

* fix : update minor structure

- make `compile_jec.py` configurable

- make the info in `definitions.py` as dict, replace the workflow accordingly

- clean up plotting code

.gitignore

@@ -1,5 +1,10 @@
 *.coffea
+*.pyc
 .ipynb_checkpoints/
+
+src/BTVNanoCommissioning/version.py
+src/BTVNanoCommissioning.egg-info/
+
 workflows/__pycache__/
 dask-report.html
 dask-worker-space/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Andrzej Novak
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

MANIFEST.in

@@ -0,0 +1,7 @@
+prune *
+graft src
+graft tests
+
+include LICENSE README.md pyproject.toml setup.py setup.cfg
+recursive-include src/BTVNanoCommissioning/data/ *
+global-exclude __pycache__ *.py[cod] .*

README.md

@@ -4,6 +4,9 @@ Repository for Commissioning studies in the BTV POG based on (custom) nanoAOD sa
 
 ## Requirements
 ### Setup 
+
+:heavy_exclamation_mark: suggested to install under `bash` environment
+
 ```
 # only first time 
 git clone [email protected]:cms-btv-pog/BTVNanoCommissioning.git 
@@ -20,14 +23,19 @@ bash Miniconda3-latest-Linux-x86_64.sh
 ```
 NOTE: always make sure that conda, python, and pip point to local Miniconda installation (`which conda` etc.).
 
-You can either use the default environment`base` or create a new one:
+You can either use the default environment `base` or create a new one:
 ```
 # create new environment with python 3.7, e.g. environment of name `coffea`
 conda create --name coffea python3.7
 # activate environment `coffea`
 conda activate coffea
 ```
-Install coffea, xrootd, and more:
+You could simply create the environment through the existing `env.yml` under your conda environment
+```
+conda env create -f env.yml -p ${conda_dir}/envs/coffea
+```
+
+Or install manually for the required packages, coffea, xrootd, and more:
 ```
 pip install git+https://github.com/CoffeaTeam/coffea.git #latest published release with `pip install coffea`
 conda install -c conda-forge xrootd
@@ -38,6 +46,7 @@ conda install -c anaconda bokeh
 conda install -c conda-forge 'fsspec>=0.3.3'
 conda install dask
 ```
+
 ### Other installation options for coffea
 See https://coffeateam.github.io/coffea/installation.html
 ### Running jupyter remotely
@@ -93,13 +102,13 @@ python runner.py --workflow ${workflow} --json metadata/test.json
 - Dileptonic ttbar phase space : check performance for btag SFs, muon channel
 
 ```
-python runner.py --workflow ttdilep_sf --json metadata/94X_doublemu_PFNano.json
+python runner.py --workflow (e)ttdilep_sf --json metadata/94X_doublemu_PFNano.json
 ```
 
 - Semileptonic ttbar phase space : check performance for btag SFs, muon channel
 
 ```
-python runner.py --workflow ttsemilep_sf --json metadata/94X_singlemu_PFNano.json
+python runner.py --workflow (e)ttsemilep_sf --json metadata/94X_singlemu_PFNano.json
 ```
 
 </p>
@@ -112,26 +121,26 @@ python runner.py --workflow ttsemilep_sf --json metadata/94X_singlemu_PFNano.jso
 - Dileptonic ttbar phase space : check performance for charm SFs, bjets enriched SFs, muon channel
 
 ```
-python runner.py --workflow ctag_ttdilep_sf --json metadata/94X_doublemu_PFNano.json
+python runner.py --workflow (e)ctag_ttdilep_sf --json metadata/94X_doublemu_PFNano.json
 ```
 
 
 - Semileptonic ttbar phase space : check performance for charm SFs, bjets enriched SFs, muon channel
 
 ```
-python runner.py --workflow ctag_ttsemilep_sf --json metadata/94X_singlemu_PFNano.json
+python runner.py --workflow (e)ctag_ttsemilep_sf --json metadata/94X_singlemu_PFNano.json
 ```
 
 - W+c phase space : check performance for charm SFs, cjets enriched SFs, muon  channel
 
 ```
-python runner.py --workflow ctag_ttdilep_sf --json metadata/94X_singlemu_PFNano.json
+python runner.py --workflow (e)ctag_Wc_sf --json metadata/94X_singlemu_PFNano.json
 ```
 
 - DY phase space : check performance for charm SFs, light jets enriched SFs, muon channel
 
 ```
-python runner.py --workflow ctag_ttdilep_sf --json ctag_DY_mu_PFNano.json
+python runner.py --workflow (e)ctag_DY_sf --json ctag_DY_mu_PFNano.json
 ```
 
 </p>
@@ -204,12 +213,22 @@ Use the `fetch.py` in `filefetcher`, the `$input_DAS_list` is the info extract f
 python fetch.py --input ${input_DAS_list} --output ${output_json_name} --site ${site}
 ```
 
+## Create compiled corretions file(`pkl.gz`)
+
+Compile correction pickle files for a specific JEC campaign by changing the dict of jet_factory, and define the output file name by passing it as argument to the python script:
+
+```
+python -m utils.compile_jec data/JME/UL17_106X/jec_compiled.pkl.gz
+```
+
+
+
 ## Plotting code
 
 - data/MC comparison code
 
 ```python
-python plotdataMC.py --lumi ${lumi} --phase ctag_ttdilep_sf --output ctag_ttdilep_sf (--discr zmass --log True/False --data data_runD)
+python -m plotting.plotdataMC --lumi ${lumi} --phase ctag_ttdilep_sf --output ctag_ttdilep_sf (--discr zmass --log True/False --data data_runD)
 # lumi in /pb
 # phase = workflow 
 # output coffea file output = hist_$output$.coffea 
@@ -221,7 +240,7 @@ python plotdataMC.py --lumi ${lumi} --phase ctag_ttdilep_sf --output ctag_ttdile
 - data/data, MC/MC comparison
 
 ```python
-python comparison.py --phase ctag_ttdilep_sf --output ctag_ttdilep_sf -ref 2017_runB --compared 2017_runC 2017_runD (--discr zmass --log True/False --sepflav True/False)
+python -m plotting.comparison --phase ctag_ttdilep_sf --output ctag_ttdilep_sf -ref 2017_runB --compared 2017_runC 2017_runD (--discr zmass --log True/False --sepflav True/False)
 # phase = workflow 
 # output coffea file output = hist_$output$.coffea 
 # ref = reference data/MC sample