Adding DeepMET SONIC Producer to CMSSW #37963

wpmccormack · 2022-05-16T15:38:23Z

PR description:

This PR introduces a producer for DeepMET using SONIC for co-processor-enabled inferences-as-a-service. For more information on the status of SONIC, please refer to this presentation to JetMET DPG: https://indico.cern.ch/event/1143469/contributions/4799423/attachments/2416540/4135308/March_28_JetMET_SONIC.pdf. Please note that the producer introduced here does not affect the main DeepMET producer in any way. This is an alternate way to run the standard DeepMET model using inference as a service. As noted in the linked slides, this producer has been validated locally on Tier 2 resources and at scale with cloud computing, but with this PR, we hope to perform more "official" production tests. But again, this producer does not any workflow unless explicitly called. A helper-function macro is slightly modified, as are a few config files. There will be an accompanying PR into https://github.com/cms-data/RecoMET-METPUSubtraction, which adds needed model files.

Tagging @kpedro88 @yongbinfeng @violatingcp @nhanvtran @jmduarte

PR validation:

This PR was tested on LPC using a Triton server running on the GPU-enabled AILab machines at FermiLab. Tests have also been performed using the DeepMET SONIC producer at Purdue T2 and in Google Cloud. When the producer is not called, there is no performance impact, and when it is called, the MET results are the same as for the generic DeepMET producer.

wpmccormack · 2022-05-16T15:43:52Z

The accompanying model-file PR is here: cms-data/RecoMET-METPUSubtraction#6

cmsbuild · 2022-05-16T15:47:19Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37963/30035

This PR adds an extra 36KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py modified in PR(s): Changes to enable new DeepTauId trainig #37892

cmsbuild · 2022-05-16T15:47:38Z

A new Pull Request was created by @wpmccormack (Patrick McCormack) for master.

It involves the following packages:

Configuration/ProcessModifiers (operations)
PhysicsTools/PatAlgos (reconstruction)
RecoMET/METPUSubtraction (reconstruction)

@perrotta, @clacaputo, @cmsbuild, @slava77, @jpata, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @hatakeyamak, @emilbols, @Martin-Grunewald, @mbluj, @ahinzmann, @demuller, @seemasharmafnal, @mmarionncern, @missirol, @makortel, @jdolen, @azotz, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @fabiocos, @AlexDeMoor, @JyothsnaKomaragiri, @gpetruc, @mariadalfonso, @andrzejnovak this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

kpedro88 · 2022-05-16T15:53:26Z

test parameters:
pull_request = cms-data/RecoMET-METPUSubtraction#6

kpedro88 · 2022-05-16T15:53:35Z

please test

cmsbuild · 2022-05-16T22:15:27Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6f1ee4/24748/summary.html
COMMIT: 30a4ccd
CMSSW: CMSSW_12_5_X_2022-05-16-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37963/24748/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@guitargeek Clean unused dependencies from BuildFiles #37915

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6f1ee4/24748/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6f1ee4/24748/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-6f1ee4/11634.301_TTbar_14TeV+2021_Run3FS+TTbar_14TeV_TuneCP5_GenSim+HARVESTNano

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3741432
DQMHistoTests: Total failures: 103
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3741306
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

jpata · 2022-05-17T07:02:51Z

would it make sense to define a single test workflow where all the sonic modifiers for production models are enabled?

enableSonicTriton,deepMETSonicTriton,particleNetSonicTriton,...

similar to what was done here: https://github.com/cms-sw/cmssw/pull/37134/files#diff-8c163203cbaa64f0e0e6aa2b16346cfab3eaf84a57a008c7fa37193e8b9b2adbR516

jpata · 2022-05-17T07:23:40Z

@michaelwassmer @cms-sw/jetmet-pog-l2 could you please review this from the JME side?

michaelwassmer · 2022-05-17T07:27:47Z

Yes, I can do that.
EDIT: From my perspective everything seems fine. The regular MET workflow does not seem to be affected by this. It has to be actively used to switch from the regular producer to the sonic producer.

kpedro88 · 2022-05-17T14:07:20Z

@jpata this workflow is already defined as .9001:

cmssw/Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

Line 1904 in 5bc18d0

    
           stepDict[stepName][k] = merge([{'--procModifiers': 'allSonicTriton'}, stepDict[step][k]])

hence the modification of https://github.com/cms-sw/cmssw/blob/master/Configuration/ProcessModifiers/python/allSonicTriton_cff.py in this PR.

jpata · 2022-05-18T12:22:37Z

test parameters:

pull_request = Adding DeepMET files in structure needed for SONIC cms-data/RecoMET-METPUSubtraction#6
workflow = 11824.9001,23434.9001
relvals_opt= -w standard,highstats,pileup,generator,extendedgen,production,upgrade,ged

jpata · 2022-05-18T12:23:38Z

@cmsbuild please test

cmsbuild · 2022-07-13T16:45:43Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37963/31030

This PR adds an extra 72KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/nano_cff.py modified in PR(s): Refactoring gen weight storage in EDM + Nano integration #32167
- File PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py modified in PR(s): Adding HZZ electron MVA ID #38355, Don't recluster jets in HI miniAOD for Run 3 #38711

cmsbuild · 2022-07-13T16:46:09Z

Pull request #37963 was updated. @perrotta, @rappoccio, @gouskos, @clacaputo, @fabiocos, @cmsbuild, @fgolf, @jpata, @mariadalfonso, @qliphy, @davidlange6 can you please check and sign again.

kpedro88 · 2022-07-13T16:53:44Z

please test

cmsbuild · 2022-07-13T23:32:46Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6f1ee4/26212/summary.html
COMMIT: 8c3bac9
CMSSW: CMSSW_12_5_X_2022-07-13-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37963/26212/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@bsunanda Run3-gex136E Try too remove compilation warnings in PhysicsTools/PatAlgos #38670

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6f1ee4/26212/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6f1ee4/26212/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-6f1ee4/11824.9001_TTbar_13+2021PU_SonicTriton+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoNanoPU+HARVESTNanoPU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-6f1ee4/23434.9001_TTbar_14TeV+2026D49PU_SonicTriton+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 3923752
DQMHistoTests: Total failures: 19
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3923710
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 51 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 216 log files, 54 edm output root files, 52 DQM output files
TriggerResults: no differences found

mariadalfonso · 2022-07-15T08:42:37Z

+xpog

the dual inference engine is for the same model, and just for performance evaluation. not for production.

only rebase since last signoff

clacaputo · 2022-07-15T09:20:11Z

+reconstruction

resign after rebase

perrotta · 2022-07-15T10:31:04Z

+1

cmsbuild · 2022-07-15T10:31:24Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

cmsbuild added this to the CMSSW_12_5_X milestone May 16, 2022

cmsbuild added code-checks-pending operations-pending orp-pending pending-signatures reconstruction-pending tests-pending labels May 16, 2022

wpmccormack mentioned this pull request May 16, 2022

Adding DeepMET files in structure needed for SONIC cms-data/RecoMET-METPUSubtraction#6

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels May 16, 2022

cmsbuild added requires-external tests-started and removed tests-pending labels May 16, 2022

cmsbuild mentioned this pull request May 16, 2022

Adding SONIC ParticleNet Producer to CMSSW #37964

Merged

cmsbuild added tests-approved and removed tests-started labels May 16, 2022

cmsbuild added tests-started and removed tests-approved labels May 18, 2022

cmsbuild added tests-pending xpog-pending labels Jul 13, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 13, 2022

cmsbuild added tests-started and removed tests-pending labels Jul 13, 2022

cmsbuild added tests-approved and removed tests-started labels Jul 13, 2022

cmsbuild added xpog-approved and removed xpog-pending labels Jul 15, 2022

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jul 15, 2022

cmsbuild added fully-signed operations-approved orp-approved and removed operations-pending pending-signatures orp-pending labels Jul 15, 2022

cmsbuild merged commit a70261c into cms-sw:master Jul 15, 2022

perrotta mentioned this pull request Jul 15, 2022

Update tag for RecoMET-METPUSubtraction to V01-02-00 cms-sw/cmsdist#7995

Merged

This was referenced Jul 15, 2022

Uodated GEM-CSC triager LUTs cms-data/L1Trigger-CSCTriggerPrimitives#13

Merged

Update BeamSpot in Run3 MC GTs + swap Run3RoundOptics25ns13TeVLowSigmaZ with Realistic25ns13p6TeVEarly2022Collision vtx smearing #38760

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding DeepMET SONIC Producer to CMSSW #37963

Adding DeepMET SONIC Producer to CMSSW #37963

wpmccormack commented May 16, 2022

wpmccormack commented May 16, 2022

cmsbuild commented May 16, 2022

cmsbuild commented May 16, 2022

kpedro88 commented May 16, 2022

kpedro88 commented May 16, 2022

cmsbuild commented May 16, 2022

jpata commented May 17, 2022

jpata commented May 17, 2022

michaelwassmer commented May 17, 2022 •

edited

Loading

kpedro88 commented May 17, 2022

jpata commented May 18, 2022 •

edited

Loading

jpata commented May 18, 2022 •

edited

Loading

cmsbuild commented Jul 13, 2022

cmsbuild commented Jul 13, 2022

kpedro88 commented Jul 13, 2022

cmsbuild commented Jul 13, 2022

mariadalfonso commented Jul 15, 2022

clacaputo commented Jul 15, 2022

perrotta commented Jul 15, 2022

cmsbuild commented Jul 15, 2022

Adding DeepMET SONIC Producer to CMSSW #37963

Adding DeepMET SONIC Producer to CMSSW #37963

Conversation

wpmccormack commented May 16, 2022

PR description:

PR validation:

wpmccormack commented May 16, 2022

cmsbuild commented May 16, 2022

cmsbuild commented May 16, 2022

kpedro88 commented May 16, 2022

kpedro88 commented May 16, 2022

cmsbuild commented May 16, 2022

Comparison Summary

jpata commented May 17, 2022

jpata commented May 17, 2022

michaelwassmer commented May 17, 2022 • edited Loading

kpedro88 commented May 17, 2022

jpata commented May 18, 2022 • edited Loading

jpata commented May 18, 2022 • edited Loading

cmsbuild commented Jul 13, 2022

cmsbuild commented Jul 13, 2022

kpedro88 commented Jul 13, 2022

cmsbuild commented Jul 13, 2022

Comparison Summary

mariadalfonso commented Jul 15, 2022

clacaputo commented Jul 15, 2022

perrotta commented Jul 15, 2022

cmsbuild commented Jul 15, 2022

michaelwassmer commented May 17, 2022 •

edited

Loading

jpata commented May 18, 2022 •

edited

Loading

jpata commented May 18, 2022 •

edited

Loading