v3.230911

Improvement on Tape Recall and Rucio ASO (including fix for Publisher), plus convenience script for delete the old Rucio rules.

• report back progress of tape recall. FIx #7593 by @belforte in #7856
• Taperecall: allow to recall any data tier if size less than threshold by @novicecpp in #7862
• Store Rucio ASO information to tasksDB by @novicecpp in #7858
• Attach Rucio's replication rule to Publish Container by @novicecpp in #7860
• RUCIO_Transfers: Mark file as fail to transfer when add_replicas raise exception by @novicecpp in #7866
• report # files still to be pub. Fix #7854 by @belforte in #7855
• add script to delete old Rucio rules by @belforte in #7859

Full Changelog: v3.230824.1...v3.230911

v3.230824.1

a few changes to Publisher_rucio via #7853

• this brings in a new functionality to allow to test migrations w/o digging for obscure datasets not migrated yet
  • use config.TaskPublisher.DBShost to switch publication to int/phys03
• these are "cosmetics"
  • more cleanup and simplification Fix #7842
  • pylint and pep8

There is only one Warning left in pylint, related to a TODO captured in #7854

v3.230824

This release introduces a new Spark data pipeline for CRAB by our 2023 CERN Summer Student @ekongat, and new Rucio ASO code replacing the old Rucio ASO scripts.

And some (not so small) changes on TW/Pub.

What's Changed

• spark pipeline for crab usage analytics by @ekongat in #7837
• Merge rucio_transfers to master by @novicecpp in #7848
• stop blacklisting KIPT datalocation. Fix #7846 by @belforte in #7847
• Refactor task publish rucio by @belforte in #7849

Full Changelog: v3.230818...v3.230824

v3.230818

changes how to start Publisher and brings in a working Publisher_rucio.

we need to make sure we properly validate change to start Publisher procedure

v3.230817

Consolidates changes to Jenkins validation to check Publication and Resubmission
Mostly a way to get out of the way a very long list of similarly named commits where I found no cleaner way to do thing than pushing changes to master all of the times in order to test them. Sorry about that.

v3.230810

FIxes bug in v3.230803 ( #7799 )and brings in a couple hopefully harmless improvements.
Candidate for validation in preprod

List of changes

• refactor getCentralConfig Fix #7758 (#7792)
• Fix #7799 (#7800)
• stop reading tm_rest_* from DB. Fix #7795 (#7796)

Validation

• #7803
• #7804

also checked publication for largest SST tasks

v3.230803

BROKEN - fix on the way see #7799

consolidate recent changes, nothing dramatic.
Major change is recaftoring and cleanup of DBSDataDisvovery in #7750

list of commits:

• allow Publisher to retrieve dbs_blockname and block_complete (#7777)
• TapeRecallManager acts task created by any TW. Fix #7788 (#7789)
• TW - set job classads to match arch in first and second matchmaking (#7779)
• document format of Publisher_files (#7781)
• remove ref to obsolete tm_rest*. Fix #7721 (#7722)
• move DataRecall and DataLock to separate file (#7750)

v3.230714

a TW release
purpose is to restore monitoring of tape_recall_quota after moving TW to crab-prod-tw02 and adding monitoring of crab_input quota

• use recurring action to report Rucio quotas. Fix #7737 (#7760)

change to recurring action was already tested and is now deployd in crab-prod-tw02
So this is safe to build an deploy, just a useful bookkeping

v3.230712

This is a TaskWorker release.
FIxes one critical (even if currently rarely occurring) bug and introduce
new feature to create a Rucio rule for each inputDataset to guarantee that stays on disk.

LIst of changes:

• do not lock data when using partialDataset (#7744)
• beware rules created by other accounts. Fix #7742 (#7743)
• lock input data when submitting. Fix #7592 (#7732)

Validation in #7748

v3.230705

Hotfix on tape recall operation

• remove use of freeRSE. Fix #7714 by @belforte in #7715

Changes in TaskWorker

• expand tar files in user sandbox. Fix #7661 by @belforte in #7680
• expand user tar files in /srv . Correct fix for #7661 by @belforte in #7708
• Cleanup cms run analysis.py by @belforte in #7681

Misc

• properly hande the case of no replicating/pending rule by @belforte in #7677
• avoid belforte's private clone and cernbox by @belforte in #7678

Full Changelog: v3.230616...v3.230705

Tests:

• #7717 CMSSW_13_0_2
• #7723 CMSSW_12_6_4
• #7724 CMSSW_11_3_4
• #7725 CMSSW_7_6_5
• #7726 CMSSW_10_6_26
• #7729 CMSSW_13_0_2