v3.231117

Hotfix on getNativeRucioClient() where it pick up wrong configuration,
and Stefano's data processing on the corrupt file report.

What's Changed

• much better version of ProcessBadFiles.py by @belforte in #8004
• add checkFiles.py by @belforte in #8009
• Improve process input files by @belforte in #8010
• bunch of typos in last commit by @belforte in #8011
• Fix getNativeRucioClient load wrong config by @novicecpp in #8006

Full Changelog: v3.231110...v3.231117

v3.231110

A working tag to capture some recent minor enhancements to TaskWorker
to allow testing e.g. with Jenkins before more, unrelated, commits are merged.
No changes to REST nor Publisher

Content:

• properly report HTTP 507 for WebDav
• better messages to user when some data is not available
• lock input data also when using partialDataset

List of commits:

• Better messages fix 7596 (#8000)
• fix #7948 also restrict partialDataset to disk data (#8001)
• Update TapeRecallManager.py
• avoid commas in transfer_reasons. Fix #7864 (#7993)
• Update ProcessBadFilesList.py
• add FTS error status code 507 for WebDav. Fix #7863 (#7998)
• move STAGEOUT_ERROR from WMCore to ServerUtilities (#7995)
• Update TapeRecallManager.py: wait 7 days is enough

Full changelog: v3.231103...v3.231110

v3.231103

Rucio stageout now checks the quota before submitting tasks to Schedd.

• Add playbook to download task_process from schedd by @novicecpp in #7992
• Move Rucio check write logic to ServerUtilities.py by @novicecpp in #7987

Full Changelog: v3.231101...v3.231103

v3.231101

"Mergency" release to fix a bad bug in TapeRecallManager #7990
Already tested (and implemented as a hack) in crab-prod-tw01

brings in a few other minor changes in areas irrelevant for production.
List of commit:

• fix #7990 (#7991)
• Update CheckDiskAvailability.py
• add cmssw_10_1_0 to the list of what we test (#7974)
• fix stdout/pj urls reported for corrupted file (#7973)
• Do not skip UA_KIPT in CheckDiskAvailability.py

Full list of changes : v3.231016...v3.231101

v3.231016

Small fixes in Rucio ASO and TapeRecallManager.

• Fix keep all successfully transferred files to publish container. by @novicecpp in #7949
• Small change to msg to user in TapeRecallManager.py. by @belforte in 300cd66

Full Changelog: v3.231013.1...v3.231016

v3.231013.1

I had forgotten to change loglevel to INFO also for PublisherMasterRucio and a couple more messages not parsed by logstash, but simply to make log a bit more clear now that default level is INFO.
Commit:

• same as #7945 also for Publisher_rucio and a couple more places (#7946)

Validation

tested in preprod. Will deploy on prod on Monday

v3.231013

Minor update release, firtst change below is important for monitoring :

• restore plot of acquired publication by task. (belforte) - Fix #7944 (#7945)
• improve reporting of TapeRecall status to users (belforte) - Fix #7900 (#7901)
• fixes for Jenkins tests (belforte) - #7919 , 50a9f8a and Fix #7896 (#7899)
• some cleanup (Remove old TapeRecallStatus) (beflorte) - #7902

v3.231006

A Bug fix to support a separate certificate for Rucio client, plus some other fixes and enchantments.

• crabspark - use cmscrab keytab, write parquets to /cms/users/cmscrab by @mapellidario in #7890
• Better catching of bad input files by @belforte in #7891
• restcache, download sandbox - check username present if not upload by @mapellidario in #7892
• Fix PreDAG.py to always use user proxy for rucio client by @novicecpp in #7897

Full Changelog: v3.230921...v3.231006

v3.230921

A few enhancements:
TW

• Detect and report corrupted files (#7867)
• Process corruption reports fix 7883 (#7887)
• remove all uses of -p, --parameter-set with cmsRun (#7888)
• RUCIO_Transfers: support group account (#7886)
• list also suspended rule. Fix #7884 (#7885)

REST

• fix monit url in ui (#7882)

v3.230913

Small fix on deployment scripts.

• Bump crabtaskworker baseimage to 20230427-stable by @novicecpp in #7868
• Fix including ASO dir to RPM build by @novicecpp in #7872

Full Changelog: v3.230911...v3.230913