v3.240602

This brings in changes to TaskWorker to adapt to modern HTCondor, avoid the use
of deprecated bindings (xquery) or deprecated ways to call them (schedd.Submit(classAds,...))
I.e. first part of #8337

Also fixed problems with htcondor2 discovered by @novicecpp in #8272

Changes to submission are "pervasive" and a good validation is required.

It was preliminary tested on an TW running HTCondor 23.7, but there is no known reason why it should not work on current TW setup with HTCondor 10.2 or the coming 23.0

What's Changed

• new format of schedd.submit)/spool() fix #8336 fix #8333 by @belforte in #8448
• schedd.xquery is deprecated. Use schedd.query. Fix #8447 by @belforte in #8449
• pass string, not bytes, to htcondor.param Fix #8450 by @belforte in #8452
• do not set RequestCpus in task submission JDL. Fix #8456 by @belforte in #8457

Full Changelog: v3.240530...v3.240602

v3.240530

Cutting new release to fix log lines that will be parsed to successful_publication metrics.
Plus small but significant changes for the PyPI image.

What's Changed

• add env.sh scripts, monit scripts compatible with pypi containers by @mapellidario in #8434
• add voms-clients-java to PyPI container. Fix #8437 by @belforte in #8438
• Fix successful_publication log line by @novicecpp in #8446

Full Changelog: v3.240527...v3.240530

v3.240527

This release ensure extraJDL will be sanitized before submitting to Schedd. Plus, many changes on new CI system.

What's Changed

• [Gitlab CI] SKIP_DEPLOY also skip the test, remove unuse SKIP_BUILD by @novicecpp in #8419
• Remove GIT_STRATEGY: clone by @novicecpp in #8422
• sanitize extraJDL. See #8420 by @belforte in #8423
• mount /etc/vomses from the host VM. Fix #8426 by @belforte in #8427
• Add RSE to FTS-ASO metadata 7960 by @belforte in #8430
• beware extra spaces in accelerator_jdl too by @belforte in #8431
• oracle - drop unused columns about panda from tasks table by @mapellidario in #8432
• Release pipeline by @novicecpp in #8421

Full Changelog: v3.240517...v3.240527

v3.240517

Cutting the new release for TW's new bind-mount scheme, plus many fixes.
Together with new CRABClient (still to be tagged and deployed) this will also allow to provide a list of files to be processed in the input dataset, similar to blockList, aka "fileMask"

And thank you @oljemark for the PR!

What's Changed

• Fix typos in output strings by @oljemark in #8408
• typo in default value. Fix #8409 by @belforte in #8413
• Use user input files list as a filemask 8371 by @belforte in #8410
• add 1min delay in AdjustSites.py fix #8411 by @belforte in #8415
• Reduce bind mount to single path for apps data by @novicecpp in #8414

New Contributors

• @oljemark made their first contribution in #8408

Full Changelog: v3.240515...v3.240517

v3.240515

Changes to REST to support use of userInputFiles configuration parameter as a "fileMask" on the input dataset

Brings in also improvements to Gitlab-CI pipeline

What's Changed

• make REST accept both inputDataset and userInputFiles at same time fix #8372 by @belforte in #8395

• handle inputdata='' in REST like if it were missing by @belforte in #8399

• allow for inputdata arg to be missing. Fix #8404 by @belforte in #8407

• Deploy both TW and Publisher with Gitlab-CI by @novicecpp in #8397

Full Changelog: v3.240510...v3.240515

v3.240510

Add automatic handling of failed DBS migration and restructuring of Publisher code to put functions common to Publisher_schedd and Publisher_rucio in Publisher(Dbs)Utils

Also a minor fix to REST

List of changes:

What's Changed

• Fix failed migration handling 8382 by @belforte in #8386
• fix "created" value in FMD retriecal. Fix #8388 by @belforte in #8390
• move common code from PubMaster* to PubUtils. Fix #8387 by @belforte in #8392

Full Changelog: v3.240508...v3.240510

Validation: #8393

v3.240508

What's Changed

Tag created mainly for new env variable in jobs for frontier monitoring and WMCore version 2.3.3

• define env.var. FRONTIER_ID Fix #8366 by @belforte in #8367
• requirements - wmcore upgraded to 2.3.3 by @mapellidario in #8383

but this tag brings also many other changes

Publisher:

• Publisher support for PyPI image by @novicecpp in #8368
• Deal with failed migrations 8244 by @belforte in #8378 (still buggy)

Corrupted files reporting (not part of CRAB services proper)

• Update ProcessBadFilesList.py require at least 3 failure reports by @belforte in #8379
• do not crash ProcessBadList.py in case of no reports by @belforte in #8365
• no need for a flag in /etc anymore. Use -p(arent) in gfal-copy by @belforte in #8369

TaskWorker

• only user reliable RSEs for tape recall. Fix #8356 by @belforte in #8381

Oracle

• Oracle - many new things cumulated over many months by @mapellidario in #8326

Full Changelog: v3.240426...v3.240508

v3.240426

• small fixes for CI/CI pipeline
• make DBSDataDiscovery test work and save output to JSON file
• add fix by David Lange to speedup parent finding (x100)

List of commits:

• attempt to streamline parent file discovery (#8352) by @davidlange6
• make test of DBSDataDiscovery work again and add printout of output (#8362) by @belforte
• allow to fall back proxy to installed cert in runner machine (#8359) by @novicecpp

List of difference: v3.240423...v3.240426

v3.240423

What's Changed

• Remove T2_DE_DESY hardcode and use generic RucioException for add_replicas() instead by @novicecpp in #8325
• Clean up cmsbot's Rucio storage space by @novicecpp in #8327
• stop using AuthenticatedSubprocess. Fix #8277 by @belforte in #8331
• PoC CI pipeline on Gitlab to verify PyPI image by @novicecpp in #8328
• use classad.quote, not HTCondorUtils. Fix #8332 by @belforte in #8346
• monit - run monit scripts inside docker containers by @mapellidario in #8194
• Fix deployTW.sh script, fix CI job tags to crab3 and crab3-shell by @novicecpp in #8348

Full Changelog: v3.240325...v3.240423

v3.240325

fix previous broken release v3.240318 via #8302

Validation
ran 3 Jenkins tests at same time

• CV + CCV + ST #8304
• CV+ CCV +ST #8305
• CCV + ST #8306

all OK.