v3.241107

Main goal change is sandbod handling restructuring for implementing dryrun via preparelocal

Also some cleanup

What's Changed

• enable opensearch ssl_verify back and set PYTHONUNBUFFERED for run.py by @novicecpp in #8746
• remove debug_files handling in TW. Do it in scheduler by @belforte in #8747
• Spelling and missing 'not' in exception message by @oljemark in #8751
• remove duplicates sandbox untarring. Fix #8725 by @belforte in #8753
• CI pipeline for test2 to use crab-dev from IB by @belforte in #8756
• Cleanup deployment files by @novicecpp in #8757
• use dm_weight Fix #8724 by @belforte in #8762
• Make dryrun work again by @belforte in #8767

Full Changelog: v3.241018...v3.241107

v3.241018

implements first part of #6544 : restructure code to avoid handling user sandbox in TaskWorker, but download it directly in the scheduler DAG boostrap process. Just minor, changes of code from @novicecpp in #8645 plus rebasing

Also check that object is visible in S3 after uploading

What's Changed

• add checkS3Object and verify S3 upload with it by @belforte in #8740
• Download sandbox from s3 to scheduler by @belforte in #8743
• Stop downloadind user sandbox in tw by @belforte in #8745

BEWARE

last commit above breaks compatibility with current prod client v3.240930
a compatible Client comes with v3.241025
CRABClient MUST be updated before we can deploy this or later versions of TW

Full Changelog: v3.241017...v3.241018

validation pipeline https://gitlab.cern.ch/crab3/CRABServer/-/pipelines/8349112

v3.241017

What's Changed

• Run spark cronjobs on ipynb directly by @novicecpp in #8735
• correct use of rucio.get_local_account_usage. Fix #8738 by @belforte in #8739
• adding dryrun option for task scheduling by @aspiringmind-code in #8734
• Changing the message when no tasks selected by @aspiringmind-code in #8742
• Various fix for process controller scripts (manage.sh/manage.py/etc.) by @novicecpp in #8737
• RucioUtils instead of python.RucioUtils by @aspiringmind-code in #8744

Full Changelog: v3.241009...v3.241017

v3.241009

This release brings in small fixes, cleanup of unnecessary files and some restructuring.

What's Changed

• fix object has no attribute in manage.py by @novicecpp in #8690
• fix the X509_USER_CERT path, and also add X509_USER_PROXY here, by @novicecpp in #8684
• Enable install WMCore from private repository by @novicecpp in #8683
• Fix #8661 changing local account usage to tape recall usage in quota report by @aspiringmind-code in #8697
• Fix #8663 adding getsubmitrefusedtasks api by @aspiringmind-code in #8698
• Fix retag monit image in release pipeline by @novicecpp in #8706
• fix #8703 and minor improv. to CI by @belforte in #8708
• Cleanup job wrapper by @belforte in #8704
• Remove extra files from TW tarballs by @belforte in #8718
• remove use of site.ad file Fix #8720 by @belforte in #8721
• Add Waiting task status to GenerateMonit by @aspiringmind-code in #8729

Full Changelog: v3.240904...v3.241009

v3.240904

What's Changed

• fix pipeline by @aspiringmind-code in #8637
• Fixing registry caching for monit image by @novicecpp in #8634
• make runContainer.sh help works by @novicecpp in #8639
• Add bash-completion to pypi container by @novicecpp in #8641
• add instuctions for TW logstash config by @belforte in #8642
• monit - set CMD to bash by @mapellidario in #8640
• monit - runcontainer improved by @mapellidario in #8643
• jenkins test - add client validation test for crab recover by @mapellidario in #8087
• add comments by @belforte in #8633
• fix .gitlab-ci.yml comment by @novicecpp in #8649
• Change CI variables name for triggering the pipelines by @novicecpp in #8653
• monitoring image - some new additions by @mapellidario in #8656
• stop using username_CRAB for myproxy. Fix #8655 by @belforte in #8657
• No locations found should result in SubmissionRefused by @belforte in #8662
• fix logging format by @belforte in #8668
• add test14 by @novicecpp in #8669
• fix test14 by @novicecpp in #8670
• Fix rucio cleanup.sh and add SUBMITREFUSED to ST by @novicecpp in #8671
• spark monitoring cronjobs - a small refactor by @mapellidario in #8672
• use CRAB_Requested* ads in resubmission. Fix #8654 by @belforte in #8674
• suspend tape recall rule before deleting. Fix #8647 by @belforte in #8652
• Refactor process controller scripts by @novicecpp in #8658
• add status subresource to RESTTask and use it in task_info.js by @belforte in #8680
• fix 8658: manage.py which cause manage.sh always run on debug mode by @novicecpp in #8682
• bump WMCore to 2.3.rc2 by @belforte in #8687
• Task scheduling by @aspiringmind-code in #8676

Full Changelog: v3.240819...v3.240904

v3.240809.p1

simply bump WMCore to 2.3.6rc2 to pick up fix to voms-proxy-info concurrency issue

What's Changed

• bump WMCore to 2.3.6rc2 (#8687) by @belforte in #8688

Full Changelog: v3.240809...v3.240809.p1

v3.240819

This is mostly a way to update crabtwfilebeat:v3.latest container to fix CheckTapeRecall.
It also brings in:

• faster container building
• changes in new v2 condor bindings usage (default is to use v1)
• small changes to TW for paving the way to v2 binding which needs more testing in preprod

What's Changed

• use TaskWorkerConfig to enable use of V2 bindings by @belforte in #8595
• improve message. Fix #7857 by @belforte in #8608
• do not use deprecated schedd.transacion. Fix #8597 by @belforte in #8610
• workaround to fix #8604 (for v2 bindings) by @belforte in #8612
• Change xquery to query 8613 by @belforte in #8615
• add SUBMITREFUSED to task statuses to monitor. Fix #8620 by @belforte in #8621
• cmsrun - adapt to deprecated -p parameter by @mapellidario in #8623
• trim DBSDataDiscovery logging and add dataset name by @belforte in #8629
• Kaniko to DinD & implement registry caching #8522 by @jannakam in #8627
• rename classad TaskType to CRAB_DAGType. Fix #8488 by @belforte in #8616
• Fix small bugs by @belforte in #8632

New Contributors

• @jannakam made their first contribution in #8627

Full Changelog: v3.240809...v3.240819

v3.240809

Bump WMCore version to 2.3.3, small fixes and upgrades in TW/Pub.

What's Changed

• publisher monit - updated grok for PUBSTART by @mapellidario in #8580
• tw filebeat - new docker image by @mapellidario in #8576
• bump wmcore to 2.3.3 in pypi images and add deprecate notice to requirements.txt by @novicecpp in #8581
• fix #8585 by @belforte in #8586
• Adding site error code 10040 as recoverable error by @aspiringmind-code in #8596
• update HTC to 23.9.6 in TW pypi image by @belforte in #8605

Full Changelog: v3.240731...v3.240809

v3.240716.patch1

Cherry pick #8581 to the v3.240716 to bump WMCore version to 2.3.3. Fixing "Unsupported architecture el9" error.

v3.240731

This tag mainly brings in :

• a new task status, to differentiate between tasks submission failures that are caused by user problems and by CRAB system problems: Add submitrefused 7989 by @belforte in #8564
• a fix about monitoring in crabserver REST: Add monitoring script back to REST's PyPI image by @novicecpp in #8569

What's Changed

Moreover, this tag brings in the new CRABUtils/TaskUtils.py module

• add CRABUtils/TaskUtils.py by @belforte in #8557
• use common functions from CRABUtils/TaskUtils fix #7671 by @belforte in #8570

an improvement of the resilience of FTS_Transfers to FTS errors

• put all calls to fts3.get_job_status in try/except Fix #8556 by @belforte in #8558

an improvement in Publisher pypi image:

• Fixing PB entrypoint by @novicecpp in #8562

and some other changes

• Small fixes for rucio publisher by @belforte in #8566
• Add vchakrav remote repository to addGH.sh by @aspiringmind-code in #8551
• More verbose for cicd/rucio/cleanup.py by @novicecpp in #8565
• Fix TW config section does not exist when hardcode some values in pdb mode by @novicecpp in #8552
• [CI] fix SUBMIT_STATUS_TRACKING by @novicecpp in #8553
• [CI] Support CRABClient GH by @novicecpp in #8554
• [CI] Add Vijay dev env by @novicecpp in #8555
• Update .gitignore by @novicecpp in #8560

Full Changelog: v3.240716...v3.240731