Merge rucio_transfers to master (#7848)

See commit history in https://github.com/dmwm/CRABServer/tree/rucio_transfers

* RUCIO_Transfer refactoring: Boostrap refactoring and add building task's dataset (#7587)

* first draft

* Fix anchor script and add comment

* adjust code according to Stefano's comments

* fix comment

* fix ruleid bookkeeping (#7636)

* RUCIO_Transfers.py: Add RegisterReplicas action and separate transfer container from publish container. (#7649)

* RUCIO_Transfers.py: add MonitorLockStatus action (#7675)

* Improving readability (#7716)

- The separation between rucio replica and the metadata we use for crab rest. The metadata is called fileDoc.
- Change the return variable structure from prepare().
- Rename update to rest method to reflect state change.
- Remove initReplicasInContainer() and use populateLFN2DatasetMap() directly where it needed.
- remove "skip monitoring lock for the files in the same run.", the useless statement.

* add checksums in "addoutput" (#7719)

* fix update rule id when upload transfer state to to rest (#7776)

* unique FakeDataset name (#7773)

* fix creating new dataset only when needed and close when time passes (#7772)

* skip logfiles transfer (#7774)

* read transfer info from file that need to publish and filter out not-to-publish files from publish container (#7775)

* fix wrong condition for selecting item from transferItems (#7786)

* RUCIO_Transfers.py: Fix bookkeeping block complete (#7785)

* remove T2_UK_SGrid_Bristol hardcode (#7791)

* Fix WebUI to display Rucio URL to the rule (#7801)

* add ASO dir to CRAB3.zip

* add gnu time to logs max mem rss usage

* changes some default value of Rucio ASO

* remove OldRucioTransfer.py

* RUCIO_Transfers.py: fix pylint before merge to master (#7850)

bin/htcondor_make_runtime.sh

@@ -56,7 +56,7 @@ if [[ "x$RPM_RELEASE" != "x" ]]; then
     popd
 
     pushd $ORIGDIR/build/lib
-    zip -rq $STARTDIR/CRAB3.zip RESTInteractions.py HTCondorUtils.py HTCondorLocator.py TaskWorker CRABInterface  TransferInterface -x \*.pyc || exit 3
+    zip -rq $STARTDIR/CRAB3.zip RESTInteractions.py HTCondorUtils.py HTCondorLocator.py TaskWorker CRABInterface  TransferInterface ASO -x \*.pyc || exit 3
     popd
 
     mkdir -p bin
@@ -93,7 +93,7 @@ else
     popd
 
     pushd $CRABSERVER_PATH/src/python
-    zip -rq $STARTDIR/CRAB3.zip RESTInteractions.py HTCondorUtils.py HTCondorLocator.py TaskWorker CRABInterface TransferInterface -x \*.pyc || exit 3
+    zip -rq $STARTDIR/CRAB3.zip RESTInteractions.py HTCondorUtils.py HTCondorLocator.py TaskWorker CRABInterface TransferInterface ASO -x \*.pyc || exit 3
     popd
 
     mkdir -p bin

scripts/cmscp.py

@@ -27,6 +27,7 @@
 from WMCore.Storage.Registry import retrieveStageOutImpl
 from WMCore.Algorithms.Alarm import Alarm, alarmHandler
 import WMCore.WMException as WMException
+from Utils.FileTools import calculateChecksums
 
 ## See the explanation of this sentry file in CMSRunAnalysis.py.
 with open('wmcore_initialized', 'w') as fd_wmcore:
@@ -314,6 +315,9 @@ def add_output_file_to_job_report(file_name, key = 'addoutput'):
         print(msg)
     else:
         output_file_info['size'] = file_size
+
+    (adler32, cksum) = calculateChecksums(file_name)
+    output_file_info['checksums'] = {'adler32': adler32, 'cksum': cksum}
     is_ok = add_to_job_report([(key, output_file_info)], \
                               ['steps', 'cmsRun', 'output'], 'update')
     if not is_ok:
@@ -449,7 +453,7 @@ def perform_local_stageout(local_stageout_mgr, \
         signal.alarm(0)
     if retval == 0:
         dest_temp_file_name = os.path.split(dest_temp_lfn)[-1]
-        
+
         # If fallback stageout happens, PNN can be different as source
         if 'PNN' in stageout_info:
             print("INFO: PNN is defined in site-local-config. %s changed to %s" % (source_site, stageout_info['PNN']))