-
Notifications
You must be signed in to change notification settings - Fork 107
DBS3Upload
ticoann edited this page May 4, 2018
·
3 revisions
The DBS3Upload component does following tasks:
- Find all files that have not yet been uploaded to DBS, and the fileset-algo pairs that go along with them.
- Upload these files. This is handled through DBSInterface which first sends the algo, then the dataset, and then commits all of the files. It also handles migrating to global is you specify a global URL and the file blocks are full (have reached the limit in the DBS config)
- The code cleans up at the end, check for any open blocks that have exceeded their timeout.
This is component maintains separate tables (dbsbuffer_*) for its tasks. Those tables contain a subset information of wmbs. Since wmbs data gets cleaned up when request is finished, it needs to be copied over to dbsbuffer until data gets to be uploaded to DBS. Currently there is no trigger for deleting data from dbsbuffer tables.
The data is filled from WorkQueueManager (input data) and JobAccountant (output data). Also file parentage is derived in JobAccountant.