Bulk drone submission / operations

[maxfischer2781]

The HTCondor batch system¹ allows bulk operations and especially bulk submission. Using these from TARDIS could greatly improve scalability.

Bulk operations, e.g. removing two jobs at once, remove some overhead from invoking the API and concurrency protection. Bulk submission, e.g. submitting two identical jobs at once, allows for more efficient handling of jobs by the batch system.

---

From a programming standpoint, it is easy enough to retrofit bulk operations onto individual operations. For example, instead of directly running condor_rm 123.0 and condor_rm 321.0, we can buffer the Job IDs and run a single condor_rm 123.0 321.0 after a moment. However, we should discuss whether any higher-level routines need changes to properly deal with the concurrency and delays introduced by such changes.

Using bulk submission introduces some additional complexity: we need to separate common from individualised job features. Preliminary discussion has identified the TardisDroneUuid as a critical, individual job feature. We should discuss whether it is appropriate to use HTCondor's features like the queue from command, or whether there are other alternatives to making drones less individualised.

---

I assume these would not be either-or changes. There are likely some low-hanging fruits, like aggregating deletions, that can be added separately and faster from other changes. Similarly, there are probably different levels of streamlining job submission.

Feel encouraged to discuss individual changes as well as "not perfect" changes if there's likely good benefit for reasonable effort.

---

¹ This may also apply to other systems for which we have site adapters. Suggestions welcome.

[giffels]

I agree that supporting bulk submission or array jobs is a very good idea. As you mentioned, the TardisDroneUuid needs to be unique for each job. At least in HTCondor this is possible by using queue from syntax. Other batch systems like Slurm or Moab need to checked for a similar functionality.

I could imagine to queue "commands" and execute them in one go as soon as

• A certain threshold concerning the number of queued commands is reached
• or commands are queued for a certain time

This would indeed improve the scalability and reduce the load on the batch system as well.

[maxfischer2781]

I think in practice we can guess reasonable defaults for both threshold and delay. For HTCondor, a job ID is usually less than 12 char/byte – even a small ARG_MAX neatly fits some thousand jobs. Having bulks in the range of 100-1000 and delays in the range of 0.1s - 1s seems adequate.

[maxfischer2781]

To document the internal workflow and where we can hook in:

Right now TARDIS uses COBalD's FactoryPool, providing it with the create_drone factory function which is a thin wrapper around constructing Drone pools. After construction, each Drone acts independently and may suspend/remove its resource at any time.
This means that with the current setup, the FactoryPool splits "need n more supply" into n separate Drones. Notably, FactoryPool just expects the Drones to spawn with a specific demand; it's fine for a Drone to provide adequate supply later. Nothing synchronous is expected anywhere here.

At the lowest level, we can hook into the SiteAdapter and queue requests. See Manuel's post.
I think we can rather easily provide a queuing-framework for SiteAdapters to hook into. Instead of directly running the command in an executor, we would submit it to a queue and await some success/error token. That's not too different from how the new SSHExecutor setup (see #218) works, but we need to provide a template to stitch together the commands.

At a higher level we can modify the FactoryPool, which ties into #172 as well.

• Modify FactoryPool to request multiple drones/child-pools at once. The factory callback would receive the missing demand and construct however many drones are appropriate.
• Create a custom Pool for TARDIS. This would directly represent bulks of drones, and could subsequently submit bulk commands to the SiteAdapter. Consequently, SiteAdapter would have stop_resource_bulk (name pending) in addition to stop_resource, similarly for other methods.

[giffels]

IMHO I would prefer to extend the SiteAdapter rather than to change the FactoryPool. In case we extend the FactoryPool, we will lose the Drone as atomic unit, but deal only with sets of drones, if I got that right.
In case of extending the SiteAdapter we can keep the Drone as atomic unit and can have the possibilty to trigger a bulk submission individualy either after requests are queued for a certain time or after a certain number of queued requests is archived. IMHO I think this approach is more flexible.

[giffels]

Initially I thought about inheriting from the exisiting adapters, so the user can easily choose (in the configuration) if she/he would like to use bulk submission or single submission. However, since single submission is sometimes the culprit that things fail, we could also make bulk submission the only supported method.

[maxfischer2781]

I'm on board with making bulk operations the only supported method. If we get it working it should definitely be the default, and I cannot think of a reason why one would want to switch it off; using a bulk size of 1 would be equivalent for the batch system. There's not much point in supporting single submission, then.

[giffels]

Exactly!