New option --max-total-diskbytes to restrict disk storage used by a Hurl session

[lambrospetrou]

Problem to solve

I want to control how much disk space a single hurl ... execution can use.

The existing --max-filesize option works at the single response level, but a Hurl file could (intentionally or not) repeat a request hundreds/thousands of times and saturating the disk at some point.

I understand that the vast majority of users probably control their own Hurl usage on their own systems, so they don't need this, hence why it's a feature proposal 😄 But, for running Hurl on a platform this becomes very useful to avoid using more system specific approaches (e.g. Docker, cgroups, hardware isolation, etc).

Proposal

Add a new option, for example --max-total-diskbytes that will limit the amount of responses saved locally across the whole Hurl execution. The execution should not necessarily fail, but do not store responses on disk any more, only keep them in memory as per the script to allow its assertions to function.

This limit is enforced only per execution, so no need to keep state across invocations, but should apply across all files (even when running in parallel).

Depending on the implementation, this can lead to some races since parallel execution is using separate threads, but my guess is that adding some lock-contention just before writing responses to the disk would be negligible overhead compared to the actual network downloads, but immensely useful.

Additional context and resources

This is for the platform https://www.skybear.net/ that runs user-provided Hurl files on cloud hosts.
Other CI systems, even internal to a company it could be useful to avoid some rogue scripts that have bugs from taking down CI hosts.

Tasks to complete

• ...