bottomless: stream gzip snapshot

[neubaner]

Stream gzip snaphosts using S3's multipart upload.

S3 requires Content-Length to be set when doing a PutObject operation and since we can't know the size of the gzip stream upfront, we send it in parts to S3 using a fixed-sized buffer that increases every 16 chunks up to 100 MiB, ensuring small allocations for small databases.

[psarna]

Nice work! Thanks for contributing, left a few nitpicks. I wonder if error handling for multipart uploads doesn't get handled by itself already -- e.g. when https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html is configured. The temporary leftover file for storing the last part is also handled on failure, since it will get deleted eventually -- and if you switch to tempfile, it would probably even get deleted as part of its drop routine.

[neubaner]

Nice work! Thanks for contributing, left a few nitpicks. I wonder if error handling for multipart uploads doesn't get handled by itself already -- e.g. when https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html is configured. The temporary leftover file for storing the last part is also handled on failure, since it will get deleted eventually -- and if you switch to tempfile, it would probably even get deleted as part of its drop routine.

Yes, but you are still billed by how much you retain the incomplete parts. I also don't know if this can be done for other S3-compatible stores.

As a best-effort we can wrap the upload parts in a tokio::spawn and check if the JoinHandle returned JoinError for panics or the Result is Err.

[neubaner]

I decided to not abort the multipart upload on panics, that was harder than what I initially thought because tokio::spawn requires the Future to be 'static.

Only catching errors should be enough for now, and we can leave the panic case to the AbortIncompleteMultipartUpload lifecycle setting.

[Horusiath]

In general this PR should be married with #574 - one cover the issue of asynchronous snapshot upload between checkpoints (which is a must have in order to avoid latency spikes), another covers multipart upload.