Rclone: stream files as they are being listed

[Michal-Leszczynski]

This PR changes the rclone version used by SM-agent server.
The goal is improve huge dir listing by streaming the files to the SM as the dir is being listed. Rclone lists files in chunks (1000 is the default value for s3), so we can start streaming those files after each chunk has been listed, not only after the whole dir has been listed.
This allows for more reliable timeout handling on SM side, and also decreases memory pressure on agent.

Since this is timeout/performance change, I didn't prepare a dedicated test for it, but I created a benchmark for that (see last commit for details). It measures some interesting averages when listing a flat dir with 5555 files 1000 times. Note that this has been tested on my local docker setup without any rate limiting, but I still think that the results show what needs to be seen.

The results without the fix:

Avg total time: 157.916848ms           (average time of listing the whole dir)
Avg first latency: 146.564896ms        (average time after the first item has been listed)
Avg max cross chunk latency: 19.196µs  (average time between listing the last item from given chunk and the first from the next chunk)
Avg max within chunk latency: 112.92µs (average time between listing files from the same chunk)

The results after the fix:

Avg total time: 143.211603ms
Avg first latency: 26.344503ms
Avg max cross chunk latency: 26.144269ms
Avg max within chunk latency: 320.72µs

What is important, is that this fix reduces the time needed for listing the first item from ~146ms to ~26ms. Let's have in mind that 5555 files is not a huge amount for backed up sstables, and that the local setup has faster connections than the real one. It also slightly reduces the total time needed for listing, but the values are too similar to tell it for sure.

Fixes #4132

[VAveryanov8]

Nice benchmark! LGTM!

As I understand all the heavy lifting was done in scylladb/rclone repository?

[karol-kokoszka]

👍

Please monitor https://jenkins.scylladb.com/view/scylla-manager/job/manager-master/ SCT tests after merging.
It may be even worth to start the tests just on this branch.

[Michal-Leszczynski]

As I understand all the heavy lifting was done in scylladb/rclone repository?

Correct, but we treat it more as a playground, so it would be good to review vendored changes as a part of this PR.

[mykaul]

Is this another private patch on top of our forked rclone?

[karol-kokoszka]

Is this another private patch on top of our forked rclone?

yes

[Michal-Leszczynski]

After the discussion with @VAveryanov8 it turns out that this issue/PR does not really make sense (thanks for the effort, you saved us from unnecessary rclone patches that could result in more problems in the future).

When developing the patch, I was under the impression that we use non-recursive RcloneListDirIter when we list node''s sstables in backup location during backup's dedup stage. This came from the fact that this dir should be flat, so using flat listing should be better or not worse than recursive listing.

It turned out that we are using recursive RcloneListDirIter for that (@karol-kokoszka I think that you were implementing it - any idea why the recursive version was chosen)?

Even though this PR "improves" flat dir listing, it's not connected to https://github.com/scylladb/scylla-enterprise/issues/4861, which was a driving force behind that, and we shouldn't risk questionable improvements connected to rclone modifications. The only way in which this patch PR could help with mentioned issue, is that perhaps flat dir listing has indeed noticeably better performance than the recursive version, but I doubt that's the case.

Anyway, I'm closing this PR, perhaps it will be revisited in the future.

[karol-kokoszka]

It turned out that we are using recursive RcloneListDirIter for that (@karol-kokoszka I think that you were implementing it - any idea why the recursive version was chosen)?

I don't think there is any special cause that lead to choose recursive over the flat approach.