Support listing file sources asynchronously

[kysrpex]

The method list() from galaxy.files.sources.BaseFilesSource lists the directories and files within a file source. An optional keyword argument recursive (False by default) lets it recursively retrieve directories and files within a specific directory.

This operation is very cheap in terms of CPU and expensive in IO terms, be it network or filesystem IO. Depending on how the underlying system is built, it may support retrieving directories and files recursively or not. If it does not, then every time a directory is listed, it is necessary to make another request to list each subdirectory. This may end up involving hundreds of requests. Done sequentially, this can be extremely slow, especially if each one involves network access.

This PR makes the list() method asynchronous, which enables Galaxy to wait for the underlying system to complete the requests concurrently, resulting in a massive speedup. The price to pay is the extra complexity of using the async primitives.

Since this change implies that all functions in the chain up to the API endpoints and the test functions must also be made asynchronous, this PR also takes care of it.

The changes from this PR are meant to address the friction that arises when integrating Galaxy with eLabFTW. Due to how the eLabFTW API is designed, listing the file source with recursive=True requires sending one request for each experiment or resource that contains at least one attachment. Given a large enough eLabFTW instance, this easily translates to hundreds of requests, and the whole process takes an eternity when done serially. Making so many requests concurrently is still not ideal, but at least listing ~500 experiments or resources with attachments becomes bearable.

This is the second PR of a series of PRs that integrate eLabFTW with Galaxy via a file source (together they address issue #18665):

• Let file sources choose a path for uploaded files #19154
• Support listing file sources asynchronously #19256
• eLabFTW integration via Galaxy file source #19319

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.