Make VirtualFile::{open, open_with_options, create,sync_all,with_file} async fn

[arpad-m]

Problem

Once we use async file system APIs for VirtualFile, these functions will also need to be async fn.

Summary of changes

Makes the functions open, open_with_options, create,sync_all,with_file of VirtualFile async fn, including all functions that call it. Like in the prior PRs, the actual I/O operations are not using async APIs yet, as per request in the #4743 epic.

We switch towards not using VirtualFile in the par_fsync module, hopefully this is only temporary until we can actually do fully async I/O in VirtualFile. This might cause us to exhaust fd limits in the tests, but it should only be an issue for the local developer as we have high ulimits in prod.

This PR is a follow-up of #5189, #5190, #5195, and #5203. Part of #4743.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

1640 tests run: 1566 passed, 0 failed, 74 skipped (full report)

---

Code coverage (full report)

• functions: 52.8% (7593 of 14380 functions)
• lines: 81.4% (44633 of 54841 lines)

---

_{The comment gets automatically updated with the latest test results
eb2dd71 at 2023-09-07T22:36:49.297Z :recycle:}

[hlinnaka]

I was confused that even though this marks open_with_options as async, it still uses the synchronous std::fs::OpenOptions::open() under the hood. But I guess that was intentional; all the read and write functions are similarly using synchronous functions under the hood. I gather the plan is to change the interface first, to force all the callers be async, and switch the implementation later. Please explain that in the commit message, so that future code historians won't be confused.

[arpad-m]

I gather the plan is to change the interface first, to force all the callers be async, and switch the implementation later. Please explain that in the commit message, so that future code historians won't be confused.

Yeah Christian requested it in #4743:

All code up to the point where we issue the system call to read from Image or Delta layer files is async.
The system call itself remains synchronous, it is not converted to tokio::fs.
This allows merging the work in this PR without performance regressions caused by tokio::fs, thereby isolating any potential performance regressions to any refactoring changes required.

I will add an explanation tomorrow to the commit msg.

[arpad-m]

Hmmm seems I'm running into async closure lifetime issues (they are well known):

   --> pageserver/src/virtual_file.rs:709:58
    |
709 |           test_files("virtual_files", |path, open_options| async {
    |  ______________________________________----______________-_^
    | |                                      |                 |
    | |                                      |                 return type of closure `[async block@pageserver/src/virtual_file.rs:709:58: 712:10]` contains a lifetime `'2`
    | |                                      has type `&'1 std::path::Path`
710 | |             let vf = VirtualFile::open_with_options(path, open_options).await?;
711 | |             Ok(MaybeVirtualFile::VirtualFile(vf))
712 | |         })
    | |_________^ returning this value requires that `'1` must outlive `'2`

Maybe one could use dyn futures, given that it is tests (and also, in there it's just done once for each file)?