Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

move timeouts and cancellation handling to remote_storage #6697

Merged
merged 38 commits into from
Feb 14, 2024
Merged

move timeouts and cancellation handling to remote_storage #6697

merged 38 commits into from
Feb 14, 2024

Conversation

koivunej
Copy link
Member

@koivunej koivunej commented Feb 9, 2024 •
edited
Loading

Cancellation and timeouts are handled at remote_storage callsites, if they are. However they should always be handled, because we've had transient problems with remote storage connections.

  • Add cancellation token to the trait RemoteStorage methods
    • For download*, list* methods there is DownloadError::{Cancelled,Timeout}
    • For the rest now using anyhow::Error, it will have root cause remote_storage::TimeoutOrCancel::{Cancel,Timeout}
    • Both types have ::is_permanent equivalent which should be passed to backoff::retry
  • New generic RemoteStorageConfig option timeout, defaults to 120s
  • Start counting timeouts only after acquiring concurrency limiter permit
  • Cancellable permit acquiring
  • Download stream timeout or cancellation is communicated via an std::io::Error
  • Exit backoff::retry by marking cancellation errors permanent

Fixes: #6096
Closes: #4781

koivunej
koivunej commented Feb 9, 2024
View reviewed changes
safekeeper/src/wal_backup.rs Outdated Show resolved Hide resolved
koivunej
koivunej commented Feb 9, 2024
View reviewed changes
safekeeper/src/wal_backup.rs Outdated Show resolved Hide resolved
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 9, 2024 •
edited
Loading

2436 tests run: 2316 passed, 0 failed, 120 skipped (full report)

Flaky tests (4)

Postgres 15

  • test_create_snapshot: debug
  • test_sharding_split_unsharded: release

Postgres 14

  • test_sharding_split_unsharded: release
  • test_sharding_split_smoke: release

Code coverage (full report)

  • functions: 55.9% (12889 of 23067 functions)
  • lines: 82.4% (69922 of 84825 lines)
The comment gets automatically updated with the latest test results
31ade7f at 2024-02-14T21:01:48.911Z :recycle:

koivunej
koivunej commented Feb 9, 2024
View reviewed changes
pageserver/src/tenant.rs Outdated Show resolved Hide resolved
Base automatically changed from move_timeout_cancel2 to main February 9, 2024 12:53
@arpad-m arpad-m self-requested a review February 9, 2024 13:59
@jcsp jcsp mentioned this pull request Feb 12, 2024
Closed
@koivunej
remove Unpin requirement from DownloadStream
a0b5ea1 
I am unsure why that was even there; we do not need Unpin but also I am
surprised it ever was.
@koivunej
feat: introduce the DownloadStream adapter
c3bea0b 
it returns an error on the FIRST timeout or cancel, which can be ignored
to disable it.
@koivunej
enhance TimeoutOrCancel, re-export it
5b06378 
as it will be used as the root cause of anyhow, we have no choice but to
do the pointer chasing on queries.
@koivunej
impl: upload
a6ee8b3
@koivunej koivunej force-pushed the move_timeout_cancel3 branch 3 times, most recently from 1005167 to df74b57 Compare February 13, 2024 13:57
@koivunej koivunej force-pushed the move_timeout_cancel3 branch from e47e937 to eb2449b Compare February 13, 2024 15:09
@koivunej koivunej marked this pull request as ready for review February 13, 2024 15:54
@koivunej koivunej requested review from a team as code owners February 13, 2024 15:54
koivunej and others added 17 commits February 14, 2024 08:00
@koivunej
s3_bucket: move cancel to acquring (all methods)
0b016ae
@koivunej
remote_storage: move the timeout as internal config
c501ead 
this has the minimal changes, no api is changed yet.
@koivunej
external changes
e64b1d2
@koivunej
azure+s3: timeout individual requests
6528b4f
@koivunej
rework: align anyhow errors in cancellation
abdd547
@koivunej
sk FIXMEs were wrong
830fafc
@koivunej
s/state/status + manual fmt
11c45ba
@koivunej
tenant: solve the fixme by understanding retries
2cd795c
@koivunej
fix: timeout parsing
ce95b0b
@koivunej
azure: download on individual requests
f4bd828
@koivunej
azure: more per request timeout
cb121c7
@koivunej
azure: use tokio::time::timeout
d321b14
@koivunej
remote_storage: move errors to error.rs
c03514c
@koivunej @arpad-m
refactor: combine enum variant arms
dfb5393 
Co-authored-by: arpad-m <[email protected]>
@koivunej @arpad-m
doc: no longer buffered writers
aa885d0 
Co-authored-by: arpad-m <[email protected]>
@koivunej
fix: delete_oids0: per request timeout
71b1991
@koivunej
cleanup: delete_oids0
a2eeb13 
I added this method in this PR because of permit issues: timetravel
already has a permit, but it was not propagated but I also did not want
to do an implementation which required no permit. in code review it was
noticed this additional hop was unnecessary and could be melt into
delete_oids.
@koivunej koivunej force-pushed the move_timeout_cancel3 branch from eb2449b to a2eeb13 Compare February 14, 2024 08:33
@koivunej
Copy link
Member Author

Latest force push got rid of the extra zst file I had accidentially added and missed in review mode.

@koivunej koivunej enabled auto-merge (squash) February 14, 2024 20:21
@koivunej koivunej merged commit 80854b9 into main Feb 14, 2024
49 checks passed
@koivunej koivunej deleted the move_timeout_cancel3 branch February 14, 2024 23:24
@koivunej koivunej mentioned this pull request Feb 20, 2024
Merged
koivunej added a commit that referenced this pull request Feb 21, 2024
@koivunej
fix: remaining missed cancellations and timeouts (#6843)
4146432 
As noticed in #6836 some occurances of error conversions were missed in
#6697:
- `std::io::Error` popped up by `tokio::io::copy_buf` containing
`DownloadError` was turned into `DownloadError::Other`
- similarly for secondary downloader errors

These changes come at the loss of pathname context.

Cc: #6096
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arpad-m arpad-m arpad-m approved these changes

@petuhovskiy petuhovskiy Awaiting requested review from petuhovskiy petuhovskiy was automatically assigned from neondatabase/safekeepers

@conradludgate conradludgate Awaiting requested review from conradludgate conradludgate is a code owner automatically assigned from neondatabase/proxy

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

pageserver: more elegant cancellation for remote operations
2 participants
@koivunej @arpad-m