Skip to content

Commit

Permalink
feat(pageserver): compute aux file size on initial logical size calcu…
Browse files Browse the repository at this point in the history
…lation (#7958)

close #7822
close #7443

Aux file metrics is computed incrementally. If the size is not
initialized, the metrics will never show up. This pull request adds the
functionality to compute the aux file size on initial logical size
calculation.

Signed-off-by: Alex Chi Z <chi@neon.tech>
  • Loading branch information
skyzh authored Jun 4, 2024
1 parent 3d6e389 commit 1a8d53a
Showing 3 changed files with 25 additions and 8 deletions.
3 changes: 2 additions & 1 deletion pageserver/src/aux_file.rs
Original file line number Diff line number Diff line change
@@ -178,7 +178,8 @@ impl AuxFileSizeEstimator {
}
}

pub fn on_base_backup(&self, new_size: usize) {
/// When generating base backup or doing initial logical size calculation
pub fn on_initial(&self, new_size: usize) {
let mut guard = self.size.lock().unwrap();
*guard = Some(new_size as isize);
self.report(new_size as isize);
14 changes: 13 additions & 1 deletion pageserver/src/pgdatadir_mapping.rs
Original file line number Diff line number Diff line change
@@ -718,10 +718,22 @@ impl Timeline {
result.insert(fname, content);
}
}
self.aux_file_size_estimator.on_base_backup(sz);
self.aux_file_size_estimator.on_initial(sz);
Ok(result)
}

pub(crate) async fn trigger_aux_file_size_computation(
&self,
lsn: Lsn,
ctx: &RequestContext,
) -> Result<(), PageReconstructError> {
let current_policy = self.last_aux_file_policy.load();
if let Some(AuxFilePolicy::V2) | Some(AuxFilePolicy::CrossValidation) = current_policy {
self.list_aux_files_v2(lsn, ctx).await?;
}
Ok(())
}

pub(crate) async fn list_aux_files(
&self,
lsn: Lsn,
16 changes: 10 additions & 6 deletions pageserver/src/tenant/timeline.rs
Original file line number Diff line number Diff line change
@@ -2787,17 +2787,21 @@ impl Timeline {
crate::metrics::initial_logical_size::START_CALCULATION.retry(circumstances)
};

match self_ref
let calculated_size = self_ref
.logical_size_calculation_task(
initial_part_end,
LogicalSizeCalculationCause::Initial,
background_ctx,
)
.await
{
Ok(calculated_size) => Ok((calculated_size, metrics_guard)),
Err(e) => Err(e),
}
.await?;

self_ref
.trigger_aux_file_size_computation(initial_part_end, background_ctx)
.await?;

// TODO: add aux file size to logical size

Ok((calculated_size, metrics_guard))
}
};

1 comment on commit 1a8d53a

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3268 tests run: 3116 passed, 0 failed, 152 skipped (full report)


Flaky tests (2)

Postgres 15

  • test_pageserver_restarts_under_worload: release
  • test_subscriber_restart: release

Code coverage* (full report)

  • functions: 31.6% (6606 of 20934 functions)
  • lines: 48.5% (51092 of 105257 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
1a8d53a at 2024-06-04T19:07:49.641Z :recycle:

Please sign in to comment.