Skip to content

Commit

Permalink
test_runner: use immediate shutdown in test_sharded_ingest (#9984)
Browse files Browse the repository at this point in the history
## Problem

`test_sharded_ingest` ingests a lot of data, which can cause shutdown to
be slow e.g. due to local "S3 uploads" or compactions. This can cause
test flakes during teardown.

Resolves #9740.

## Summary of changes

Perform an immediate shutdown of the cluster.
  • Loading branch information
erikgrinaker authored Dec 3, 2024
1 parent dcb24ce commit bbe4dfa
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions test_runner/performance/test_sharded_ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ def test_sharded_ingest(
# Start the endpoint.
endpoint = env.endpoints.create_start("main", tenant_id=tenant_id)
start_lsn = Lsn(endpoint.safe_psql("select pg_current_wal_lsn()")[0][0])

# Ingest data and measure WAL volume and duration.
with closing(endpoint.connect()) as conn:
with conn.cursor() as cur:
Expand All @@ -104,6 +105,8 @@ def test_sharded_ingest(
wait_for_last_flush_lsn(env, endpoint, tenant_id, timeline_id)

end_lsn = Lsn(endpoint.safe_psql("select pg_current_wal_lsn()")[0][0])

# Record metrics.
wal_written_mb = round((end_lsn - start_lsn) / (1024 * 1024))
zenbenchmark.record("wal_written", wal_written_mb, "MB", MetricReport.TEST_PARAM)

Expand Down Expand Up @@ -152,3 +155,7 @@ def test_sharded_ingest(
log.info(f"WAL ingested by each pageserver {ingested_by_ps}")

assert tenant_get_shards(env, tenant_id) == shards, "shards moved"

# The pageservers can take a long time to shut down gracefully, presumably due to the upload
# queue or compactions or something. Just stop them immediately, we don't care.
env.stop(immediate=True)

1 comment on commit bbe4dfa

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7144 tests run: 6826 passed, 0 failed, 318 skipped (full report)


Flaky tests (3)

Postgres 17

Postgres 14

Code coverage* (full report)

  • functions: 30.7% (8254 of 26897 functions)
  • lines: 47.7% (65160 of 136542 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
bbe4dfa at 2024-12-03T16:40:25.075Z :recycle:

Please sign in to comment.