Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: use high IO concurrency in test_pgdata_import_smoke, use effective_io_concurrency=2 in tests by default #10114

Merged
merged 4 commits into from
Dec 19, 2024

Conversation

jcsp
Copy link
Collaborator

@jcsp jcsp commented Dec 12, 2024

Problem

test_pgdata_import_smoke writes two gigabytes of pages and then reads them back serially. This is CPU bottlenecked and results in a long runtime, and sensitivity to CPU load from other tests on the same machine.

Closes: #10071

Summary of changes

  • Use effective_io_concurrency=32 when doing sequential scans through 2GiB of pages in test_pgdata_import_smoke. This is a ~10x runtime decrease in the parts of the test that do sequential scans.
  • Also set effective_io_concurrency=2 for tests, as I noticed while debugging that we were doing all getpage requests serially, which is bad for checking the stability of the batching code.

@jcsp jcsp added c/storage/pageserver Component: storage: pageserver a/test Area: related to testing a/tech_debt Area: related to tech debt labels Dec 12, 2024
@jcsp jcsp changed the title Jcsp/issue 10071 import test timeout tests: less data in test_pgdata_import_smoke, use effective_io_concurrency=2 in tests by default Dec 12, 2024
Copy link

github-actions bot commented Dec 12, 2024

7095 tests run: 6798 passed, 0 failed, 297 skipped (full report)


Flaky tests (3)

Postgres 17

  • test_physical_replication_config_mismatch_too_many_known_xids: debug-x86-64

Postgres 16

Postgres 14

Code coverage* (full report)

  • functions: 31.2% (8392 of 26865 functions)
  • lines: 48.0% (66633 of 138945 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
5ae4824 at 2024-12-19T10:54:30.808Z :recycle:

@jcsp jcsp force-pushed the jcsp/issue-10071-import-test-timeout branch from c288c95 to d0be9b2 Compare December 12, 2024 18:10
@jcsp jcsp changed the title tests: less data in test_pgdata_import_smoke, use effective_io_concurrency=2 in tests by default tests: use high IO concurrency in test_pgdata_import_smoke, use effective_io_concurrency=2 in tests by default Dec 12, 2024
@jcsp jcsp marked this pull request as ready for review December 17, 2024 13:36
@jcsp jcsp requested a review from problame December 17, 2024 13:36
@jcsp jcsp enabled auto-merge December 19, 2024 09:50
@jcsp jcsp added this pull request to the merge queue Dec 19, 2024
Merged via the queue into main with commit 65042cb Dec 19, 2024
82 checks passed
@jcsp jcsp deleted the jcsp/issue-10071-import-test-timeout branch December 19, 2024 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a/tech_debt Area: related to tech debt a/test Area: related to testing c/storage/pageserver Component: storage: pageserver
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test_pgdata_import_smoke timeouts
2 participants