test: test_download_remote_layers_api again

[koivunej]

The test is still flaky, perhaps more after #5233, see #3831.

Do one more timeline_checkpoint after shutting down safekeepers before shutting down pageserver. Put more effort into not compacting or creating image layers.

[github-actions]

2484 tests run: 2364 passed, 0 failed, 120 skipped (full report)

---

Flaky tests (4)

Postgres 16

• test_partial_evict_tenant: debug
• test_delete_tenant_exercise_crash_safety_failpoints[Check.RETRY_WITH_RESTART-mock_s3-tenant-delete-before-remove-timelines-dir-True]: debug
• test_wal_lagging: release

Postgres 14

• test_get_tenant_size_with_multiple_branches: debug

Code coverage (full report)

• functions: 53.0% (7751 of 14613 functions)
• lines: 81.1% (45263 of 55836 lines)

---

_{The comment gets automatically updated with the latest test results
e249c8a at 2023-09-16T15:25:54.916Z :recycle:}

[koivunej]

Regress test re-run attempts:

1. test_partial_evict_tenant[debug-pg16]
2. only error from 6e132d7, perhaps this worked?
3. same, with partial_evict_tenant 😱
4. same, with test_get_tenant_size_with_multiple_branches which is flaky as debug (?)
5. same as 3rd

Post-revert:

• pg16 other known flaky
• pg16 other known flaky
• pg16 other known flaky

[knizhnik]

May be it is better to introduce PS parameter for delaying logical size computation?
Itmay be useful not only for tests...

[koivunej]

I don't think so. Problem here is not the fact that safekeeper communication ends up asking logical size, that's what the failpoint is for, set as environment variable when starting up pageserver.

Problem was that new WAL was being received in apparently when there's other load on the test runners leading to flakyness of the test, and my previous "fix" made this more visible. Or alternatively compaction was going on, I didn't really dig too deep because I was doing work.

I should look at the comments because as noted in slack, at best they are distracting.

[koivunej]

Cleaned the test up in e249c8a not to allude to safekeepers launching logical size calculation.

[koivunej]

Checked the example I linked here #3831 (comment):

EDIT: Log analysis later on: logs clearly show inmemory flush after metrics were read during shutdown

[koivunej]

One flaky test per re-run..