feat(test_runner): allowed_errors in storage scrubber

[skyzh]

Problem

resolve #9988 (comment)

Summary of changes

• New verbose mode for storage scrubber scan metadata (pageserver) that contains the error messages.
• Filter allowed_error list from the JSON output to determine the healthy flag status.

[github-actions]

7062 tests run: 6747 passed, 0 failed, 315 skipped (full report)

---

Flaky tests (4)

Postgres 17

• test_check_visibility_map: debug-x86-64

Postgres 15

• test_pageserver_gc_compaction_smoke: release-x86-64

Postgres 14

• test_check_visibility_map: release-arm64
• test_pull_timeline[True]: release-x86-64

Code coverage* (full report)

• functions: 31.4% (8333 of 26537 functions)
• lines: 47.7% (65614 of 137569 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
3ee2bd3 at 2024-12-10T16:26:30.108Z :recycle:}

[skyzh]

Next step: figure out which test cases would produce orphan layers and allow such errors for these test cases. We can also choose to add it to default allowed errors.

[yliang412]

LGTM. I think we discussed the disable_scrub_on_exit method in the meeting. I think currently it is used for two cases:

1. We deleted the only tenant, and the scrubber fails if it detects nothing.
2. Immediate shutdown might left orphan layers.

As a next step, we potentially could add those to structured allowed errors and remove disable_scrub_on_exit if that's desired. I don't have strong opinion on this.