feat(pageserver): persist aux file policy in index part

[skyzh]

Problem

Part of #7462

Summary of changes

Tenant config is not persisted unless it's attached on the storage controller. In this pull request, we persist the aux file policy flag in the index_part.json.

Admins can set switch_aux_file_policy in the storage controller or using the page server API. Upon the first aux file gets written, the write path will compare the aux file policy target with the current policy. If it is switch-able, we will do the switch. Otherwise, the original policy will be used. The test cases show what the admins can do / cannot do.

The last_aux_file_policy is stored in IndexPart. Updates to the persisted policy are done via schedule_index_upload_for_aux_file_policy_update. On the write path, the writer will update the field.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3096 tests run: 2969 passed, 0 failed, 127 skipped (full report)

---

Flaky tests (3)

Postgres 16

• test_vm_bit_clear_on_heap_lock: debug
• test_wal_restore: debug
• test_wal_restore_initdb: debug

Code coverage* (full report)

• functions: 31.6% (6388 of 20214 functions)
• lines: 47.9% (48654 of 101533 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
24a864a at 2024-05-17T19:30:46.315Z :recycle:}

[koivunej]

Pre-emptively for https://github.com/neondatabase/neon/pull/7668/files#r1596792568.

[skyzh]

Updated the pull request with storing the field in index part, and waiting for CI runs. The pull request body also gets updated correspondingly.

[skyzh]

I also took the test cases from #7781 and adapted to my proposed semantics of these flags.

[skyzh]

And to add, I think the main confusing in this PR is that we don't plan to support v1->v2 migration for customers. New customers go into cross validation mode first and go into v2 when we remove the read/write path of v1; existing customers will jump start from v2 and not being able to access their v1 replication state. v1->v2 migration is possible but it needs careful design, and with the persistent flag, it is still possible for us to do so, but I decided not to do that for now.

[skyzh]

added AuxFilePolicy::default_tenant_config instead of implementing Default to make the semantics clear given that sometimes None is the real default value.