fix(consumption_metrics): exp backoff retry

[koivunej]

Split off from #5297. Depends on #5315.
Cc: #5175 for retry

[github-actions]

2484 tests run: 2362 passed, 0 failed, 122 skipped (full report)

---

Flaky tests (6)

Postgres 16

• test_partial_evict_tenant: debug, release
• test_get_tenant_size_with_multiple_branches: release

Postgres 14

• test_download_remote_layers_api[local_fs]: debug
• test_get_tenant_size_with_multiple_branches: release, debug

Code coverage (full report)

• functions: 53.1% (7697 of 14507 functions)
• lines: 81.0% (44971 of 55535 lines)

---

_{The comment gets automatically updated with the latest test results
3a75580 at 2023-09-15T22:18:09.185Z :recycle:}

[arpad-m]

Test failure looks legit:

__________________ test_threshold_based_eviction[debug-pg14] ___________________
[gw9] linux -- Python 3.9.2 /github/home/.cache/pypoetry/virtualenvs/neon-_pxWMzVK-py3.9/bin/python
test_runner/regress/test_threshold_based_eviction.py:180: in test_threshold_based_eviction
    assert env.pageserver.log_contains(
E   AssertionError: ensure the metrics collection worker ran
E   assert None

[arpad-m]

This one I saw multiple times:

[gw2] linux -- Python 3.9.2 /github/home/.cache/pypoetry/virtualenvs/neon-_pxWMzVK-py3.9/bin/python
test_runner/regress/test_threshold_based_eviction.py:180: in test_threshold_based_eviction
    assert env.pageserver.log_contains(
E   AssertionError: ensure the metrics collection worker ran
E   assert None

Even before your force push, all the regression test jobs failed. now they did it again. This is not how flukes look like.

[koivunej]

Thanks, I must've forgotten to cherry-pick some.

No, I did remember it (was already before latest force-push): 7fdaff8. Aah, the middle span is not yet in place...

[koivunej]

test_download_remote_layers_api[debug-pg14-local_fs]
test_partial_evict_tenant[release-pg16]

oh nice, it's partial_evict_tenant again. Trying the first one over at #5322.