release-6136

What's Changed

• tests: remove a leftover 'running' flag by @hlinnaka in #8216
• Restore running xacts from CLOG on replica startup by @knizhnik in #7288
• pageserver: add metric pageserver_secondary_resident_physical_size by @jcsp in #8204
• CI(build-tools): Remove libpq from build image by @bayandin in #8206
• fix(pageserver): include aux file in basebackup only once by @skyzh in #8207
• CI(gather-rust-build-stats): fix build with libpq by @bayandin in #8219
• tests: Make neon_xlogflush() flush all WAL, if you omit the LSN arg by @hlinnaka in #8215
• docs: Graceful storage controller cluster restarts RFC by @VladLazar in #7704
• Check status of connection after PQconnectStartParams by @knizhnik in #8210
• remote_storage config: move handling of empty inline table {} to callers by @problame in #8193
• pageserver: simpler, stricter config error handling by @jcsp in #8177
• fix: noisy logging when download gets cancelled during shutdown by @problame in #8224
• pageserver: rate limit log for loads of layers visited by @VladLazar in #8228
• Add support for reading and writing compressed blobs by @arpad-m in #8106
• L0 flush: opt-in mechanism to bypass PageCache reads and writes by @problame in #8190
• CI(benchmarking): move psql queries to actions/run-python-test-set by @bayandin in #8230
• tense of errors by @jcsp in #8234
• fix(pageserver): comments about metadata key range by @skyzh in #8236
• Add test for proper handling of connection failure to avoid 'cannot wait on socket event without a socket' error by @knizhnik in #8231
• docker: add storage_scrubber into the docker image by @skyzh in #8239
• Simplify test_wal_page_boundary_start test by @hlinnaka in #8214
• CI: update docker/* actions to latest versions by @bayandin in #7694
• pageserver: don't try to flush if shutdown during attach by @jcsp in #8235
• Only support compressed reads if the compression setting is present by @arpad-m in #8238
• add pagebench test cases for periodic pagebench on dedicated hardware by @Bodobolero in #8233
• pageserver: reduce ops tracked at per-timeline detail by @jcsp in #8245
• Fix outdated comment by @japinli in #8149
• page_service: stop exposing get_last_record_rlsn by @problame in #8244
• fix(pageserver): ensure test creates valid layer map by @skyzh in #8191
• pageserver: add supplementary branch usage stats by @jcsp in #8131
• tests: perform graceful rolling restarts in storcon scale test by @VladLazar in #8173
• proxy: cache certain non-retriable console errors for a short time by @conradludgate in #8201
• CI(build-and-test): add conclusion job by @bayandin in #8246
• pageserver: increase rate limit duration for layer visit log by @VladLazar in #8263
• Use bool param for round_trip_test_compressed by @arpad-m in #8252
• CI(pg-clients): unify workflow with build-and-test by @bayandin in #8160
• pageserver: downgrade stale generation messages to INFO by @jcsp in #8256
• Add find-large-objects subcommand to scrubber by @arpad-m in #8257
• feat(pageserver): integrate lsn lease into synthetic size by @yliang412 in #8220
• Flatten compression algorithm setting by @arpad-m in #8265
• implement rolling hyper-log-log algorithm by @knizhnik in #8068
• add checkout depth1 to workflow to access local github actions like generate allure report by @Bodobolero in #8259
• tests: extend allow list in deletion test by @jcsp in #8268
• correct error handling for periodic pagebench runner status by @Bodobolero in #8274
• tests: make location_conf_churn more robust by @jcsp in #8271
• safekeeper: add separate tombstones map for deleted timelines by @jcsp in #8253
• pageserver: add time based image layer creation check by @VladLazar in #8247
• SELECT 💣(); by @bayandin in #8270
• Increase timeout for wating subscriber caught-up by @knizhnik in #8118
• pageserver_live_connections: track as counter pair by @problame in #8227
• Improve parsing of ImageCompressionAlgorithm by @arpad-m in #8281
• Add concurrency to the find-large-objects scrubber subcommand by @arpad-m in #8291
• build(deps): bump certifi from 2023.7.22 to 2024.7.4 by @dependabot in #8301
• fix(pageserver): ensure sparse keyspace is ordered by @skyzh in #8285
• pageserver: switch to jemalloc by @jcsp in #8307
• Long-running replication tests by @save-buffer in #7896
• pageserver: respect has_relmap_file in collect_keyspace by @jcsp in #8276
• refactor: move part of sharding API from pageserver_api to utils by @yliang412 in #8254
• compute_ctl: Use 'fast' shutdown for Postgres termination by @ololobus in #8289
• tests: use smaller layers in test_pg_regress, do compaction after tests by @jcsp in #8232
• fix(pageserver): write to both v1+v2 for aux tenant import by @skyzh in #8316
• tests: stabilize test_timeline_size_quota_on_startup by @jcsp in #8255
• tests: increase approx size equal threshold to avoid test_lsn_lease_size flakiness by @yliang412 in #8282
• CI(promote-compatibility-data): take into account commit sha by @bayandin in #8283
• proxy/http: avoid spurious vector reallocations by @lucab in #8319
• chore(storage-scrubber): allow disable file logging by @skyzh in #8297
• chore: fix nightly build by @conradludgate in #8142
• fix(storage-scrubber): use default AWS authentication by @skyzh in #8299
• Refactor how migrations are ran by @tristan957 in #8326
• refactor: postgres_backend: replace abstract shutdown_watcher with CancellationToken by @problame in #8295
• fix(l0_flush): drops permit before fsync, potential cause for OOMs by @problame in #8327
• pageserver: move page_service's import basebackup / import wal to mgmt API by @problame in #8292
• proxy: pg17 fixes by @conradludgate in #8321
• tests: stabilize test_sharding_split_compaction by @jcsp in #8318
• proxy: remove some trace logs by @conradludgate in #8334
• build(deps-dev): bump zipp from 3.8.1 to 3.19.1 by @dependabot in #8329
• Remove ImageCompressionAlgorithm::DisabledNoDecompress by @arpad-m in #8300
• feat(pageserver): add k-merge layer iterator with lazy loading by @skyzh in #8053
• safekeeper: add eviction_min_resident to stop evictions thrashing by @jcsp in #8335
• Enable core dumps for postgres by @kelvich in #8272
• rfcs: add RFC for timeline archival by @jcsp in #8221
• Run Performance bench on more platforms by @Bodobolero in #8312
• pageserver: remove trace_read_requests by @problame in #8338
• storage_controller: fix ReconcilerWaiter::get_status by @VladLazar in #8341
• safekeeper: eviction metrics by @jcsp in #8348
• storage controller: add node deletion API by @jcsp in #8226
• Fix memory context of NeonWALReader allocation. by @arssher in #8360
• feat(pageserver): rewrite streaming vectored read planner by @skyzh in #8242
• test: limit test_layer_download_timeouted to MOCK_S3 by @koivunej in #8331
• Grant execute on snapshot functions to neon_superuser by @save-buffer in #8346
• Pass configured compression param to image generation by @arpad-m in #8363
• Implement decompression for vectored reads by @arpad-m in #8302
• Remove fs2 dependency by @japinli in #8350
• pageserver: circuit breaker on compaction by @jcsp in #8359
• Fix nightly warnings 2024 june by @conradludgate in #8151
• fix(pageserver): unique test harness name for merge_in_between by @skyzh in #8366
• tests: allow list breaching min resident size in statvfs test by @VladLazar in #8358
• proxy/http: switch to typed_json by @lucab in #8377
• Fix test_timeline_copy flakiness. by @arssher in #8356
• Add neon.running_xacts_overflow_policy to make it possible for RO replica to startup without primary even in case running xacts overflow by @knizhnik in #8323
• remove page_service show <tenant_id> by @problame in #8372
• feat(storcon): timeline detach ancestor passthrough by @koivunej in #8353
• pageserver: use PITR GC cutoffs as authoritative by @jcsp in #8365
• feat(timeline_detach_ancestor): success idempotency by @koivunej in #8354
• Allow reusing projects between runs of logical replication benchmarks by @save-buffer in #8393
• Allow the new clippy::doc_lazy_continuation lint by @arpad-m in #8388
• pageserver: un-Arc Timeline::layers by @jcsp in #8386
• Compute: add compatibility patch for rum by @lubennikovaav in #8381
• Avoid the storage controller in test_tenant_creation_fails by @arpad-m in #8392
• build(deps): bump setuptools from 65.5.1 to 70.0.0 by @dependabot in #8387
• pageserver: clean up GcCutoffs names by @jcsp in #8379
• Run each migration in its own transaction by @tristan957 in #8371
• fix(pageserver): limit num of delta layers for l0 compaction by @skyzh in #8391
• pageserver: exclude un-read layers from short residence statistic by @jcsp in #8396
• pageserver: reduce size of delta layer ValueRef by @jcsp in #8401
• docs/rfcs: timeline ancestor detach API by @jcsp in #6888
• Bodobolero/pgbench compare azure by @Bodobolero in #8409
• test(pageserver): more k-merge tests on duplicated keys by @skyzh in #8404
• tests: add test_compaction_l0_memory by @jcsp in #8403
• docs: update storage controller db name in doc by @yliang412 in #8411
• pageserver: layer count & size metrics by @jcsp in #8410
• test: allow requests to any pageserver get cancelled by @koivunej in #8413
• Fix comment in Control Plane by @dotdister in #8406
• NeonEnv.from_repo_dir: use storage_controller_db instead of attachments.json by @problame in #8382
• tests: fix metrics check in test_s3_eviction by @jcsp in #8419
• tests: increase test_pg_regress and test_isolation timeouts by @jcsp in #8418
• tests: turn on safekeeper eviction by default by @jcsp in #8352
• Fix partial upload bug with invalid remote state by @petuhovskiy in #8383
• pageserver: integrate k-merge with bottom-most compaction by @skyzh in #8415
• Temporarily use vanilla pgbench and psql (client) for running pgvector benchmark by @Bodobolero in #8422
• Change log level for GuardDrop error by @petuhovskiy in #8305
• Enable zstd in tests by @arpad-m in #8368
• Bodobolero/fix root permissions by @Bodobolero in #8429
• pageserver: remove obsolete cached_metric_collection_interval by @problame in #8370
• storage scrubber: GC ancestor shard layers by @jcsp in #8196
• Remove unused safekeeper runtimes by @shinyaaa in #8433
• tests: longer timeouts in test_timeline_deletion_with_files_stuck_in_upload_queue by @jcsp in #8438
• Fix openapi specification by @shinyaaa in #8273
• Add archival_config endpoint to pageserver by @arpad-m in #8414
• Use DefaultCredentialsChain AWS authentication in remote_storage by @arpad-m in #8440
• Storage & Compute release 2024-07-22 by @vipvap in #8451

New Contributors

• @japinli made their first contribution in #8149
• @lucab made their first contribution in #8319
• @dotdister made their first contribution in #8406
• @shinyaaa made their first contribution in #8433

Full Changelog: release-5940...release-6136