PebbleDB #281
Closed
PebbleDB #281
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: M. J. Fromberger <[email protected]>
Co-authored-by: M. J. Fromberger <[email protected]>
…tr.isInvalid were considered to be ineffective assignments by the linter.
|
|
4 tasks
github-merge-queue bot
pushed a commit
to cometbft/cometbft
that referenced
this pull request
Jan 29, 2024
This PR integrates pebbledb, a key-value store by Cockroachdb based on Cockroach Labs findings about cgo: * https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/ Unlike any other kv store that cometbft currently supports: * pebble is actively maintained unlike: * goleveldb * boltdb * badgerdb * pebble does not require the use of CGO like: * cleveldb * rocksdb In benchmarks: * pebble performs better than goleveldb, cleveldb and rocksdb * pebble performs consistently At Notional Ventures, Pte, we've used and submitted pebble for two years for: * archive nodes that would be crushed if they used goleveldb * high performance RPC infrastructure Here are the hitstorical pull requests to merge pebble: * tendermint/tm-db#230 * tendermint/tm-db#231 * tendermint/tm-db#281 * tendermint/tm-db#282 * tendermint/tm-db#283 * tendermint/tm-db#284 * tendermint/tm-db#304 * tendermint/tm-db#321 Pebble snapshots for cosmos blockchains are available here: * https://snapshot.notional.ventures/ --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [x] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <[email protected]>
mergify bot
pushed a commit
to cometbft/cometbft
that referenced
this pull request
Jan 29, 2024
This PR integrates pebbledb, a key-value store by Cockroachdb based on Cockroach Labs findings about cgo: * https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/ Unlike any other kv store that cometbft currently supports: * pebble is actively maintained unlike: * goleveldb * boltdb * badgerdb * pebble does not require the use of CGO like: * cleveldb * rocksdb In benchmarks: * pebble performs better than goleveldb, cleveldb and rocksdb * pebble performs consistently At Notional Ventures, Pte, we've used and submitted pebble for two years for: * archive nodes that would be crushed if they used goleveldb * high performance RPC infrastructure Here are the hitstorical pull requests to merge pebble: * tendermint/tm-db#230 * tendermint/tm-db#231 * tendermint/tm-db#281 * tendermint/tm-db#282 * tendermint/tm-db#283 * tendermint/tm-db#284 * tendermint/tm-db#304 * tendermint/tm-db#321 Pebble snapshots for cosmos blockchains are available here: * https://snapshot.notional.ventures/ --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [x] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <[email protected]> (cherry picked from commit 82fa3c0)
melekes
added a commit
to cometbft/cometbft
that referenced
this pull request
Jan 29, 2024
This PR integrates pebbledb, a key-value store by Cockroachdb based on Cockroach Labs findings about cgo: * https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/ Unlike any other kv store that cometbft currently supports: * pebble is actively maintained unlike: * goleveldb * boltdb * badgerdb * pebble does not require the use of CGO like: * cleveldb * rocksdb In benchmarks: * pebble performs better than goleveldb, cleveldb and rocksdb * pebble performs consistently At Notional Ventures, Pte, we've used and submitted pebble for two years for: * archive nodes that would be crushed if they used goleveldb * high performance RPC infrastructure Here are the hitstorical pull requests to merge pebble: * tendermint/tm-db#230 * tendermint/tm-db#231 * tendermint/tm-db#281 * tendermint/tm-db#282 * tendermint/tm-db#283 * tendermint/tm-db#284 * tendermint/tm-db#304 * tendermint/tm-db#321 Pebble snapshots for cosmos blockchains are available here: * https://snapshot.notional.ventures/ --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [x] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pebble
The problem
There is some kind of issue with pebble's iterator-- I think it is in our implementation, but I don't know.
fix: remove mutex in prefixdb #239 contains additional tests written by Khanh and Vuong. I've merged that here in hopes of finding the issue.
I've made a CI setup that runs the tests on pebble 700,000 times in 35 parallel jobs with 5000 rounds each because the specific issue here is that nodes with pebble will run for some time, then die, crying about a type error.