Pebble #284
Closed
Pebble #284
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit eeadeb8.
faddat
requested review from
alexanderbez,
cmwaters,
creachadair,
tac0turtle,
tychoish and
williambanfield
as code owners
|
Passes tests because merged #284 |
tac0turtle
reviewed
Jul 31, 2022
| BadgerDBBackend BackendType = "badgerdb" | ||
| // PebbleDBDBBackend represents pebble (uses github.com/cockroachdb/pebble) | ||
| // - EXPERIMENTAL | ||
| // - use pebble build tag (go build -tags pebbledb) |
There was a problem hiding this comment.
I believe this is pure go so no need for a build flag.
There was a problem hiding this comment.
We tag all of them currently (badger, bolt, and pebble)
There was a problem hiding this comment.
I think that if we don't tag, then we end up making chain binaries that use more than one db. This could (or might not) be desirable....
There was a problem hiding this comment.
if we want to make it default, then no flag. But I think default is like maybe next month.
Cause it is 10-27x less intense on the disk, no joke.
There was a problem hiding this comment.
There was a problem hiding this comment.
eh, need to do more benchmarking. its.... wierd.
But:
- near certain drives will last longer
- near certain will be faster with cw
- near certain will enable running on much cheaper nodes
There was a problem hiding this comment.
I took the tags off. Not sure what'll happen with tests and such
5 tasks
enable cache
Codecov Report
@@ Coverage Diff @@
## master #284 +/- ##
==========================================
+ Coverage 68.30% 69.19% +0.88%
==========================================
Files 27 28 +1
Lines 2073 2350 +277
==========================================
+ Hits 1416 1626 +210
- Misses 586 641 +55
- Partials 71 83 +12
|
|
@faddat how has pebbledb performance been compared to rocks? I am very much ready to ditch CGO. Any benchmarks yet? |
Check notional org repos, cosmodia was it I think, they testing it in prod. right now on 40 chains or something |
4 tasks
github-merge-queue bot
pushed a commit
to cometbft/cometbft
that referenced
this pull request
Jan 29, 2024
This PR integrates pebbledb, a key-value store by Cockroachdb based on Cockroach Labs findings about cgo: * https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/ Unlike any other kv store that cometbft currently supports: * pebble is actively maintained unlike: * goleveldb * boltdb * badgerdb * pebble does not require the use of CGO like: * cleveldb * rocksdb In benchmarks: * pebble performs better than goleveldb, cleveldb and rocksdb * pebble performs consistently At Notional Ventures, Pte, we've used and submitted pebble for two years for: * archive nodes that would be crushed if they used goleveldb * high performance RPC infrastructure Here are the hitstorical pull requests to merge pebble: * tendermint/tm-db#230 * tendermint/tm-db#231 * tendermint/tm-db#281 * tendermint/tm-db#282 * tendermint/tm-db#283 * tendermint/tm-db#284 * tendermint/tm-db#304 * tendermint/tm-db#321 Pebble snapshots for cosmos blockchains are available here: * https://snapshot.notional.ventures/ --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [x] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <[email protected]>
mergify bot
pushed a commit
to cometbft/cometbft
that referenced
this pull request
Jan 29, 2024
This PR integrates pebbledb, a key-value store by Cockroachdb based on Cockroach Labs findings about cgo: * https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/ Unlike any other kv store that cometbft currently supports: * pebble is actively maintained unlike: * goleveldb * boltdb * badgerdb * pebble does not require the use of CGO like: * cleveldb * rocksdb In benchmarks: * pebble performs better than goleveldb, cleveldb and rocksdb * pebble performs consistently At Notional Ventures, Pte, we've used and submitted pebble for two years for: * archive nodes that would be crushed if they used goleveldb * high performance RPC infrastructure Here are the hitstorical pull requests to merge pebble: * tendermint/tm-db#230 * tendermint/tm-db#231 * tendermint/tm-db#281 * tendermint/tm-db#282 * tendermint/tm-db#283 * tendermint/tm-db#284 * tendermint/tm-db#304 * tendermint/tm-db#321 Pebble snapshots for cosmos blockchains are available here: * https://snapshot.notional.ventures/ --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [x] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <[email protected]> (cherry picked from commit 82fa3c0)
melekes
added a commit
to cometbft/cometbft
that referenced
this pull request
Jan 29, 2024
This PR integrates pebbledb, a key-value store by Cockroachdb based on Cockroach Labs findings about cgo: * https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/ Unlike any other kv store that cometbft currently supports: * pebble is actively maintained unlike: * goleveldb * boltdb * badgerdb * pebble does not require the use of CGO like: * cleveldb * rocksdb In benchmarks: * pebble performs better than goleveldb, cleveldb and rocksdb * pebble performs consistently At Notional Ventures, Pte, we've used and submitted pebble for two years for: * archive nodes that would be crushed if they used goleveldb * high performance RPC infrastructure Here are the hitstorical pull requests to merge pebble: * tendermint/tm-db#230 * tendermint/tm-db#231 * tendermint/tm-db#281 * tendermint/tm-db#282 * tendermint/tm-db#283 * tendermint/tm-db#284 * tendermint/tm-db#304 * tendermint/tm-db#321 Pebble snapshots for cosmos blockchains are available here: * https://snapshot.notional.ventures/ --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [x] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch includes work from myself, @catShaark, and @baabeetaa. @baabeetaa solved the final issues in the pebble implementation for tm-db.
It's been running on Sifchain for about 15 hours.
We will soon have pebble snapshots for every chain that Notional validates available at:
https://github.com/notionla-labs/cosmosia
But we didn't just test this against Sif, that has led to failures in the past, so instead, we tested it millions, and millions of times, against both the tm-db test suite and iavl.
Like the other databases, pebble would perform faster if this pr:
were merged.
Like all the PR's here, this pr would pass tests if this pr:
were merged -- it fixes the master branch of this repository.