Use Badger4 as datastore #20
Conversation
Badger4 with optimize settings for gateways, including writethrough Blockservice/Store. Badger4 is designed to work with SSDs (Value log requires fast random lookup speed). I think we should try to put the datastore on SSDs and a very large http cache on spinning disk (if available). TODO: figure out GC (later).
| github.com/ipfs/boxo v0.13.1 | ||
| github.com/ipfs/go-cid v0.4.1 | ||
| github.com/ipfs/go-datastore v0.6.0 | ||
| github.com/ipfs/go-ds-flatfs v0.5.1 | ||
| github.com/ipfs/go-ds-badger4 v0.0.0-20231006150127-9137bcc6b981 |
There was a problem hiding this comment.
Pending a tag when ipfs/github-mgmt#174 resolved.
| &cli.IntFlag{ | ||
| Name: "inmem-block-cache", | ||
| Value: 1 << 30, | ||
| Usage: "Size of the in-memory block cache. 0 to disable (disables compression too)", | ||
| }, |
There was a problem hiding this comment.
Total memory consumption would be something like block-cache + memtables + indexes + bloom filters + LSM.
Now, I am not sure how much of the LSM lives in memory in badger. They seem to suggest that the LSM can live in memory, and in V0 LSM's open option was set to MemoryMap, so very probably it depends on the OS how many memory pages it keeps live.
There was a problem hiding this comment.
Seems fine by me if you'd like to merge it although given that a lot of knobs were tweaked within the code configuration here (which seem pretty reasonable) idk if you either want to expose more options for testing or run the branch through testing (e.g. thunderdome, staging gateway, etc.) before you decide you want to hit the merge button.
Going to merge. Until the code arrives to a point where it is "ready" to deploy to staging or similar, no need to have multiple things half done waiting for testing. |
Badger4 with optimize settings for gateways, including writethrough Blockservice/Store.
Badger4 is designed to work with SSDs (Value log requires fast random lookup speed). I think we should try to put the datastore on SSDs and a very large http cache on spinning disk (if available).
TODO: figure out GC (later).