Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework blobovnicza tree #2453

Closed
roman-khimov opened this issue Jul 24, 2023 · 10 comments
Closed

Rework blobovnicza tree #2453

roman-khimov opened this issue Jul 24, 2023 · 10 comments
Assignees
@cthulhu-rider
Labels
neofs-storage Storage node application issues
Milestone
v0.38.0

Comments

@roman-khimov
Copy link
Member

Is your feature request related to a problem? Please describe.

#2382, current bbcz tree is needlessly complex and fragile.

Describe the solution you'd like

One DB per shard is sufficient.

Describe alternatives you've considered

Keeping some number of them doesn't seem to be helpful.

Additional context

We need migration.
@roman-khimov roman-khimov added neofs-storage Storage node application issues refactor labels Jul 24, 2023
@roman-khimov roman-khimov added this to the v0.38.0 milestone Jul 24, 2023
@carpawell
Copy link
Member

carpawell commented Jul 24, 2023 •
edited
Loading

One DB per shard is sufficient.

How about one layer of DBs? Can it be more optimal for storing small objects? No opened DBs caches and no complex logic at all but just one layer.

@roman-khimov
Copy link
Member Author

Let's microbenchmark this using current structure:

  • N depth, M width
  • 1 depth, M width
  • 1 DB

The expectation is to get about the same number of operations per second, then a single DB case can be improved by removing the tree.

@roman-khimov
Copy link
Member Author

BTW, speaking of names, blobovnicza is not really appropriate here as these are not BLOBs. They're more of a BSOB or BMOB.

@carpawell carpawell mentioned this issue Jul 24, 2023
Merged
@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Jul 25, 2023 •
edited
Loading

speaking of names

blobovnicza should be definitely replaced imo. I thought about new one a bit: "bolt with tiny objects" - boltin.

This was referenced Jul 25, 2023
Merged
Merged
@roman-khimov
Copy link
Member Author

boltiny: 👍
peapod: 🎉
pisum: 🚀

@carpawell
Copy link
Member

IMO, naming should not depend on a certain DB name.

@cthulhu-rider cthulhu-rider mentioned this issue Jul 29, 2023
Merged
3 tasks
@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Jul 29, 2023 •
edited
Loading

pisum is ok to me, but ceteris paribus i'd select any other one

@cthulhu-rider cthulhu-rider mentioned this issue Aug 1, 2023
Merged
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
Introduce new storage component Peapod
8dfbd8c 
Currently, storage node saves relatively small NeoFS objects in
Blobovnicza tree component: group of BoltDB wrappers managed as a tree.
This component has pretty complex data structure, code implementation
and dubious performance results.

Peapod is a new storage component introduced to replace Blobovnicza one
as more simple and effective. It also bases on single BoltDB instance,
but organizes batch writes in a specific way. In future, Peapod is going
to be used as a storage of small objects by the BlobStor.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
Introduce tool to migrate objects from Blobovnicza tree to Peapod
ae76dad 
Add `cmd/blobovnicza-to-peapod` application which accepts YAML
configuration file of the storage node and, for each configured shard,
overtakes data from Blobovnicza tree to Peapod created in the parent
directory.

The tool is going to be used for phased and safe rejection of the
Blobovnicza trees and the transition to Peapods.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
d2d12b5 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
a8e41ff 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
Introduce tool to migrate objects from Blobovnicza tree to Peapod
327fb86 
Add `cmd/blobovnicza-to-peapod` application which accepts YAML
configuration file of the storage node and, for each configured shard,
overtakes data from Blobovnicza tree to Peapod created in the parent
directory.

The tool is going to be used for phased and safe rejection of the
Blobovnicza trees and the transition to Peapods.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
f355306 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
1ac5ee0 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 11, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
f4e2512 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 15, 2023
@cthulhu-rider
Introduce new storage component Peapod
b28c1f2 
Currently, storage node saves relatively small NeoFS objects in
Blobovnicza tree component: group of BoltDB wrappers managed as a tree.
This component has pretty complex data structure, code implementation
and dubious performance results.

Peapod is a new storage component introduced to replace Blobovnicza one
as more simple and effective. It also bases on single BoltDB instance,
but organizes batch writes in a specific way. In future, Peapod is going
to be used as a storage of small objects by the BlobStor.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 15, 2023
@cthulhu-rider
Introduce tool to migrate objects from Blobovnicza tree to Peapod
f475cca 
Add `cmd/blobovnicza-to-peapod` application which accepts YAML
configuration file of the storage node and, for each configured shard,
overtakes data from Blobovnicza tree to Peapod created in the parent
directory.

The tool is going to be used for phased and safe rejection of the
Blobovnicza trees and the transition to Peapods.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 15, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
7af737f 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 15, 2023
@cthulhu-rider
Introduce tool to migrate objects from Blobovnicza tree to Peapod
5ed4d42 
Add `cmd/blobovnicza-to-peapod` application which accepts YAML
configuration file of the storage node and, for each configured shard,
overtakes data from Blobovnicza tree to Peapod created in the parent
directory.

The tool is going to be used for phased and safe rejection of the
Blobovnicza trees and the transition to Peapods.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 15, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
29dafda 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 15, 2023
@cthulhu-rider
Introduce tool to migrate objects from Blobovnicza tree to Peapod
e36ad52 
Add `cmd/blobovnicza-to-peapod` application which accepts YAML
configuration file of the storage node and, for each configured shard,
overtakes data from Blobovnicza tree to Peapod created in the parent
directory.

The tool is going to be used for phased and safe rejection of the
Blobovnicza trees and the transition to Peapods.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Aug 15, 2023
@cthulhu-rider
node: Support Peapod as BlobStor's sub-storage
c060b16 
Support `peapod` sub-storage type in BlobStor configuration.

Refs nspcc-dev#2453.

Signed-off-by: Leonard Lyubich <[email protected]>
@roman-khimov
Copy link
Member Author

Fixed in #2462, but not closed.

This was referenced Aug 16, 2023
Closed
Closed
Closed
Closed
@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Aug 16, 2023 •
edited
Loading

Fixed in #2462, but not closed.

do we need anything else within this issue?

@roman-khimov
Copy link
Member Author

It's done to me. We will drop bbcz eventually and we will optimize peapod additionally, we may simplify configurations, but all of those are different things.

@roman-khimov roman-khimov mentioned this issue Dec 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cthulhu-rider cthulhu-rider

Labels
neofs-storage Storage node application issues
Projects
None yet
Milestone
v0.38.0
Development

No branches or pull requests
3 participants
@roman-khimov @carpawell @cthulhu-rider