Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

forky: Fixed Chunk Data Size Store #2017

Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a2d5edc
storage/localstore: integrate fcds
janos Dec 5, 2019
2506b88
storage/{fcds,localstore}: add comments and minor adjustments
janos Dec 6, 2019
c8f3622
storage/fcds: add doc.go
janos Dec 6, 2019
ec14da3
cmd/swarm, storage/localstore: support breaking migrations
janos Dec 9, 2019
e352b88
cmd/swarm, storage/localstore: improve migrations, export and import
janos Dec 9, 2019
c8c16e1
storage/localstore: export pins
janos Dec 9, 2019
66aa7fd
Merge branch 'master' into fcds
janos Dec 9, 2019
7aed3b1
storage/{fcds,localstore}: address Viktor's comments
janos Dec 12, 2019
26bcb48
storage/fcds: correctly return explicit nil in getOffset
janos Dec 13, 2019
1e94680
storage/fcds: add WithCache optional argument to New constructor
janos Dec 13, 2019
db658c7
storage/fcds: address most of Petar's comments
janos Dec 18, 2019
26f6626
storage/fcds: add offsetCache ttl
janos Dec 18, 2019
5be3c25
Revert "storage/fcds: add offsetCache ttl"
janos Dec 18, 2019
c222011
Merge branch 'master' into fcds
janos Dec 19, 2019
661a7f5
storage/fcds: rename fcds.Interface to fcds.Storer
janos Jan 14, 2020
f51c6d8
storage/fcds: improve some commenting
janos Jan 14, 2020
91fc21f
storage/localstore: improve comment in the Import method
janos Jan 14, 2020
60e3938
storage/localstore: improve migrateDiwali migration message
janos Jan 14, 2020
c542f32
storage/fcds: ensure that chunk data is no longer the the max value
janos Jan 14, 2020
0fc5e3a
storage/localstore: terminate import goroutine in case of errors
janos Jan 14, 2020
842f7d8
storage/localstore: do not put existing chunks
janos Mar 5, 2020
d9341b8
storage/fcds/test: correctly handle storage path
janos Mar 5, 2020
0f13d3b
strage/fcds: check if chunk exists before it is put
janos Mar 5, 2020
4b6f726
storage/fcds: add and use MetaStore.Has
janos Mar 5, 2020
39d328a
storage/fcds: optimize locking
janos Mar 6, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions cmd/swarm/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ import (
"fmt"
"io"
"os"
"path/filepath"

"github.com/ethereum/go-ethereum/cmd/utils"
"github.com/ethereum/go-ethereum/common"
Expand Down Expand Up @@ -168,10 +167,6 @@ func dbImport(ctx *cli.Context) {
}

func openLDBStore(path string, basekey []byte) (*localstore.DB, error) {
if _, err := os.Stat(filepath.Join(path, "CURRENT")); err != nil {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is removed in order to allow dbImport to create localstore directory if it does not exist, to simplify migration steps (remove the need to start swarm node in between export and import, as it was required in v0.5.0 release).

return nil, fmt.Errorf("invalid chunkdb path: %s", err)
}

return localstore.New(path, basekey, nil)
}

Expand Down
13 changes: 12 additions & 1 deletion cmd/swarm/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package main
import (
"crypto/ecdsa"
"encoding/hex"
"errors"
"fmt"
"io/ioutil"
"net"
Expand Down Expand Up @@ -46,6 +47,7 @@ import (
"github.com/ethersphere/swarm/internal/debug"
swarmmetrics "github.com/ethersphere/swarm/metrics"
"github.com/ethersphere/swarm/network"
"github.com/ethersphere/swarm/storage/localstore"
"github.com/ethersphere/swarm/storage/mock"
mockrpc "github.com/ethersphere/swarm/storage/mock/rpc"
"github.com/ethersphere/swarm/tracing"
Expand Down Expand Up @@ -367,7 +369,16 @@ func registerBzzService(bzzconfig *bzzapi.Config, stack *node.Node) {
// create a node store for this swarm key on global store
nodeStore = globalStore.NewNodeStore(common.HexToAddress(bzzconfig.BzzKey))
}
return swarm.NewSwarm(bzzconfig, nodeStore)
s, err := swarm.NewSwarm(bzzconfig, nodeStore)
if err != nil {
var e *localstore.BreakingMigrationError
if errors.As(err, &e) {
fmt.Println(e.Manual)
utils.Fatalf("Manual storage migration required.")
}
return nil, err
}
return s, nil
}
//register within the ethereum node
if err := stack.Register(boot); err != nil {
Expand Down
43 changes: 43 additions & 0 deletions storage/fcds/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// Copyright 2019 The Swarm Authors
// This file is part of the Swarm library.
//
// The Swarm library is free software: you can redistribute it and/or modify
// it under the terms of the GNU Lesser General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// The Swarm library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public License
// along with the Swarm library. If not, see <http://www.gnu.org/licenses/>.

// Package fcds provides storage layers for storing chunk data only.
//
// FCDS stands for Fixed Chunk Data Storage.
//
// Swarm Chunk data limited size property allows a very specific chunk storage
// solution that can be more performant than more generalized key/value
// databases. FCDS stores chunk data in files (shards) at fixed length offsets.
// Relations between chunk address, file number and offset in that file are
// managed by a separate MetaStore implementation.
//
// Package fcds contains the main implementation based on simple file operations
// for persisting chunk data and relaying on specific chunk meta information
// storage.
//
// The reference chunk meta information storage is implemented in fcds/mem
// package. It can be used in tests.
//
// LevelDB based chunk meta information storage is implemented in fcds/leveldb
// package. This implementation should be used as default in production.
//
// Additional FCDS Store implementation is in fcds/mock. It uses mock store and
// can be used for centralized chunk storage options that mock storage package
// provides.
//
// Package fcds/test contains test functions which can be used to validate
// behaviour of different FCDS or its MetaStore implementations.
package fcds
Loading