Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[neox] Merge neo-project/neox to neo-project/master #1793

Closed
wants to merge 25 commits into from

Conversation

ZhangTao1596
Copy link

@ZhangTao1596 ZhangTao1596 commented Jul 24, 2020

MPT Integration Specification

introduction

As we have already merged MPT into neo, now it's time for us to integrate it with neo. In this article, I will clarify the detail of the solution in neox.

persistence

Using DataCache implemented by MPT, we can replace the Storage in snapshot as the contracts' storage.

SnapshotView

Field Type Description
Blocks StoreDataCache<UInt256, TrimmedBlock>
Transactions StoreDataCache<UInt256, TransactionState>
Contracts StoreDataCache<UInt160, ContractState>
HeaderHashList StoreDataCache<SerializableWrapper, HeaderHashList>
BlockHashIndex StoreMetaDataCache
HeaderHashIndex StoreMetaDataCache
ContractId StoreMetaDataCache
LocalStateRoot StoreDataCache<SerializableWrapper, HashIndexState> State root hash of every block calculated by local node
ValidatorsStateRoot StoreMetaDataCache State root verified by validators, only store the latest one
Storages MPTDataCache<StorageKey, StorageItem> Storage of contracts, implemented with MPT

consensus

Validators confirm state root of a block in next consensus.
Primary node proposal PrepareRequest with signature of previous block state root. Backup nodes verify the state root signature with their local state root and send their signatures in PrepareResponse if succeed.
As long as a validator collects M preparation messages, it can generate state root of last block.
Because state root is a determined value, validators just vote for right or not. Therefore validators can determine state root in one consensus turn.

PrepareRequest

Field Type Description
Timestamp ulong
Nonce ulong
TransactionHashes UInt256[]
StateRootSignature byte[] signature of previous block state root

PrepareResponse

Field Type Description
PreparationHash UInt256
StateRootSignature byte[] signature of previous block state root

network

State root defination and GetStateRoot message for state root sync.

StateRoot

Field Type Description
Version byte Version of state root caculate algorithm
Index uint Index of block which state root is calculated from
RootHash UInt256 State root hash
Witness Witness Verification script and invocation script

GetStateRoot

Field Type Description
Index uint The minimum index of the requested stateroot

ledger

​ When receiving the block, the node will calculate state root after the persist is completed, and it will check the state root cache for state root of the block that have been confirmed by the validators.
​ When receiving state root signed by validators, the node will check and verify it and compare it with local state root, update state height if they are equal.

@erikzhang @shargon @vncoelho

{
Version = StateRoot.CurrentVersion,
Index = Block.Index - 1,
RootHash = Blockchain.Singleton.GetLocalStateRoot(Block.Index - 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't have the state synchronized , what's happend? we should not be able to be a CN, isn't it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it doesn't matter.
Because all consensus nodes vote for their calculated StateRoot. As long as the node block synchronization is completed, it has the ability to verify StateRoot, regardless of whether the State Root synchronization is completed. Even if the StateRoot is not synchronized, its StateRoot may be correct. If the StateRoot is incorrect, it will be rejected by other nodes.

src/neo/Consensus/PrepareResponse.cs Outdated Show resolved Hide resolved
src/neo/Consensus/PrepareRequest.cs Outdated Show resolved Hide resolved
src/neo/Persistence/Helper.cs Show resolved Hide resolved
src/neo/Ledger/Blockchain.State.cs Outdated Show resolved Hide resolved
ZhangTao and others added 10 commits August 12, 2020 18:56
* don't use snapshot directly in plugin

* persistence and ut

* fix mpt

* precommit

* fix commit

* rm LocalRootHashIndex

* fix ut

* pre commit

* fix clone view

* change name

* rename

* abstract

* comment

* fix ReadOnlyView

* rm precommit

* rm HashState

* add MPTDataCache

* optimze

* optimize

* remove blank line

* StateRoot verify fee

* expose Root in MPTDataCache

* fix some and ut

* merge master

* master

* fix mpt ut

* fix Storages and name

* rename

* add comment

* proof prefix

* fix

* format

* format

* format

* add StateRoot ut

* reset mpt prefix

* rm GetMessage

* throw exception when no script hash in state root

* UpdateLocalStateRoot when Storages changed

* rename

Co-authored-by: Tommo-L <[email protected]>
* don't use snapshot directly in plugin

* recover

* consensus state root

* fix build error

* rename

* remerge

* rm GetMessage

* rename

* throw exception when no script hash in state root

* Clean double enter

* Clean big line

* format

* put root sig into request and response

* update consensus ut

* Check Json.Serialize map keys (neo-project#1705)

* don't use snapshot directly in plugin

* recover

* consensus state root

* fix build error

* rename

* remerge

* rm GetMessage

* rename

* throw exception when no script hash in state root

* Clean double enter

* Clean big line

* format

* put root sig into request and response

* update consensus ut

* fix some

* fix some

* format

* comment

* fix

* requst in recovery

* StateRootSignature fixed size

* no need ?

* format

* Update ConsensusContext.cs

* Check sc.Complete

Co-authored-by: Shargon <[email protected]>
* network

* format

* add GetStateRootPayload

* recover ping

* rm state height in Update

* get state root of Blockchain.Height - 1

* rm Blockchain.State.cs
* cn state root verify and persist

* check state root cache

* remove mpt proof, move to plugin

* TryAdd optimization

* fix

* ..

* file name

* filename

* fix error

Co-authored-by: Shargon <[email protected]>
KickSeason added 2 commits August 13, 2020 09:36
@ZhangTao1596
Copy link
Author

ZhangTao1596 commented Aug 27, 2020

I will reopen this after disk usage optimization applied on neox. Referring #1876

@ZhangTao1596 ZhangTao1596 reopened this Sep 8, 2020
@ZhangTao1596
Copy link
Author

ZhangTao1596 commented Sep 8, 2020

I reopen this. I will open another pr to optimize disk usage after this.

@erikzhang
Copy link
Member

I have an idea. Can we decouple StateRoot from the consensus algorithm? We can add a new role to the chain, the status verifier. They are designated by the committee, communicate through additional channels, sign StateRoot and broadcast.

@ZhangTao1596
Copy link
Author

I have an idea. Can we decouple StateRoot from the consensus algorithm? We can add a new role to the chain, the status verifier. They are designated by the committee, communicate through additional channels, sign StateRoot and broadcast.

What's the problem you think we have in current design exchanging signature in PrepareRequest and PrepareResonse
? We don't add much more extra payload in consensus.
Besides, I think it is reasonable to confirm their state of each node in consensus.

@roman-khimov
Copy link
Contributor

I think it is reasonable to confirm their state of each node in consensus.

It's not just reasonable, it's absolutely necessary for consensus nodes to be in agreement wrt current state because it directly affects transaction validation and execution.

And #1526 is still relevant, it'd be nice to discuss any changes to the current scheme there.

@ZhangTao1596
Copy link
Author

ZhangTao1596 commented Sep 9, 2020

Since we have new discussion about how to sign StateRoot in #1526, I will remove the consensus change from this pr.

@ZhangTao1596
Copy link
Author

Close this. Move to neo-modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants