Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Block by Block Synchronous Sate Sync #812

Open
wants to merge 126 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
702cc81
WIP debugging
gokutheengineer Apr 13, 2023
bbf5458
wip
gokutheengineer Apr 13, 2023
6e65731
Add debug channels
Olshansk Apr 13, 2023
b3a953f
tests pass
gokutheengineer Apr 14, 2023
89fad5d
clean
gokutheengineer Apr 14, 2023
0fabffc
clean
gokutheengineer Apr 14, 2023
3ae3b88
clean, add comments
gokutheengineer Apr 14, 2023
e1091d0
syncs and adds blocks
gokutheengineer Apr 14, 2023
fd52e69
proceeds except when leader is one of new nodes
gokutheengineer Apr 14, 2023
c730357
leader election works
gokutheengineer Apr 14, 2023
f9a3424
debugging leader re-election
gokutheengineer Apr 15, 2023
1810cda
debug
gokutheengineer Apr 16, 2023
5a909ad
syncs up successfully, if itis not leader
gokutheengineer Apr 16, 2023
7cd1f4b
clean
gokutheengineer Apr 16, 2023
74def84
wip
gokutheengineer Apr 19, 2023
97d3f87
Merge branch 'main' into consensus/state-sync-channels
gokutheengineer Apr 19, 2023
48a912b
update state sync module
gokutheengineer Apr 19, 2023
92c2acf
clean, add comments
gokutheengineer Apr 19, 2023
de491dd
clean
gokutheengineer Apr 19, 2023
05161bd
fix typos
gokutheengineer Apr 19, 2023
5e70748
Update consensus/e2e_tests/utils_test.go
gokutheengineer Apr 21, 2023
9c17b18
Update consensus/e2e_tests/state_sync_test.go
gokutheengineer Apr 21, 2023
3270ee9
Update consensus/e2e_tests/utils_test.go
gokutheengineer Apr 21, 2023
a8fd744
Update consensus/module_consensus_state_sync.go
gokutheengineer Apr 21, 2023
1c4eb53
fixes
gokutheengineer Apr 25, 2023
c081464
move isValidator to persistence
gokutheengineer Apr 25, 2023
64bbfb1
address comments
gokutheengineer Apr 25, 2023
0a4140a
add block committed event
gokutheengineer Apr 26, 2023
c42790d
merge
gokutheengineer May 3, 2023
0034be9
update config
gokutheengineer May 3, 2023
386de6d
wip fixes
gokutheengineer May 3, 2023
92328c5
tests pass
gokutheengineer May 3, 2023
74675d1
clean commented parts
gokutheengineer May 3, 2023
8224e9a
address comments
gokutheengineer May 4, 2023
da8bdd8
updates
gokutheengineer May 4, 2023
a481e51
clean
gokutheengineer May 4, 2023
0671fd1
add generated helm docs
invalid-email-address May 4, 2023
959f60a
address comments, fix errors
gokutheengineer May 4, 2023
5068558
Merge branch 'state-sync' of github.com:pokt-network/pocket into stat…
gokutheengineer May 4, 2023
992a7d9
nits
gokutheengineer May 4, 2023
582d6f0
add helper
gokutheengineer May 5, 2023
97fb9ef
Lowercase some local testing types for consensus
Olshansk May 24, 2023
19a3a2e
Rename PKs to privKeys
Olshansk May 24, 2023
81af631
Remove privKeys from placeholderBlocks
Olshansk May 24, 2023
f74fb4c
Cleaning up some test code
Olshansk May 24, 2023
3619377
Finished reviewing consensus/e2e_tests/utils_test.go
Olshansk May 24, 2023
8150b76
Simplified state sync module interface
Olshansk May 24, 2023
373dc48
All tests still passing
Olshansk May 24, 2023
a0ddcfc
review consensus/fsm_handler.go
Olshansk May 25, 2023
f01d704
Remove isValidator
Olshansk May 25, 2023
66a0148
Merge with main
Olshansk May 25, 2023
84e2bc3
Tests pass after merging to main
Olshansk May 25, 2023
a0ec8d6
Rename background processes
Olshansk May 26, 2023
85f2156
Reviewed the block application loop
Olshansk May 26, 2023
476ade9
Reviewed server.go
Olshansk May 26, 2023
7dc5870
Cleaning up some code
Olshansk May 26, 2023
9945564
Cleanup consensus module a bit
Olshansk May 26, 2023
8d9f824
Completely removed the state sync from the consensus module
Olshansk May 26, 2023
1b16ab5
Merge branch 'main' into issues/352/state_sync
Olshansk May 26, 2023
60966cb
Interim commit to help me pick up
Olshansk May 30, 2023
0d760ba
Merge with main
Olshansk Jun 8, 2023
f86b491
add generated helm docs
invalid-email-address Jun 8, 2023
f7e0fbe
Add some notes & qualifiers to the protocol state sync document
Olshansk Jun 8, 2023
a4b7a7b
Moved WithDebugEventsChannel into a file with the test tag
Olshansk Jun 8, 2023
239661f
Moved some helpers into a debug helper file
Olshansk Jun 8, 2023
da3684e
Code cleanup figuring things out before diving into the business logi…
Olshansk Jun 8, 2023
0bcc33a
Add some comments
Olshansk Jun 8, 2023
ab8db90
Removed SetBlock
Olshansk Jun 9, 2023
759788e
Cleanup to some of the consensus testing utilities
Olshansk Jun 9, 2023
5205d9a
Update factory functions for consensus submodules
Olshansk Jun 9, 2023
b24871f
Updated StartSynchronousStateSync
Olshansk Jun 9, 2023
e82e41d
WIP - simplifying the test
Olshansk Jun 9, 2023
5696d48
Using broadcast instead of send with a loop where appropriate
Olshansk Jun 9, 2023
54fcd1b
A little bit of cleanup
Olshansk Jun 12, 2023
f0570da
Reverted utility module changes
Olshansk Jun 12, 2023
51864e2
Interim commit while working on 'TestStateSync_UnsyncedPeerSyncs_Succ…
Olshansk Jun 12, 2023
3a6cd23
State sync test passes with time.Sleep hack
Olshansk Jun 13, 2023
839cb3d
Merge with main
Olshansk Jun 13, 2023
c82860a
Merge with main
Olshansk Jun 13, 2023
b20b43a
Merge with main
Olshansk Jun 14, 2023
ed16e80
Some code cleanup while validating that unit tests all pass and doing…
Olshansk Jun 14, 2023
918e65b
Renamed eventsChannel to sharedNetworkEvents in consensus
Olshansk Jun 14, 2023
d2ed2bb
Added runtime/debug_helpers.go
Olshansk Jun 14, 2023
d2d7d01
Removed waitForNetworkFSMEvents
Olshansk Jun 14, 2023
8f540f2
Removed waitForNetworkFSMEvents
Olshansk Jun 14, 2023
7a1e146
Removed all the 'OLSH' comments
Olshansk Jun 14, 2023
c9bdebf
Removed an unnecessary check for m.utilityUnitOfWork
Olshansk Jun 14, 2023
1fa2a69
Removed unnecessary checks in hotstuff_replica
Olshansk Jun 14, 2023
730a1bf
Reverted changelogs
Olshansk Jun 15, 2023
c1057b0
Removed unused debug channels
Olshansk Jun 15, 2023
17384e9
Removed publishStateSyncBlockCommittedEvent
Olshansk Jun 15, 2023
2771a84
Renamed module_state_sync to state_sync_helpers
Olshansk Jun 15, 2023
54f6bce
Consolidated handlers and helpers for state sync
Olshansk Jun 15, 2023
e9fd550
Minor formatting issues in the .feature files
Olshansk Jun 15, 2023
5a4ec3b
Merge branch 'main' into issues/352/state_sync
Olshansk Jun 26, 2023
4efaa53
Merge with main
Olshansk Jun 29, 2023
54caf5b
Merge branch 'main' into issues/352/state_sync
Olshansk Jun 30, 2023
680f49b
Merge branch 'main' into issues/352/state_sync
Olshansk Jul 11, 2023
0cdcd9a
Merge branch 'main' into issues/352/state_sync
Olshansk Jul 12, 2023
d6d6b99
Merge with main
Olshansk Jul 18, 2023
c9fe524
Merge with main
Olshansk Jul 21, 2023
6856feb
Merge branch 'main' into issues/352/state_sync
Olshansk Jul 24, 2023
e6d0d5d
Merge with main
Olshansk Jul 26, 2023
c2cc4ae
Remove code from merge conflict
Olshansk Jul 26, 2023
209d8cc
Tend to go lint errors
Olshansk Jul 27, 2023
21e4441
Merge branch 'main' into issues/352/state_sync
Olshansk Jul 28, 2023
e74cc90
E2E State Sync Test
Olshansk Jul 28, 2023
9d840ed
Linting improvements
Olshansk Jul 29, 2023
9118167
Merge with main
Olshansk Aug 1, 2023
8418390
Merge with main
Olshansk Aug 1, 2023
2395a61
Update e2e/tests/steps_init_test.go
Olshansk Aug 1, 2023
489978f
Update documentation related to installing rsync
Olshansk Aug 1, 2023
4cc8405
Added # IMPROVE(#959): Remove time-based waits from tests
Olshansk Aug 2, 2023
da5c48f
Merge with main
Olshansk Aug 2, 2023
2792210
Follow up on minor comments
Olshansk Aug 2, 2023
9ce7882
s/validator/node in a few places
Olshansk Aug 2, 2023
4df3e28
Add account.feature and validator.feature
Olshansk Aug 2, 2023
34f635a
Remove kubectl_check from install_cli_deps
Olshansk Aug 2, 2023
1289063
Remove kubectl_check from install_cli_deps
Olshansk Aug 2, 2023
031c10c
Skip E2E test
Olshansk Aug 3, 2023
f5c68ba
Updated TODOs
Olshansk Aug 3, 2023
fa5577c
Added tags to the ./argo-linux-amd64 commands when running e2e-tests
Olshansk Aug 3, 2023
8a4dc0b
Merge with issues/829/state_sync_test
Olshansk Aug 3, 2023
46cd8e5
Merge with main
Olshansk Aug 3, 2023
819f990
Updated a few comments
Olshansk Aug 3, 2023
6ffe41d
Last week's state sync debugging changes
Olshansk Aug 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove code from merge conflict
  • Loading branch information
Olshansk committed Jul 26, 2023
commit c2cc4ae54b366d93fbbb13fadfd2c5a81cf028c4
43 changes: 0 additions & 43 deletions app/client/cli/debug.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
package cli

import (
"errors"
"fmt"
"os"

"github.com/manifoldco/promptui"
@@ -11,9 +9,7 @@ import (

"github.com/pokt-network/pocket/app/client/cli/helpers"
"github.com/pokt-network/pocket/logger"
"github.com/pokt-network/pocket/p2p/providers/peerstore_provider"
"github.com/pokt-network/pocket/shared/messaging"
"github.com/pokt-network/pocket/shared/modules"
)

// TECHDEBT: Lowercase variables / constants that do not need to be exported.
@@ -211,42 +207,3 @@ func sendDebugMessage(cmd *cobra.Command, debugMsg *messaging.DebugMessage) {
logger.Global.Error().Err(err).Msg("Failed to send debug message")
}
}

// fetchPeerstore retrieves the providers from the CLI context and uses them to retrieve the address book for the current height
func fetchPeerstore(cmd *cobra.Command) (typesP2P.Peerstore, error) {
bus, ok := helpers.GetValueFromCLIContext[modules.Bus](cmd, helpers.BusCLICtxKey)
if !ok || bus == nil {
return nil, errors.New("retrieving bus from CLI context")
}
// TECHDEBT(#810, #811): use `bus.GetPeerstoreProvider()` after peerstore provider
// is retrievable as a proper submodule
pstoreProvider, err := bus.GetModulesRegistry().GetModule(peerstore_provider.PeerstoreProviderSubmoduleName)
if err != nil {
return nil, errors.New("retrieving peerstore provider")
}
currentHeightProvider := bus.GetCurrentHeightProvider()

height := currentHeightProvider.CurrentHeight()
pstore, err := pstoreProvider.(peerstore_provider.PeerstoreProvider).GetStakedPeerstoreAtHeight(height)
if err != nil {
return nil, fmt.Errorf("retrieving peerstore at height %d", height)
}
// Inform the client's main P2P that a the blockchain is at a new height so it can, if needed, update its view of the validator set
err = sendConsensusNewHeightEventToP2PModule(height, bus)
if err != nil {
return nil, errors.New("sending consensus new height event")
}
return pstore, nil
}

// sendConsensusNewHeightEventToP2PModule mimicks the consensus module sending a ConsensusNewHeightEvent to the p2p module
// This is necessary because the debug client is not a validator and has no consensus module but it has to update the peerstore
// depending on the changes in the validator set, which is based on the on-chain state.
// TODO(#613): Make the debug client mimic a full node.
func sendConsensusNewHeightEventToP2PModule(height uint64, bus modules.Bus) error {
newHeightEvent, err := messaging.PackMessage(&messaging.ConsensusNewHeightEvent{Height: height})
if err != nil {
logger.Global.Fatal().Err(err).Msg("Failed to pack consensus new height event")
}
return bus.GetP2PModule().HandleEvent(newHeightEvent.Content)
}