Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail after update #189

Open
hackhowtofaq opened this issue Sep 15, 2022 · 5 comments
Open

Fail after update #189

hackhowtofaq opened this issue Sep 15, 2022 · 5 comments

Comments

@hackhowtofaq
Copy link

I 've got the following, after updating to 3.4.0 and restart my node.

[2022-09-15 10:34:38]  INFO [snapshot] Snapshot loaded successfully.
[2022-09-15 10:34:38] DEBUG [ledger] Reseting state to height 14760131, hash 0xaba8e95b98bc6f1555c84b304b6b5e853cc6fd19d34abd4dee9345a5be6c4e0b

[2022-09-15 10:34:38]  INFO [p2p] Connecting to seed and persisted peers...
[2022-09-15 10:34:38]  WARN [p2p] Failed to retrieve previously persisted peers
[2022-09-15 10:34:38] DEBUG [consensus] Enter epoch 14845755
[2022-09-15 10:34:38] DEBUG [consensus] shouldPropose=false: epoch is behind epoch=14845755 tip=0x6062f42237d7ab0021ebe7df7e98c9e43b14579406dfce2ff009b9fefd800c38 tip.Epoch=14845755
[2022-09-15 10:34:38]  INFO [p2p] InboundPeerListener listen routine started, seedPeerOnly set to false
[2022-09-15 10:34:38] DEBUG peers is : %v, stringbuilder is : %s 
[]
[2022-09-15 10:34:38] DEBUG [request] refreshCounter: 0
[2022-09-15 10:34:38] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 10:34:38] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x6062f42237d7ab0021ebe7df7e98c9e43b14579406dfce2ff009b9fefd800c38]
[2022-09-15 10:34:38] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 10:34:38]  WARN [reporter] Reporter failed to create request: Post http://guardian-metrics.thetatoken.org:9000/peers/set: dial tcp 34.220.181.170:9000: connect: connection refused
[2022-09-15 10:34:44] DEBUG [request] refreshCounter: 1
[2022-09-15 10:34:44] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 10:34:44] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x6062f42237d7ab0021ebe7df7e98c9e43b14579406dfce2ff009b9fefd800c38]
[2022-09-15 10:34:44] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 10:34:50] DEBUG [consensus] Epoch timeout. Repeating epoch e.epoch=14845755
[2022-09-15 10:34:50] ERROR Failed to create tree store for: 0x3ef8fcf3d1e495467b0ca0e53e35868a2db42bbcb385e051699e5490c4935471: missing trie node 3ef8fcf3d1e495467b0ca0e53e35868a2db42bbcb385e051699e5490c4935471 (path )
[2022-09-15 10:34:50] PANIC [ledger] Failed to load state for validator pool block.BlockHeader.StateHash=0x3ef8fcf3d1e495467b0ca0e53e35868a2db42bbcb385e051699e5490c4935471 block.HCC.BlockHash=0x11b729ab9262339575b163a24d212a1e3531318828028b7d07c49ce92339e081 block.Hash=0x7dad8c2d7f38b43c5cc3329e9ac076dfdbba72a4f069c0001fbd643e99a69841 block.Height=14760129 block.Status.IsTrusted()=false
panic: (*logrus.Entry) (0x12bb840,0xc0049e96c0)

goroutine 117 [running]:
github.com/sirupsen/logrus.Entry.log(0xc00008a180, 0xc01fb54e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:239 +0x2cf
github.com/sirupsen/logrus.(*Entry).Log(0xc0049e9650, 0x0, 0xc00b66d050, 0x1, 0x1)
	/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:268 +0xc8
github.com/sirupsen/logrus.(*Entry).Panic(0xc0049e9650, 0xc00b66d050, 0x1, 0x1)
	/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:306 +0x55
github.com/thetatoken/theta/ledger.(*Ledger).GetFinalizedValidatorCandidatePool(0xc00015f800, 0x3cb4387f2d8cad7d, 0xdf76c09a9e32c35c, 0xc069f0a472badb, 0x4198a6993e64bd1f, 0xc0231b0400, 0x0, 0x20, 0x20)
	/go/src/github.com/thetatoken/theta/ledger/ledger.go:137 +0x6d5
github.com/thetatoken/theta/consensus.selectTopStakeHoldersAsValidatorsForBlock(0x16441e0, 0xc00ba5e180, 0xabd73722f46260, 0xe4c9987edfe7eb21, 0x2fcedf069457143b, 0x380c80fdfeb909f0, 0xc0231b0400, 0x20)
	/go/src/github.com/thetatoken/theta/consensus/validator.go:161 +0x83
github.com/thetatoken/theta/consensus.(*RotatingValidatorManager).GetValidatorSet(0xc0001d5ae0, 0xabd73722f46260, 0xe4c9987edfe7eb21, 0x2fcedf069457143b, 0x380c80fdfeb909f0, 0xa01a9e37efce6e6a)
	/go/src/github.com/thetatoken/theta/consensus/validator.go:128 +0x50
github.com/thetatoken/theta/consensus.(*ConsensusEngine).shouldVoteByID(0xc00ba5e180, 0xefce6e6a472f3cfe, 0x6790e9bda01a9e37, 0x22f462602a1f9943, 0xdfe7eb2100abd737, 0x9457143be4c9987e, 0xfeb909f02fcedf06, 0x2fcedf06380c80fd, 0x380c80fdfeb909f0)
	/go/src/github.com/thetatoken/theta/consensus/engine.go:827 +0x4b
github.com/thetatoken/theta/consensus.(*ConsensusEngine).shouldVote(0xc00ba5e180, 0xabd73722f46260, 0xe4c9987edfe7eb21, 0x2fcedf069457143b, 0x380c80fdfeb909f0, 0x0)
	/go/src/github.com/thetatoken/theta/consensus/engine.go:823 +0x9d
github.com/thetatoken/theta/consensus.(*ConsensusEngine).vote(0xc00ba5e180)
	/go/src/github.com/thetatoken/theta/consensus/engine.go:835 +0xd7
github.com/thetatoken/theta/consensus.(*ConsensusEngine).mainLoop(0xc00ba5e180)
	/go/src/github.com/thetatoken/theta/consensus/engine.go:283 +0x6ce
created by github.com/thetatoken/theta/consensus.(*ConsensusEngine).Start
	/go/src/github.com/thetatoken/theta/consensus/engine.go:162 +0x1ed

Any clues?

Ty.
@jieyilong
Copy link
Member

Thanks for reporting the issue. This is strange, looks like the DB was corrupted somehow. One option is to restart the node from the latest snapshot

# Replace <CONFIG_FOLDER> with the actual path to your config folder
move <CONFIG_FOLDER>/db <CONFIG_FOLDER>/db.old
move <CONFIG_FOLDER>/snapshot <CONFIG_FOLDER>/snapshot.old
wget -O <CONFIG_FOLDER>/snapshot `curl -k https://mainnet-data.thetatoken.org/snapshot`

Then restart the Theta node as usual. Thanks.

@hackhowtofaq
Copy link
Author

Thank you for your response, I had to delete db folder in bin/ as well.
Now I'm repeatedly getting this

[2022-09-15 16:46:32] DEBUG [request] refreshCounter: 3
[2022-09-15 16:46:32] DEBUG [request] Reset refreshCounter
[2022-09-15 16:46:32] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:32] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:32] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:35] DEBUG [consensus] Epoch timeout. Repeating epoch e.epoch=17185574
[2022-09-15 16:46:35] DEBUG [consensus] Enter epoch 17185574
[2022-09-15 16:46:35] DEBUG [consensus] shouldPropose=false: epoch is behind epoch=17185574 tip=0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552 tip.Epoch=17185574
[2022-09-15 16:46:39] DEBUG [request] refreshCounter: 0
[2022-09-15 16:46:39] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:39] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:39] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:45] DEBUG [request] refreshCounter: 1
[2022-09-15 16:46:45] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:45] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:45] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:47] DEBUG [consensus] Epoch timeout. Repeating epoch e.epoch=17185574
[2022-09-15 16:46:47] DEBUG [consensus] Enter epoch 17185574
[2022-09-15 16:46:47] DEBUG [consensus] shouldPropose=false: epoch is behind epoch=17185574 tip=0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552 tip.Epoch=17185574
[2022-09-15 16:46:51] DEBUG [request] refreshCounter: 2
[2022-09-15 16:46:51] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:51] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:51] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:58] DEBUG [request] refreshCounter: 3
[2022-09-15 16:46:58] DEBUG [request] Reset refreshCounter
[2022-09-15 16:46:58] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:58] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:58] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:59]  INFO [p2p] Connecting to seed and persisted peers...
[2022-09-15 16:46:59]  WARN [p2p] Failed to retrieve previously persisted peers
[2022-09-15 16:46:59] DEBUG Memory usage: Alloc = 713.227 MiB	TotalAlloc = 148619.333 MiB	Sys = 2352.441 MiB	NumGC = 312	StackInuse = 1.031 MiB	StackSys = 1.031 MiB	HeapInuse = 800.391 MiB	HeapSys = 2238.969 MiB

Is this normal?

@hackhowtofaq
Copy link
Author

Probably this is my issue?

 Could not perform UPNP discover: write udp4 0.0.0.0:57859->239.255.255.250:1900: i/o timeout

I tried from different VPS but I get the same message.

@hackhowtofaq
Copy link
Author

hackhowtofaq commented Sep 19, 2022 •
edited
Loading

For a really strange reason my config.yaml was missing!!

Downloaded and all good!

You can close it, thank you.

@YuXiaoCoder YuXiaoCoder mentioned this issue Jan 13, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hackhowtofaq @jieyilong and others