RPC Node Stuck

[czarcas7ic]

v ✰ Thanks for creating an issue! ✰
☺ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -->

Background

I have attached heap and cpu profiles for an RPC node from Interstellar that was stuck.

Heap: https://drive.google.com/file/d/1p9OM7tF2tNvk4CP2QbDMjRxL8ZANdlRI/view?usp=sharing
CPU: https://drive.google.com/file/d/15Rr55BC9UN-mYnUsozBqxUCQWyvW1tRa/view?usp=sharing

My quick first pass guess is that its stuck on GetConsensusRound (see the heap profile), which is why we still see queries passing through the node, but no block progress.

They had really low peer settings (0/8), but this shouldn't block a node indefinitely from making progress.

Suggested Design

Review the profiles and determine what caused the node to be stuck.

Outcome:

• fix this issue

or

• create a detailed description of the issue with steps to fix

[deividaspetraitis]

While someone probably with more experience with the project can make sense of the attached profiles to determine the issue,
I was wondering is it possible to have Goroutine profile dumps included as well?

[czarcas7ic]

@deividaspetraitis I will ask to also take the goroutine profile next time as well, thanks