performance issue for gov/tally

[carameleon]

Summary of Bug

This is a result query to full node. ( ~ 1TB data)

func TestGetProposals(t *testing.T) {
	queryClient := cli.GetGovQueryClient()
	request := govtypes.QueryTallyResultRequest{ProposalId: 63}

	tallyResp, err := queryClient.TallyResult(context.Background(), &request)
	require.NoError(t, err)

	t.Log(tallyResp.Tally)
}

=== RUN   TestGetProposals
    /cosmos/client/gov_test.go:20: "yes": "23472585477427"
        abstain: "20014268669628"
        "no": "2690448414295"
        no_with_veto: "43925427829"

--- PASS: TestGetProposals (311.73s)

it takes time 5mins to get gov/tally about single prop (63)

pruned node1 (under 10GB ./data)

takes 20 secs to get gov/tally for prop 62 ,63

pruned node2 (115G ./data)

Mar 14 16:02:23 start proposal update : 62
Mar 14 16:04:07 finish proposal update : 62

Mar 21 08:07:31 start updating proposals : 62, 63
Mar 21 08:14:27 finish update proposals : 62, 63

It takes 7mins to get gov/tally about prop 62, 63.
That means node will stop for 7mins to complete return gov/tally.
In my memory, It has been happening since prop 60.(means node slow down clearly)

I'm wondering

• is there any approach to get gov/tally result faster.
• will be improved upcoming mainnet upgrade?

Version

gaiad v6.0.0

Steps to Reproduce

try to get tally result using gRPC.

    queryClient := govtypes.NewQueryClient(c.GRPC)
    request := govtypes.QueryTallyResultRequest{ProposalId: resp.Proposal.ProposalId}
    tally, err := queryClient.TallyResult(context.Background(), &request)

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[alexanderbez]

Are these for proposals that are still in their voting periods? If so, the cost would come from Keeper#Tally which iterates over all bonded validators and iterates over all votes, which I imagine incurs a majority of the cost, although I don't have evidence to prove this.

That being that case, idk what we can really do here. Hope that SMT and/or latest IAVL improvements from Osmosis helps?

[ValarDragon]

We're testing what we hope to be the last fix for IAVL stability, hopefully we can work on upstreaming it after that. I think thats the only thing that can be done to fix this computation overhead :(

We could build accumulators into the state machine, which is probably good long term, but the root cause of slowness here is definitely IAVL & bad levelDB configs.

I think on top of the osmosis work, getting a roadmap of levelDB things that we should try in prod to help speed things up is a great move. I know @ValNodes and @marbar have been working on both code to compact more leveldb data for speed + playing around with configs.

[fig666]

• the incentive comes from the gov. treasury toll of over paid deposits and refunds from contributions of all disbanded users of ended projects from github's database.

[julienrbrt]

Given that TallyResut just call keeper.Tally, closing this as dup of #10353, so we have one tracking issue for the improvements.