[statesync] module=blockchain failed commit first block

[dualsystems]

Describe the bug
Some times statesync is failed. Snapshot with start height=317300, can't commit first block.

Screenshots or log output

10:12AM INF Applied snapshot chunk to ABCI app chunk=0 format=1 height=317300 module=statesync total=1
10:12AM INF Verified ABCI app appHash="U�R!�X4-�a\x03��e��TYZD�\x11 @:pl��G" height=317300 module=statesync
10:12AM INF Snapshot restored format=1 hash="<'qU0�\x04Y�4\x14�~o��0������FhE���I��" height=317300 module=statesync
10:12AM INF Starting BlockPool service impl=BlockPool module=blockchain
10:12AM INF Inflation minted coins echf=405 edkk=1011 eeur=106375 enok=1517 esek=1011 module=x/inflation ungm=22304341
10:12AM INF minted coins from module account amount=405echf,1011edkk,106375eeur,1517enok,1011esek,22304341ungm from=inflation module=x/bank
10:12AM ERR WARNING: Attempt to allocate proposer rewards to unknown proposer . This should happen only if the proposer unbonded completely within a single block, which generally should not happen except in exceptional circumstances (or fuzz testing). We recommend you investigate immediately. module=x/distribution
10:12AM INF Endblock: Block 317301 was proposed by E666CD57C621041F400ACD7D760D9CCF4D8B10A4 module=emz
10:12AM INF executed block height=317301 module=state num_invalid_txs=0 num_valid_txs=0
10:12AM INF commit synced commit=436F6D6D697449447B5B3732203232332039203138352031363920323238203720383420313620323535203233352031343420323532203833203231342032343420343820323039203133342034382039392031333920323332203133203434203232372032362035392038332031303920313231203232365D3A34443737357D
10:12AM INF committed state app_hash=48DF09B9A9E4075410FFEB90FC53D6F430D18630638BE80D2CE31A3B536D79E2 height=317301 module=state num_txs=0
10:12AM INF indexed block height=317301 module=txindex
panic: Failed to process committed block (317302:5C9F6985500C9A35F78D1BE1E23968AACB2EC446E451887AC8B61B1CA1FFED40): wrong Block.Header.AppHash.  Expected 48DF09B9A9E4075410FFEB90FC53D6F430D18630638BE80D2CE31A3B536D79E2, got 2EBCEC82893EC508A8D4B1B11A183C4BEB08D1FF0CA354B54BE97162D275E600

Additional context
Next snapshot with start height=317500 is sync normal.

cosmos/cosmos-sdk#10281

[haasted]

Thanks for letting us know about this, and for testing the snapshots. We'll look into whether we can add support for the rollback command that was added.

[ryuash]

Had the same issue, snapshot at height 393000 failed but snapshot at 396000 worked