gaia stuck in error: couldn't find validators at height 15214022

[flynnji]

Problem

firstly, i met the same error this issue #2406 , so i try to do something to fix the node. and then i execute the commands as below

# gaiad rollback 15214023
# gaiad rollback 15214022

after that i got a panic error while i trying to run gaiad.
what can i do to fix it?

:16AM INF starting node with ABCI Tendermint in-process
9:16AM INF service start impl=multiAppConn module=proxy msg={}
9:16AM INF service start connection=query impl=localClient module=abci-client msg={}
9:16AM INF service start connection=snapshot impl=localClient module=abci-client msg={}
9:16AM INF service start connection=mempool impl=localClient module=abci-client msg={}
9:16AM INF service start connection=consensus impl=localClient module=abci-client msg={}
9:16AM INF service start impl=EventBus module=events msg={}
9:16AM INF service start impl=PubSub module=pubsub msg={}
9:19AM INF service start impl=IndexerService module=txindex msg={}
9:19AM INF ABCI Handshake App Info hash="^lo��D`�n\x1d#��a\x1d+��\ns�v\x03濮�B\v��1" height=15214023 module=consensus protocol-version=0 software-version=HEAD-321d15a574def0f338ceacc5c060159ebba95edc
9:19AM INF ABCI Replay Blocks appHeight=15214023 module=consensus stateHeight=15214023 storeHeight=15214024
9:19AM INF Replay last block using real app module=consensus
panic: couldn't find validators at height 15214022 (height 15214023 was originally requested): %!w(<nil>)

goroutine 1 [running]:
github.com/tendermint/tendermint/state.getBeginBlockValidatorInfo(0xc000001680, {0x2689528, 0xc009424648}, 0x4f5b97)
	github.com/tendermint/[email protected]/state/execution.go:346 +0x3ea
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x267c260?, 0xc24b6155c0}, {0x26823f0, 0xc010c83d90}, 0xc000001680, {0x2689528, 0xc009424648}, 0xe825c7?)
	github.com/tendermint/[email protected]/state/execution.go:293 +0x219
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, {{0xc019214240, ...}, ...}, ...}, ...)
	github.com/tendermint/[email protected]/state/execution.go:140 +0x171
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, {{0xc019214240, ...}, ...}, ...}, ...)
	github.com/tendermint/[email protected]/consensus/replay.go:503 +0x23c
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, {{0xc019214240, ...}, ...}, ...}, ...)
	github.com/tendermint/[email protected]/consensus/replay.go:416 +0x7ae
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc001227e70, {0x268a760, 0xc0105f60d0})
	github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, ...}, ...)
	github.com/tendermint/[email protected]/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc000c22640, {0x2677ee0, 0xc00019ca00}, 0xc0015ea3f0, {0x2661a60, 0xc00d2ee390}, 0x0?, 0x0?, 0xc0015ea660, {0x267c260, ...}, ...)
	github.com/tendermint/[email protected]/node/node.go:779 +0x597
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2691928, 0xc000c13440}, {0x0, 0x0}, {0x267fd30, 0xc0010fede0}, ...}, ...)
	github.com/cosmos/[email protected]/server/start.go:280 +0x81b
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc001142f00?, {0x356bf20?, 0x0?, 0x0?})
	github.com/cosmos/[email protected]/server/start.go:128 +0x169
github.com/spf13/cobra.(*Command).execute(0xc001142f00, {0x356bf20, 0x0, 0x0})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000ec5200)
	github.com/spf13/[email protected]/command.go:1044 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/[email protected]/command.go:961
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc00110c210, 0xb})
	github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x1eb
main.main()
	github.com/cosmos/gaia/v9/cmd/gaiad/main.go:16 +0x2c

[bb4L]

same here

[mmulji-ic]

@flynnji @bb4L think it probably means that block isn't available with the validators that your connected to. Will check with the Comet team, cc @alijnmerchant21

panic: couldn't find validators at height 15214022 (height 15214023 was originally requested): %!w(<nil>)

[mmulji-ic]

@flynnji @bb4L sorry picking this up from awhile back, we discussed the rollback feature and it doesn't work as advertised. As the error message stated you don't seem to be connected to a node that has that block, maybe due to maxxed out connections. Pinging @alijnmerchant21 who's on the Comet team.

[mmulji-ic]

Hi @andynog can you comment on the rollback functionality?

[MSalopek]

Closing as there has been no activity for > 6months from the original issue authors.

The discussion has decayed.