[Question] Trying to run a new consumer locally, but chain gets stuck on height 0.

[Roc8Trppn]

Hi,

I'm running a provider locally and I'm trying to add a consumer to the provider and run it locally as well.

Note: I have 1 validator on the provider and I want them to validate the consumer (i.e. the consumer should not have its own validators)

Steps that I have done:

1. I have initialised the consumer via $CONSUMER_BINARY init zone_hub --chain-id zone_chain
2. I created the genesis hash and the binary of the genesis.json
3. I updated the default create-consumer.json with chain_id spawn_time, genesis_hash and binary, e.g:

{
  "chain_id": "zone_chain",
  "metadata": {
    "name": "chain consumer",
    "description": "description",
    "metadata": "{\"forge_json_url\": \"url\", \"stage\": \"mainnet\"}"
  },
  "initialization_parameters": {
    "initial_height": {
      "revision_number": 0,
      "revision_height": 1
    },
    "genesis_hash": "4644414d4331c349c89c8a6b7bbdaf7b9a5a1d6b4a9fb9550c6fb04cf1995461",
    "binary_hash": "46bcec3836ca40e935837dfdaa75b371adafb643f4cbbbd77c0f8ac71e6d5f3f",
    "spawn_time": "2024-11-30T01:26:25Z",
    "unbonding_period": 1728000000000000,
    "ccv_timeout_period": 2419200000000000,
    "transfer_timeout_period": 1800000000000,
    "consumer_redistribution_fraction": "0.85",
    "blocks_per_distribution_transmission": 1000,
    "historical_entries": 10000
  }
}

4. Then I ran CreateConsumerMessage via $PROVIDER_BINARY tx provider create-consumer <path_to_json> --from= validator --gas auto --fees 5000stake --chain-id provider-chain (where validator is the account name of my only validator of the provider). This step works fine and gives the consumer chain a consumer ID of 0.
5. Next I opt in with my validator (this is before the spawn time ends): $PROVIDER_BINARY tx provider opt-in 0 --from validator --gas auto --fees 5000stake. This sets the state to CONSUMER_PHASE_INITIALIZED if checking it with $PROVIDER_BINARY query provider consumer-chain 0.
6. Next I waited until the spawn time passes, then I export the ccvconsumer data via: $PROVIDER_BINARY query provider consumer-genesis -o json > ccvconsumer.json. That one looks like this:

{
  "params": {
    "enabled": true,
    "blocks_per_distribution_transmission": "1000",
    "distribution_transmission_channel": "",
    "provider_fee_pool_addr_str": "",
    "ccv_timeout_period": "2419200s",
    "transfer_timeout_period": "1800s",
    "consumer_redistribution_fraction": "0.85",
    "historical_entries": "10000",
    "unbonding_period": "1728000s",
    "soft_opt_out_threshold": "0",
    "reward_denoms": [],
    "provider_reward_denoms": [],
    "retry_delay_period": "3600s",
    "consumer_id": "0"
  },
  "provider": {
    "client_state": {
      "chain_id": "provider-chain",
      "trust_level": { "numerator": "1", "denominator": "3" },
      "trusting_period": "1197504s",
      "unbonding_period": "1814400s",
      "max_clock_drift": "10s",
      "frozen_height": { "revision_number": "0", "revision_height": "0" },
      "latest_height": { "revision_number": "0", "revision_height": "206" },
      "proof_specs": [
        {
          "leaf_spec": {
            "hash": "SHA256",
            "prehash_key": "NO_HASH",
            "prehash_value": "SHA256",
            "length": "VAR_PROTO",
            "prefix": "AA=="
          },
          "inner_spec": {
            "child_order": [0, 1],
            "child_size": 33,
            "min_prefix_length": 4,
            "max_prefix_length": 12,
            "empty_child": null,
            "hash": "SHA256"
          },
          "max_depth": 0,
          "min_depth": 0,
          "prehash_key_before_comparison": false
        },
        {
          "leaf_spec": {
            "hash": "SHA256",
            "prehash_key": "NO_HASH",
            "prehash_value": "SHA256",
            "length": "VAR_PROTO",
            "prefix": "AA=="
          },
          "inner_spec": {
            "child_order": [0, 1],
            "child_size": 32,
            "min_prefix_length": 1,
            "max_prefix_length": 1,
            "empty_child": null,
            "hash": "SHA256"
          },
          "max_depth": 0,
          "min_depth": 0,
          "prehash_key_before_comparison": false
        }
      ],
      "upgrade_path": ["upgrade", "upgradedIBCState"],
      "allow_update_after_expiry": false,
      "allow_update_after_misbehaviour": false
    },
    "consensus_state": {
      "timestamp": "2024-11-28T20:10:01.007487Z",
      "root": { "hash": "kIv0Wri1NiJCHSwwULtB8oaZzmcTici8yW7n3OUDjNc=" },
      "next_validators_hash": "10A8537BE02F62547FA38B181E77B01C39DDF2A7A1421CA2B034DD57FFE475AD"
    },
    "initial_val_set": [
      {
        "pub_key": {
          "ed25519": "FOfb7qkhd1zAA8fwLeKloaC/PJZhKTjkZ6ik0CKQ32Q="
        },
        "power": "5000000"
      }
    ]
  },
  "new_chain": true
}

8. Then I updated the ccvconsumer part of the consumer's genesis.json file.
9. Then I change the ports of the consumer (since both chains should run locally)

• rpc_laddr = "tcp://127.0.0.1:26667"
  -p2p_laddr = "tcp://0.0.0.0:26666"
• proxy_app = "tcp://127.0.0.1:26668"
  -node = "tcp://localhost:26660"
• grpc_laddr = "tcp://127.0.0.1:9092" and in app.toml address = "localhost:9091"
  Provider:
• rpc_laddr = "tcp://0.0.0.0:26657"
• p2p_laddr = "tcp://0.0.0.0:26656"
• proxy_app = "tcp://127.0.0.1:26658"
• node = "tcp://localhost:26657"
• grpc_laddr = "tcp://0.0.0.0:9090" and in app.toml address = "localhost:9090"

10. I add a persistent peer via persistent_peers = "[email protected]:26656" in config.toml of the consumer. Note: I've tried @0:0:0:0:26656 and@localhost:26656 but every time I get the same error (see logs below)

Then i run the consumer (provider is already running). And I always get the following logs:

9:15PM INF No addresses to dial. Falling back to seeds module=pex
9:15PM ERR Can't add peer's address to addrbook err="Cannot add non-routable address bb31ae15e86a7a1d4f70f1269a7fcfa5f3000b4d@[127:0:0:1]:26656" module=p2p
9:15PM INF Saving AddrBook to file book=.../.interchain-security-c/config/addrbook.json module=p2p size=0
9:15PM INF starting gRPC server... address=localhost:9091 module=grpc-server
9:15PM ERR Error dialing peer err="incompatible: peer is on a different network. Got provider-chain, expected zone_chain" module=p2p

So to me it looks like something in the configuration went wrong, since the consumer chain seems to search for its own nodes rather then from the provider...But I'm not sure if this is the right interpretation...

Do you have any ideas where the problem might be? Am I missing something in the configuration?
Any ideas are very welcome...This quite a hard one to debug..

Note: concerning the relayer, when i try to create a channel via hermes create connection --a-chain zone_chain --a-client 07-tendermint-0 --b-client 07-tendermint-0 and both chains are running, I get the output:

INFO ThreadId(01) using default configuration from '....hermes/config.toml'
2024-11-30T02:07:21.276323Z  INFO ThreadId(01) running Hermes v1.10.4+542e14f
2024-11-30T02:07:21.558650Z  WARN ThreadId(01) Failed to fetch version specs for chain 'zone_chain': ABCI query returned an error: AbciQuery { code: Err(26), log: "interchain-security-c is not ready; please wait for first block: invalid height", info: "", index: 0, key: [], value: [], proof: None, height: block::Height(0), codespace: "sdk" }
2024-11-30T02:07:21.561202Z  WARN ThreadId(01) Will fall back on using the node version: 0.38.15
ERROR failed while querying client '07-tendermint-0' on chain 'zone_chain' with error: error decoding protobuf: error converting message type into domain type: the client state was not found

The config.toml of hermes looks like this:

[global]
log_level = "info"

[mode]

[mode.clients]
enabled = true
refresh = true
misbehaviour = true

[mode.connections]
enabled = false

[mode.channels]
enabled = false

[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = false
auto_register_counterparty_payee = false

[[chains]]
id = "provider-chain"   
type = "CosmosSdk"
rpc_addr = "http://127.0.0.1:26657"     
grpc_addr = "http://127.0.0.1:9090"   
rpc_timeout = "10s"
account_prefix = "cosmos"             
key_name = "provider-key"
store_prefix = "ibc"
default_gas = 2000000
max_gas = 4000000
gas_multiplier = 1.5
max_msg_num = 30
max_tx_size = 180000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
ccv_consumer_chain = false
memo_prefix = ""
sequential_batch_tx = false

[chains.event_source]
mode = "push"
url = "ws://localhost:26657/websocket"
batch_delay = "500ms"

[chains.trust_threshold]
numerator = "1"
denominator = "3"

[chains.gas_price]
price = 0.025
denom = "stake"

[chains.packet_filter]
policy = "allow"
list = [["transfer", "channel-3108"]]

[chains.packet_filter.min_fees]

[chains.address_type]
derivation = "cosmos"

[[chains]]
id = "zone_chain"                     
type = "CosmosSdk"
ccv_consumer_chain = true
rpc_addr = "http://127.0.0.1:26667"      
grpc_addr = "http://127.0.0.1:9091"  
rpc_timeout = "10s"
account_prefix = "cosmos"            
key_name = "consumer-key"          
store_prefix = "ibc"
default_gas = 2000000
max_gas = 4000000
gas_multiplier = 1.5
max_msg_num = 30
max_tx_size = 180000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
ccv_consumer_chain = false
memo_prefix = ""
sequential_batch_tx = false

[chains.event_source]
mode = "push"
url = "ws://localhost:26658/websocket"
batch_delay = "500ms"

[chains.trust_threshold]
numerator = "1"
denominator = "3"

[chains.gas_price]
price = 0.1
denom = "utia"

[chains.packet_filter]
policy = "allow"
list = [["transfer", "channel-0"]]

[chains.packet_filter.min_fees]

[chains.address_type]
derivation = "cosmos"

So with this config I can run hermes, but I get a bunch of errors (which is logical since I can't create the channels properly...)

 INFO ThreadId(01) running Hermes v1.10.4+542e14f
2024-11-30T01:33:14.056758Z  INFO ThreadId(01) telemetry: telemetry disabled
2024-11-30T01:33:14.056804Z  INFO ThreadId(01) rest: REST server disabled
2024-11-30T01:33:14.953483Z  INFO ThreadId(01) health_check{chain=provider-chain}: chain is healthy
2024-11-30T01:33:15.327112Z  WARN ThreadId(01) health_check{chain=zone_chain}: Failed to fetch version specs for chain 'zone_chain': ABCI query returned an error: AbciQuery { code: Err(26), log: "interchain-security-c is not ready; please wait for first block: invalid height", info: "", index: 0, key: [], value: [], proof: None, height: block::Height(0), codespace: "sdk" }
2024-11-30T01:33:15.327587Z  WARN ThreadId(01) health_check{chain=zone_chain}: Will fall back on using the node version: 0.38.15
2024-11-30T01:33:15.698096Z  WARN ThreadId(16) health_check{chain=zone_chain}: health check failed for chain 'zone_chain'
2024-11-30T01:33:15.698120Z  WARN ThreadId(16) health_check{chain=zone_chain}: reason: node at http://127.0.0.1:26658/ running chain zone_chain not caught up
2024-11-30T01:33:15.698131Z  WARN ThreadId(16) health_check{chain=zone_chain}: some Hermes features may not work in this mode!
2024-11-30T01:33:15.698156Z  WARN ThreadId(01) health_check{chain=zone_chain}: chain is not healthy: node at http://127.0.0.1:26658/ running chain zone_chain not caught up
2024-11-30T01:33:15.698222Z  INFO ThreadId(01) scan.chain{chain=provider-chain}: scanning chain...
2024-11-30T01:33:15.698292Z  INFO ThreadId(01) scan.chain{chain=provider-chain}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-11-30T01:33:15.698300Z  INFO ThreadId(01) scan.chain{chain=provider-chain}: allowed ports/channels: transfer/channel-3108
2024-11-30T01:33:15.698309Z  INFO ThreadId(01) scan.chain{chain=provider-chain}: querying allowed channels...
2024-11-30T01:33:15.698327Z  INFO ThreadId(01) scan.chain{chain=provider-chain}:scan.channel{port=transfer channel=channel-3108}: querying channel...
2024-11-30T01:33:15.699209Z ERROR ThreadId(01) scan.chain{chain=provider-chain}: failed to scan channel, reason: channel 'transfer/channel-3108' on chain 'provider-chain' is uninitialized channel=channel-3108
2024-11-30T01:33:15.699249Z  INFO ThreadId(01) scan.chain{chain=zone_chain}: scanning chain...
2024-11-30T01:33:15.699301Z  INFO ThreadId(01) scan.chain{chain=zone_chain}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-11-30T01:33:15.699309Z  INFO ThreadId(01) scan.chain{chain=zone_chain}: allowed ports/channels: transfer/channel-0
2024-11-30T01:33:15.699316Z  INFO ThreadId(01) scan.chain{chain=zone_chain}: querying allowed channels...
2024-11-30T01:33:15.699324Z  INFO ThreadId(01) scan.chain{chain=zone_chain}:scan.channel{port=transfer channel=channel-0}: querying channel...
2024-11-30T01:33:15.699830Z ERROR ThreadId(01) scan.chain{chain=zone_chain}: failed to scan channel, reason: channel 'transfer/channel-0' on chain 'zone_chain' is uninitialized channel=channel-0
2024-11-30T01:33:15.699852Z  INFO ThreadId(01) scanned chains:
2024-11-30T01:33:15.699857Z  INFO ThreadId(01) # Chain: provider-chain
# Chain: zone_chain
2024-11-30T01:33:15.699969Z  INFO ThreadId(01) spawn:chain{chain=provider-chain}: spawning Wallet worker: wallet::provider-chain
2024-11-30T01:33:15.700019Z  INFO ThreadId(01) spawn:chain{chain=zone_chain}: spawning Wallet worker: wallet::zone_chain
2024-11-30T01:33:15.700157Z ERROR ThreadId(18) spawn:chain{chain=zone_chain}:wallet{chain=zone_chain}: task aborting after encountering fatal error: failed to get key in use by the relayer: signature key not found: consumer-key: cannot find key file at '.../.hermes/keys/zone_chain/keyring-test/consumer-key.json'

Any ideas how to get unstuck here would be greatly appreciated.