bug: registry service not starting

[fmorency]

The registry service fails to start with

Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id

Configuration is

name: starship-manifestjs
version: v0.2.12

chains:
  - id: manifest-ledger-beta
    name: custom
    image: ghcr.io/liftedinit/manifest-ledger:v0.0.1-alpha.12
    home: /root/.manifest
    binary: manifestd
    prefix: manifest
    denom: umfx
    coins: 1000000000000000000umfx
    hdPath: m/44'/118'/0'/0/0
    coinType: 118
    repo: https://github.com/liftedinit/manifest-ledger
    numValidators: 1
    ports:
      rest: 1317
      rpc: 26657
  - id: cosmos-2
    name: cosmoshub
    numValidators: 1
    ports:
      rest: 1313
      rpc: 26653

#relayers:
#  - name: manifest-cosmos
#    type: hermes
#    replicas: 1
#    chains:
#      - manifest-ledger-beta
#      - cosmos-2

registry:
  enabled: true
  ports:
    rest: 8081

# kubectl get pods
NAME                             READY   STATUS     RESTARTS   AGE
cosmos-2-genesis-0               3/3     Running    0          5m45s
manifest-ledger-beta-genesis-0   3/3     Running    0          5m45s
registry-f997744df-p524s         0/1     Init:0/1   0          5m45s

# helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
starship-manifestjs     default         1               2024-09-05 09:47:04.036022695 -0400 EDT deployed        devnet-0.2.12   0.1.1 

Backend is kind.

[Anmol1696]

Hey could you share the logs from the registry pods.
Can run something like: kubectl logs registry-f997744df-p524s --all-containers

[fmorency]

Hey could you share the logs from the registry pods. Can run something like: kubectl logs registry-f997744df-p524s --all-containers

I needed to run the log command many times to get something meaningful.

➜  manifestjs git:(starship) ✗ kubectl logs -f registry-f997744df-n7wlz --all-containers=true
Error from server (BadRequest): container "registry" in pod "registry-f997744df-n7wlz" is waiting to start: PodInitializing
➜  manifestjs git:(starship) ✗ kubectl logs -f registry-f997744df-n7wlz --all-containers=true
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Error from server (BadRequest): container "registry" in pod "registry-f997744df-n7wlz" is waiting to start: PodInitializing
➜  manifestjs git:(starship) ✗ kubectl logs -f registry-f997744df-n7wlz --all-containers=true
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id
Genesis validator does not seem to be ready for: manifest-ledger-beta. Waiting for it to start...
Checking: http://manifest-ledger-beta-genesis.default.svc.cluster.local:8081/node_id
Error from server (BadRequest): container "registry" in pod "registry-f997744df-n7wlz" is waiting to start: PodInitializing
➜  manifestjs git:(starship) ✗ kubectl logs -f registry-f997744df-n7wlz --all-containers=true
Error from server (BadRequest): container "registry" in pod "registry-f997744df-n7wlz" is waiting to start: PodInitializing
➜  manifestjs git:(starship) ✗ kubectl logs -f registry-f997744df-n7wlz --all-containers=true
Error from server (BadRequest): container "registry" in pod "registry-f997744df-n7wlz" is waiting to start: PodInitializing

[Anmol1696]

This is weird. let me try it on my end as well. It seems that the custom chain is not been able to be waited for.

Could you also try this command once.

kubect exec -it manifest-ledger-beta-genesis-0 -- /bin/bash

# in the command run curl command
curl http://0.0.0.0:8081/node_id

[fmorency]

This is weird. let me try it on my end as well. It seems that the custom chain is not been able to be waited for.

Could you also try this command once.

kubect exec -it manifest-ledger-beta-genesis-0 -- /bin/bash

# in the command run curl command
curl http://0.0.0.0:8081/node_id

✗ kubectl exec -it manifest-ledger-beta-genesis-0 -- /bin/bash
Defaulted container "validator" out of: validator, exposer, faucet, init-genesis (init), init-config (init)
manifest-ledger-beta-genesis-0:/opt# curl http://0.0.0.0:8081/node_id
{"node_id":"b11dc39fbf0d04583295e12f463696291bc30802"}

[Anmol1696]

Hmm this looks correct. Not sure what is the issue now. Could be a DNS issue with Kind. Let me run it with docker-desktop.

[Anmol1696]

Ok i think i was able to re-create the issue. I will have to dig a little into this. Let me have a look and get back to you about the same.
Thanks for bringing this up.

[Anmol1696]

Could you try with docker-desktop instead for the backend?
https://docs.docker.com/desktop/kubernetes/#install-and-turn-on-kubernetes

I can see this working here.

My prediction is that the kind backend could be the issue with the DNS that we use for internal routing.

[fmorency]

I was able to make it work with docker-desktop as described in https://docs.cosmology.zone/starship/get-started/step-2

This issue looks related to kind like you mentioned.

[Anmol1696]

Prefect. Let me make sure we remove kind as a supported backend from the docs as well.