ethd keys import - sometimes not working - jq: error

[ThomasBlock]

to reproduce:

ethd cmd run --rm deposit-cli-new
ethd keys import

This works fine on
besu-teku
nethermind-lighthouse

Proceeding without slashing protection import.
Validator key was successfully imported: 0xXXXXXXX

Imported 1 keys
Skipped 0 keys

IMPORTANT: Only import keys in ONE LOCATION.
Failure to do so will get your validators slashed: Greater 1 ETH penalty and forced exit.

but on
geth-prysm
i see:

Proceeding without slashing protection import.
jq: error (at <stdin>:1): Cannot index number with string "message"
parse error: Invalid numeric literal at line 1, column 8
The pubkey was formatted wrong. Error: 

ethd terminated with exit code 1 on line 20
This happened during ethd keys import

[yorickdowne]

Can you run this with --debug at the end please. I just recently tested it and it worked. Not sure debug will show what's going wrong, but it's worth a try.

[ThomasBlock]

Can you run this with --debug at the end please. I just recently tested it and it worked. Not sure debug will show what's going wrong, but it's worth a try.

no this only works for ethd keys list --debug

ethd keys import --debug
Error: Unknown option: --debug

[yorickdowne]

Indeed. I fixed the --debug.

This is one of those "it works here" problems 😅

Can you tell me whether you have any special characters in your keystore password, and if so which? Special characters would be anything that's not standard latin letters or arabic numerals - spaces, $, or unicode characters would all fall into that.

The debug should now show you the keystore and password and attempt a jq on those, before running it for real.

[yorickdowne]

Just to be sure, please also do a git status and make sure there isn't anything that would block a git pull, which ./ethd update executes. I want to be sure this is the current version of Eth Docker

[ThomasBlock]

Just to be sure, please also do a git status and make sure there isn't anything that would block a git pull, which ./ethd update executes. I want to be sure this is the current version of Eth Docker

thank you for the feedback
i did pull, status, ethdupdate . i can now use --debug. but still no luck.
the password has no special charcters

ethd keys import --debug
[+] Creating 2/0
 ✔ Container eth-docker-consensus-1  Running                                                                                                                            0.0s 
 ✔ Container eth-docker-validator-1  Running                                                                                                                            0.0s 
WARNING - imported keys are immediately live. If these keys exist elsewhere,
you WILL get slashed. If it has been less than 15 minutes since you deleted them elsewhere,
you are at risk of getting slashed. Exercise caution

I understand these dire warnings and wish to proceed with key import (No/yes) yes
Please enter the password for your validator key(s): 
Please re-enter the password: 

No viable slashing protection import file found for 0x93XXX.
This is expected if this is a new key.
Proceeding without slashing protection import.
The keystore reads as {"crypto": {"kdf": {"function": "scrypt", "params": XXX }
And your password is YYYYY
Testing jq on these
{
  "keystores": [
    "{\"crypto\": {\"kdf\": {\"function\": \"scrypt\",XXX}"
  ],
  "passwords": [
    "YYYY"
  ]
}
Called vc:7500/eth/v1/keystores with method POST and the following data
@/tmp/apidata.txt
The token was cfc2XXX
XXX from /tmp/api-token.txt
The return code was 400 and if we had result data, here it is.
400 Bad Request
jq: error (at <stdin>:1): Cannot index number with string "message"
parse error: Invalid numeric literal at line 1, column 8
The pubkey was formatted wrong. Error: 

ethd terminated with exit code 1 on line 20
This happened during ethd keys import --debug

[yorickdowne]

That helps. Prysm returns a 400 but I am not parsing the error message because it's formatted differently. I changed the code to special-case for Prysm and just output the entire response - hopefully that gets us a step further

[ThomasBlock]

That helps. Prysm returns a 400 but I am not parsing the error message because it's formatted differently. I changed the code to special-case for Prysm and just output the entire response - hopefully that gets us a step further

Called vc:7500/eth/v1/keystores with method POST and the following data
@/tmp/apidata.txt
The token was cfcXXX from /tmp/api-token.txt
The return code was 400 and if we had result data, here it is.
400 Bad Request
parse error: Invalid numeric literal at line 1, column 8
The pubkey was formatted wrong. Error: 400

ethd terminated with exit code 1 on line 20
This happened during ethd keys import --debug

[yorickdowne]

Thanks for being patient. Once more please with ./ethd update and an import. This time I'm not trying to parse JSON on 400 at all if it's Prysm and key import.

Once there is an error message to look at, hopefully that gives us a clue.

[ThomasBlock]

Thanks for being patient. Once more please with ./ethd update and an import. This time I'm not trying to parse JSON on 400 at all if it's Prysm and key import.

Once there is an error message to look at, hopefully that gives us a clue.

The return code was 400 and if we had result data, here it is.
400 Bad Request
Bad format. Error: 400 Bad Request

ethd terminated with exit code 1 on line 20
This happened during ethd keys import --debug

[yorickdowne]

I'm stumped. The trouble is that "it works here". Let me run it here once more and compare the json to yours

[yorickdowne]

Ok, in your output you cut off so much that I can't tell. This is what a healthy file looks like in debug

{
  "keystores": [
    "{\"crypto\": {\"kdf\": {\"function\": \"scrypt\", \"params\": {\"dklen\": 32, \"n\": 262144, \"r\": 8, \"p\": 1, \"salt\": \"5d689d0466b0ce01733b81470d618c3da20824ff80a85d778bb9175ecbaabad0\"}, \"message\": \"\"}, \"checksum\": {\"function\": \"sha256\", \"params\": {}, \"message\": \"028d11cc6d4e2f39551b404d8ef7ef6b0e596539124a0b14bb7f9d9139c4d8c8\"}, \"cipher\": {\"function\": \"aes-128-ctr\", \"params\": {\"iv\": \"bff231a7c61afc1de3853dcdf46f3f78\"}, \"message\": \"5fa1e9b730fa20fc3d24217d0e58c7a97de319c1584d396422c219c9df6fba12\"}}, \"description\": \"\", \"pubkey\": \"82f80bebfd3fce29e43c06d45e4debf99541abed038e1c09d05e7d90b10764007c39b9b56b7b570a4ce98bdd8ba72094\", \"path\": \"m/12381/3600/0/0/0\", \"uuid\": \"bf618eb0-aea7-4269-a1a4-bd3f6c343348\", \"version\": 4}"
  ],
  "passwords": [
    "WowSuchPasswordMuchSecure"
  ]
}

As I can't reproduce this here, could you send me a keystore file and password you created on Holesky, that fails to import.

Also check that your Prysm version is 5.0.4, please.

[ThomasBlock]

Yes everything seems right for me. Here i created a new keystore:

ethd keys import --debug
[+] Creating 2/2
 ✔ Container eth-docker-consensus-1  Recreated                                                                                                                         15.6s 
 ✔ Container eth-docker-validator-1  Recreated                                                                                                                          0.3s 
[+] Running 2/2
 ✔ Container eth-docker-consensus-1  Started                                                                                                                            0.3s 
 ✔ Container eth-docker-validator-1  Started                                                                                                                            0.2s 
WARNING - imported keys are immediately live. If these keys exist elsewhere,
you WILL get slashed. If it has been less than 15 minutes since you deleted them elsewhere,
you are at risk of getting slashed. Exercise caution

I understand these dire warnings and wish to proceed with key import (No/yes) yes
Please enter the password for your validator key(s): 
Please re-enter the password: 

No viable slashing protection import file found for 0x9697f1aa8693d41aff34ac8126d5c8d21fdadf86028777678614e5498f2e4ca7e9530ffedb3e2f87f384bd329f764cd6.
This is expected if this is a new key.
Proceeding without slashing protection import.
The keystore reads as {"crypto": {"kdf": {"function": "scrypt", "params": {"dklen": 32, "n": 262144, "r": 8, "p": 1, "salt": "6b0ac3dec26f4da867fdd7793a7cf795d61ba46d9e63b565695379ff394f443e"}, "message": ""}, "checksum": {"function": "sha256", "params": {}, "message": "36b69b2ede59f0a9ba0fdd7c8b804215570192e48b9a0783e1d5cb1bdb2fec68"}, "cipher": {"function": "aes-128-ctr", "params": {"iv": "69610a50a953bff7bad4bd4c1cb2f762"}, "message": "25dade826e152c11e4b177a4f909e7ecedee11225046611de7cd379ca4794061"}}, "description": "", "pubkey": "9697f1aa8693d41aff34ac8126d5c8d21fdadf86028777678614e5498f2e4ca7e9530ffedb3e2f87f384bd329f764cd6", "path": "m/12381/3600/0/0/0", "uuid": "40c8f6f6-60a9-48af-abf5-b82293c419e6", "version": 4}
And your password is TestPassword1234
Testing jq on these
{
  "keystores": [
    "{\"crypto\": {\"kdf\": {\"function\": \"scrypt\", \"params\": {\"dklen\": 32, \"n\": 262144, \"r\": 8, \"p\": 1, \"salt\": \"6b0ac3dec26f4da867fdd7793a7cf795d61ba46d9e63b565695379ff394f443e\"}, \"message\": \"\"}, \"checksum\": {\"function\": \"sha256\", \"params\": {}, \"message\": \"36b69b2ede59f0a9ba0fdd7c8b804215570192e48b9a0783e1d5cb1bdb2fec68\"}, \"cipher\": {\"function\": \"aes-128-ctr\", \"params\": {\"iv\": \"69610a50a953bff7bad4bd4c1cb2f762\"}, \"message\": \"25dade826e152c11e4b177a4f909e7ecedee11225046611de7cd379ca4794061\"}}, \"description\": \"\", \"pubkey\": \"9697f1aa8693d41aff34ac8126d5c8d21fdadf86028777678614e5498f2e4ca7e9530ffedb3e2f87f384bd329f764cd6\", \"path\": \"m/12381/3600/0/0/0\", \"uuid\": \"40c8f6f6-60a9-48af-abf5-b82293c419e6\", \"version\": 4}"
  ],
  "passwords": [
    "TestPassword1234"
  ]
}
Called vc:7500/eth/v1/keystores with method POST and the following data
@/tmp/apidata.txt
The token was cfc2031a3be9c9e5943f844e1cc4f5cef2e450304a6ad4845db89f1b804ad2a7
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.k4iqMIiGSyXBUzTgTM8ppw2nGtPP36rvzI_xMVpy1zU from /tmp/api-token.txt
The return code was 400 and if we had result data, here it is.
400 Bad Request
Bad format. Error: 400 Bad Request

ethd terminated with exit code 1 on line 20
This happened during ethd keys import --debug

ethd version
This is Eth Docker v2.11.0.0

ssvnode version v1.3.7-cc6f7d1402b9be88df4431a597e729e882c628b1

beacon-chain version Prysm/v5.0.4/3b184f43c86baf6c36478f65a5113e7cf0836d41. Built at: 2024-06-21 00:23:34+00:00

validator version Prysm/v5.0.4/3b184f43c86baf6c36478f65a5113e7cf0836d41. Built at: 2024-06-21 00:24:43+00:00

[yorickdowne]

Ah, there's a wrinkle. I've been testing this with prysm.yml, and you have something set up where it's an SSV node and a solo node both

Can you give me what your COMPOSE_FILE in .env looks like, please. I'm still harboring hope that I can reproduce here.

The key file you gave me is fine and imports on my Prysm here

[ThomasBlock]

Ah okay i see..

COMPOSE_FILE=geth.yml:prysm.yml:ethdo.yml:grafana.yml:grafana-shared.yml:el-shared.yml:cl-shared.yml:ee-shared.yml:mev-boost.yml:deposit-cli.yml:ssv.yml:ssv-dkg.yml

[yorickdowne]

Roger will take a look.

ee-shared.yml doesn’t seem necessary, that’s the engine api and nothing but prysm should access it

what are you using cl-shared and el-shared for? Those are the REST and RPC API respectively

[yorickdowne]

Using that COMPOSE_FILE, I can still successfully import the key

Can you think of anything that's different in your .env from default, that may impact this?

[ThomasBlock]

okay strange.. i just change ports..

SHARE_IP=192.168.X.Y
EL_P2P_PORT=20062
CL_P2P_PORT=20063
PRYSM_PORT=20063
PRYSM_UDP_PORT=20063
CL_QUIC_PORT= 20064
SSV_P2P_PORT=20060
SSV_P2P_PORT_UDP=20060
SSV_DKG_PORT=20061

nano ssv.yml
    ports:
      - 15000:15000/tcp

[yorickdowne]

That should not have any impact, but let me test it here

[yorickdowne]

Entirely unsurprisingly, that had no impact.

I have no idea what's different on your system that's causing Prysm to reject the file

[yorickdowne]

Another user reported this with just an ./ethd keys list and Prysm. They did not troubleshoot and instead set everything up from scratch again, and the issue resolved.

This means it's intermittent and likely connected to exactly what's in the Prysm volume. Which makes it hard to impossible to track down unless the issue already shows.

If you are keen to pursue this, please raise it with the Prysm team, tell them it's likely specific to the contents of the VC volume and cannot be reproduced in a fresh setup, and see what they can do to narrow it down and find root cause

[nguyenhoaibao]

I run the ethd cmd run --rm create-wallet to create Prysm wallet already, but got this error while trying to import the validator key:

Called vc:7500/eth/v1/keystores with method POST and the following data
@/tmp/apidata.txt
The token was 0xb5e1010dcea7b777568f0e47a165bfc3452f29799348430e9e83f7344855daf8 from /tmp/api-token.txt
The return code was 503 and if we had result data, here it is.
{"message":"Prysm Wallet not initialized. Please create a new wallet.","code":503}
Unexpected return code 503. Result: {"message":"Prysm Wallet not initialized. Please create a new wallet.","code":503}

My ethd version:

This is Eth Docker v2.12.1.0

beacon-chain version Prysm/v5.1.0/b8cd77945df2b8fa8fe50520df0495309a52c2f3. Built at: 2024-08-20 17:14:40+00:00

validator version Prysm/v5.1.0/b8cd77945df2b8fa8fe50520df0495309a52c2f3. Built at: 2024-08-20 17:15:36+00:00

Geth
Version: 1.14.8-stable
Git Commit: a9523b6428238a762e1a1e55e46ead47630c3a23
Architecture: amd64
Go Version: go1.22.6
Operating System: linux
GOPATH=
GOROOT=

prometheus, version 2.53.2 (branch: HEAD, revision: 6e971a7dc905696d4bc4ffa150bf282fcfac5fa9)
  build user:       root@363b0aa99939
  build date:       20240809-14:55:04
  go version:       go1.22.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Grafana Version 11.1.4 (commit: 13173c9874af312fe75545f52aa6539af02076ac, branch: v11.1.x)

My COMPOSE_FILE:

prysm.yml:geth.yml:grafana.yml:grafana-shared.yml:deposit-cli.yml

[yorickdowne]

That’s different, and the error message tells you what’s going on: You don’t have a wallet, so create one. That’s specific to Prysm. Run ethd keys and the help screen shows you the command to initialize / create the Prysm wallet

[nguyenhoaibao]

@yorickdowne the issue is I run ethd cmd run --rm create-wallet already. When I tried to run ethd keys get-prysm-wallet, I can get the generated password too. But when I tried to import validator keys, I got the error above :).

$ ethd keys create-prysm-wallet
Wallet password created
An error occurred while attempting to create a Prysm wallet
time="2024-08-29 13:22:44" level=info msg="Running on the Holesky Beacon Chain Testnet" prefix=flags
time="2024-08-29 13:22:44" level=fatal msg="Could not create a wallet" error="a wallet already exists at this location. Please input an alternative location for the new wallet or remove the current wallet" prefix=wallet

$ ethd keys import --path /home/validator/.eth/validator_keys
I understand these dire warnings and wish to proceed with key import (No/yes) yes
Please enter the password for your validator key(s):
Please re-enter the password:

No viable slashing protection import file found for 0xa55479d63db5aa858a03c12b8a16b030a0903e492593105e1fa2fb1a0203c5ac9068f07a498d980a87daae3bbb5acece.
This is expected if this is a new key.
Proceeding without slashing protection import.
Unexpected return code 503. Result: {"message":"Prysm Wallet not initialized. Please create a new wallet.","code":503}

ethd terminated with exit code 1 on line 20
This happened during ethd keys import --path /home/validator/.eth/validator_keys

[yorickdowne]

Definitely unexpected. Please open a new issue for it so we’re not cluttering this one

also, prysm may require a restart after wallet creation so its keymanager api picks it up

[victorelec14]

I have a similar problem.

Here are the steps:

1. create a new wallet to use as a CSM validator.
2. create the validator key and everything is perfect.
3. create 2 new keys with the same wallet and in another index (because the first one already existed).
4. I try to import the keys (ethd keys import) and I get error.

The error when importing is that the keys already exists and then does not continue, but when making list I also have error.

I don't know if it has something to do with it but first my validator was Lighthouse and then I changed to Prysm.

Thanks

COMPOSE_FILE=prysm.yml:geth.yml:grafana.yml:grafana-shared.yml:mev-boost.yml:deposit-cli.yml:web3signer.yml:cl-shared.yml:el-shared.yml:ee-shared.yml

xxxxxxxxx@eth-docker:~/eth-docker$ ethd keys list        
[+] Creating 2/0
 ✔ Container eth-docker-consensus-1  Running                                                                                                                                              0.0s 
 ✔ Container eth-docker-validator-1  Running                                                                                                                                              0.0s 
Validator public keys loaded into web3signer
0xa6aef24bca2998c088xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Unexpected return code 400. Result: 400 Bad Request

ethd terminated with exit code 1 on line 20
This happened during ethd keys list

victorelec14@eth-docker-consensus:~/eth-docker$ ethd keys list --debug
[+] Creating 2/0
 ✔ Container eth-docker-consensus-1  Running                                                                                                                                              0.0s 
 ✔ Container eth-docker-validator-1  Running                                                                                                                                              0.0s 
Called web3signer:9000/eth/v1/keystores with method GET and the following data
This was a call without data
The token was NIL from /tmp/api-token.txt
The return code was 200 and if we had result data, here it is.
{"data":[{"validating_pubkey":"0xa6aef24bca2998c088b525c90df51xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","derivation_path":"m/12381/3600/0/0/0","readonly":false}]}
Validator public keys loaded into web3signer
0xa6aef24bca2998c088b525c90df516xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Called vc:7500/eth/v1/remotekeys with method GET and the following data
This was a call without data
The token was e973a86ccb1857dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
eyJhbGciOiJIUzI1NiIsInR5cCIxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx from /tmp/api-token.txt
The return code was 400 and if we had result data, here it is.
400 Bad Request
Unexpected return code 400. Result: 400 Bad Request

ethd terminated with exit code 1 on line 20
This happened during ethd keys list --debug

$ ethd version  
This is Eth Docker v2.14.0.2

beacon-chain version Prysm/v5.1.2/944f94a9bf6cbd19699b319917499fd7262e2f73. Built at: 2024-10-16 19:15:24+00:00

validator version Prysm/v5.1.2/944f94a9bf6cbd19699b319917499fd7262e2f73. Built at: 2024-10-16 19:16:39+00:00

Geth
Version: 1.14.12-stable
Git Commit: 293a300d64be3d9a1c2cc92c26fcff4089deadcd
Git Commit Date: 20241119
Architecture: amd64
Go Version: go1.23.3
Operating System: linux
GOPATH=
GOROOT=

2024-11-24 09:35:56.814+00:00 | main | INFO  | Web3SignerApp | Web3Signer has started with args --version
2024-11-24 09:35:56.846+00:00 | main | INFO  | Web3SignerApp | Version = web3signer/v24.6.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17
web3signer/v24.6.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17

PostgreSQL 16.5 (Debian 16.5-1.pgdg120+1)

INFO[0000] mev-boost v1.8.1                             

prometheus, version 3.0.0 (branch: HEAD, revision: c5d009d57fcccb7247e1191a0b10d74b06295388)
  build user:       root@aa286d0eb00a
  build date:       20241114-16:40:43
  go version:       go1.23.3
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Grafana Version 11.3.1 (commit: 9225f4a1cbd1cfe8b69f1aa2d62309a9700533a5, branch: HEAD)

[victorelec14]

confirmed, the problem is only with Prysm, I just switched to lighthouse again and the commands (list and import) work correctly.