Skip to content

Latest commit

 

History

History
457 lines (343 loc) · 20.1 KB

walkthrough.md

File metadata and controls

457 lines (343 loc) · 20.1 KB
Raw

AWS Nitro Enclave Blockchain Validator (Web3Signer) Solution Walkthrough

Solution overview

Deployment overview

Deployment Overview

Application overview and bootstrapping flow

Application Architecture

Prerequisites

For this walkthrough, you must have the following prerequisites:

Note that the solution is only compatible with Python 3.9.

Deploy the solution with AWS CDK

  • virtual environments (venv) are recommended working with Python

  • AWS CDK per default leverages virtual environments.

    npm install -g aws-cdk && cdk –version

To install the sample application, complete the following steps:

  1. Install the AWS CDK and test the AWS CDK CLI:

    npm install -g aws-cdk && cdk –version

  2. Download the code from the GitHub repo and change to the new directory:

    git clone https://github.com/aws-samples/aws-nitro-enclave-blockchain-validator

  3. Change to the aws-nitro-enclave-blockchain-validator repository:

    cd aws-nitro-enclave-blockchain-validator

  4. Create a virtualenv. For macOS and Linux, use the command below. For other operating systems, refer to Activating virtualenv

    python3 -m venv .venv

  5. After the init process completes and the virtualenv is created, activate your venv

    source .venv/bin/activate

  6. Install the dependencies using the Python package manager:

    pip install -r requirements.txt

  7. Build the required binaries for Nitro Enclaves. This step requires a valid local Docker environment.

    ./scripts/build_kmstool_enclave_cli.sh

    After you run this step, a new folder (application/eth2/enclave/kms) is available that contains the required Nitro Enclaves artifacts.

    If you encounter a problem with the build_kmstool_enclave_cli.sh step, such as a network connectivity issue, you can turn on the debug output of the script by changing set +x to set -x inside the script.

    For additional information, refer to the GitHub repo.

  8. (Optional) If you have deployed the validator key table and KMS key using Generate validator keys for Ethereum with trusted code in AWS Lambda and AWS Signer, modify the code in app.py to specify the kms_arn and validator_key_table_arn. Else, skip this step.

  9. Specify the AWS region and account for your deployment:

    export CDK_DEPLOY_REGION=us-east-1
export CDK_DEPLOY_ACCOUNT=$(aws sts get-caller-identity | jq -r '.Account')

  10. Deploy the sample code with the AWS CDK CLI:

cdk deploy prodNitroValidator -O output.json

AWS CDK asks for an additional confirmation to deploy the solution, as shown in the following screenshot. CDK Confirmation

  1. Enter y to confirm.

After the deployment is complete, the terminal shows us the additional parameters like the Auto Scaling group name. These values have also been written to output.json in JSON format.

CDK Deployment Complete

Create TLS Artifacts and Eth2 Validator Keys and start Web3Signer Enclave

  1. Create an environment variable pointing out the CloudFormation stack name:

    export CF_STACK_NAME=$(jq -r '. |= keys | .[0]' output.json)
echo $CF_STACK_NAME

  2. Follow either one of the instruction below:

    Option 1: If you have deployed the validator key table and KMS key using Generate validator keys for Ethereum with trusted code in AWS Lambda and AWS Signer, follow the instructions in that repository to generate and load the validator key.

    Retrieve the CloudFormation stack UUID

    CF_STACK_ARN=$(aws cloudformation describe-stacks --stack-name $CF_STACK_NAME --query "Stacks[0].StackId" --output text)
echo $CF_STACK_ARN | sed 's/.*\///'

    You will receive an output similar to 40eec9d0-6a37-11ed-bce8-02f9df3f099e

    In the DynamoDB table, observe that the web3signer_uuid column value is none for each row

    Table

    Insert the UUID as the column value so that the key(s) are loaded into web3signer Nitro Enclave in the subsequent step. In the example below, the key starting with 8c38 will be loaded into web3signer while the key starting with a179 will not be loaded.

    Key Set

    Option 2: If you have deployed the validator key table and KMS key as part of this CDK application, you need to run a script to generate the validator keys and load into the DynamoDB table. Note that this method should only be used in non-production environments. You should use a secure method as depicted in Option 1 for production environments.

    Copy and paste the code below into a terminal and hit enter/return to generate and load validators key into DynamoDB table. The CloudFormation Stack ID will be automatically used as the web3signer_uuid:

    cd scripts/load_validator_keys
pip3 install -r requirements.txt

export DDB_TABLE_NAME=$(cat ../../output.json | jq -r ".${CF_STACK_NAME}.ValidatorKeysTableName")
echo $DDB_TABLE_NAME

export KMS_KEY_ARN=$(cat ../../output.json | jq -r ".${CF_STACK_NAME}.KMSKeyARN")
echo $KMS_KEY_ARN

python3 load_validator_keys.py
cd ../..

    If successful, the public key portion of the just created validator key will be printed on the terminal: Validator Public Key Print

  3. Generate the key policy and apply it to the KMS key. The output.json file created during the cdk deploy step needs to be passed as an input parameter. This step can take up to 15 seconds.

    Option 1: If you have deployed the validator key table and KMS key using Generate validator keys for Ethereum with trusted code in AWS Lambda and AWS Signer, issue the following command:

    # "secure-keygen" refers to the CloudFormation stack name
./scripts/generate_key_policy.sh output.json secure-keygen

    The generated KMS key policy will look like this:

     {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "Enable decrypt from enclave",
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::123456789123:role/prodNitroValidator-InstanceSSMCBFA3CF0-USZMVEEATKPJ"
       },
       "Action": "kms:Decrypt",
       "Resource": "*",
       "Condition": {
         "StringEqualsIgnoreCase": {
           "kms:RecipientAttestation:ImageSha384": "a2dcbd226f77783f679c84a5335aad788043c69d4844b62b26b18d0c371114dd9cabfefe6be21383b45bd81a381966d9"
         }
       }
     },
     {
       "Sid": "Enable encrypt from lambda",
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::123456789123:role/prodNitroValidator-NitroInvokeLambdaServiceRoleEF7-FNGK7IQ19294"
       },
       "Action": [
         "kms:Encrypt",
         "kms:GenerateDataKey*"
       ],
       "Resource": "*"
     },
     {
       "Sid": "Enable encrypt from keygen lambda",
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::123456789123:role/secure-keygen-ValidatorKeyGenFunctionRole-16FJ049ZXJB2"
       },
       "Action": [
         "kms:Encrypt",
         "kms:GenerateDataKey*"
       ],
       "Resource": "*"
     },
     {
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::123456789123:root"
       },
       "Action": [
         "kms:Create*",
         "kms:Describe*",
         "kms:Enable*",
         "kms:List*",
         "kms:Put*",
         "kms:Update*",
         "kms:Revoke*",
         "kms:Disable*",
         "kms:Get*",
         "kms:Delete*",
         "kms:ScheduleKeyDeletion",
         "kms:CancelKeyDeletion",
         "kms:GenerateDataKey*",
         "kms:TagResource",
         "kms:UntagResource"
       ],
       "Resource": "*"
     }
   ]
 }

    Option 2: If you have deployed the validator key table and KMS key as part of this CDK application, issue the following command:

    ./scripts/generate_key_policy.sh output.json

    The generated KMS key policy will look like this:

     {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "Enable decrypt from enclave",
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::123456789123:role/prodNitroValidator-InstanceSSMCBFA3CF0-USZMVEEATKPJ"
       },
       "Action": "kms:Decrypt",
       "Resource": "*",
       "Condition": {
         "StringEqualsIgnoreCase": {
           "kms:RecipientAttestation:ImageSha384": "a2dcbd226f77783f679c84a5335aad788043c69d4844b62b26b18d0c371114dd9cabfefe6be21383b45bd81a381966d9"
         }
       }
     },
     {
       "Sid": "Enable encrypt from lambda",
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::123456789123:role/prodNitroValidator-NitroInvokeLambdaServiceRoleEF7-FNGK7IQ19294"
       },
       "Action": [
         "kms:Encrypt",
         "kms:GenerateDataKey*"
       ],
       "Resource": "*"
     },
     {
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::123456789123:root"
       },
       "Action": [
         "kms:Create*",
         "kms:Describe*",
         "kms:Enable*",
         "kms:List*",
         "kms:Put*",
         "kms:Update*",
         "kms:Revoke*",
         "kms:Disable*",
         "kms:Get*",
         "kms:Delete*",
         "kms:ScheduleKeyDeletion",
         "kms:CancelKeyDeletion",
         "kms:GenerateDataKey*",
         "kms:TagResource",
         "kms:UntagResource"
       ],
       "Resource": "*"
     }
   ]
 }

  4. To use the prepared KMS key policy, open the AWS Management Console and navigate to the KMS section. Inside the KMS console, navigate to the KMS customer managed key that was created along with the other components. The key ID was printed out at the end of the deployment step prodNitroSigner.KMSKeyARN. KMS Keys

    On the Key policy tab, choose Edit.

    KMS Keys Edit Button

    Copy the prepared enclave key policy JSON document and replace the standard KMS key policy, then choose Save changes. KSM Keys Policy

  5. Use the Lambda set_tls_key function to store TLS key artifacts (password, keystore) in DynamoDB:

    FUNCTION_ARN=$(cat output.json | jq -r ".${CF_STACK_NAME}.LambdaFunctionArn")
aws lambda invoke --function-name $FUNCTION_ARN \
   --cli-binary-format raw-in-base64-out \
   --payload '{"operation": "set_tls_key"}' lambda-output

    If successful, you will get back a status code 200 from Lambda as shown in the following screenshot:

    {
 "StatusCode": 200,
 "ExecutedVersion": "$LATEST"
}

  6. Start the signing_service daemon on all EC2 instances of the ASG (enclave, watchdog) via the following command:

    ./scripts/start_signing_service.sh output.json

    It's important that the output.jsonfile does correspond with the most recent deployment. The script will take a few minutes to run. Output will be similar to:

    i-039c401dd22080c35:
● nitro-signing-server.service - Nitro Enclaves Signing Server Loaded: loaded (/etc/systemd/system/nitro-signing-server.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2023-02-17 15:50:53 UTC; 17s ago Main PID: 9409 (python3) Tasks: 5 Memory: 26.1M CGroup: /system.slice/nitro-signing-server.service ├─9409 python3 /home/ec2-user/app/watchdog.py └─9444 /bin/nitro-cli run-enclave --cpu-count 2 --memory 3806 --eif-path /home/ec2-user/app/server/signing_server.eif --enclave-cid 16 Feb 17 15:50:53 ip-10-0-172-57.ec2.internal systemd[1]: Started Nitro Enclaves Signing Server. Feb 17 15:50:53 ip-10-0-172-57.ec2.internal watchdog.py[9409]: Start allocating memory... Feb 17 15:50:54 ip-10-0-172-57.ec2.internal watchdog.py[9409]: Started enclave with enclave-cid: 16, memory: 3806 MiB, cpu-ids: [1, 3]
i-0442edc19a468a5a8:
● nitro-signing-server.service - Nitro Enclaves Signing Server Loaded: loaded (/etc/systemd/system/nitro-signing-server.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2023-02-17 15:50:53 UTC; 17s ago Main PID: 10467 (python3) Tasks: 5 Memory: 26.2M CGroup: /system.slice/nitro-signing-server.service ├─10467 python3 /home/ec2-user/app/watchdog.py └─10500 /bin/nitro-cli run-enclave --cpu-count 2 --memory 3806 --eif-path /home/ec2-user/app/server/signing_server.eif --enclave-cid 16 Feb 17 15:50:53 ip-10-0-203-84.ec2.internal systemd[1]: Started Nitro Enclaves Signing Server. Feb 17 15:50:53 ip-10-0-203-84.ec2.internal watchdog.py[10467]: Start allocating memory... Feb 17 15:50:54 ip-10-0-203-84.ec2.internal watchdog.py[10467]: Started enclave with enclave-cid: 16, memory: 3806 MiB, cpu-ids: [1, 3]
{
    "Version": 2,
    "Tier": "Standard"
}

(17/02/2023 16:51:15) service has been started and is healthy

    By the command you issued, you just triggered the following actions:

    • start signing_service
    • set initiated SSM flag to true

    Bootstrapping flow

    Architecture

    1. Signing systemd service will read encrypted web3signer config assets from DynamoDB
    2. Crypto assets will be passed into enclave along with AWS credentials
    3. enclave_init process will validate config assets.
    4. kmstool-enclave-cli will be used to decrypt the BLS12-381 private key and related information, required for Ethereum 2 validator signatures. The aws-nitro-enclaves-sdk-c based tool uses the provided vsock-proxy to send cryptographic attestation enabled decrypt request to AWS KMS.
    5. AWS KMS will decrypt the configuration and encrypt it with the enclaves public key and send it back to kmstool-enclave-cli. The cli will then decrypt the configuration using the the enclaves private key and store it in the enclaves ephemeral storage.
    6. enclave_init will start the web3signer process. If the process has been started successfully, enclave_init will also start a separate vsock-proxy process inside the enclave.
    7. The validator client will need have web3signer configured as its remote signing solution, e.g. Lighthouse Remote Signing. When configured correctly, the validator client can execute an TLS handshake with the endpoint exposed by http_server.
    8. http_server acts like a TCP proxy that translates between AF_INET and AF_VSOCK. It accepts incoming requests from the validator client and forwards each TCP packet to proxy running inside the enclave via the vsocket connection. The proxy inside the enclave acts in reverse and translates AF_VSOCK to AF_INET. As a consequence of this setup, the proxy mechanism is transparent to the validator client that wants to establish an https connection with the web3signer process running inside the enclave. The connection, once established, is fully end-to-end encrypted. Furthermore, TLS termination is being done by the web3signer process running inside the enclave. As a result of that setup, the validator client is able to ensure the validity of the web3signer endpoint by controlling its X509 certificate.

  7. Test the Web3Signer deployment from the Lambda function console to confirm that the Web3Signer is service is accessible remotely. On the Lambda console, choose the newly created Lambda function starting with the name prodNitroSigner-NitroInvokeLambdaXXXXXXX-XXXXXXX.

    Lambda Console

    After you choose the function, choose the Test tab.

    Lambda Console Test

    To test the web3signer status, invoke the Lambda function via a test, passing the following JSON snippet as the payload.

    {
  "operation": "web3signer_status"
}

    Use status for Event name, choose Save, then choose Test.

    Lambda Console Test Event

    If the web3signer_status operation explained previously is successful, web3signer will respond with the text OK, which will also be printed out on the Lambda console. Internally this call leverages the web3signer upcheck endpoint to confirm that the web3signer process is connected and running.

    Lambda Console Status Response

    In the same way as described above, web3signer_public_keys operation can be used from the Lambda console, which returns a hex-encoded list of Ethereum 2 BLS public keys for the private keys that have been loaded into web3signer. Internally this call is mapped to the Public Key endpoint of the web3signer.

    {
  "operation": "web3signer_public_keys"
}

    If successful, the response will look similar to this:

    Lambda Console Public Keys Response

    [
 "0xb0e263026696fa9bce2c03ab2dced4e15341ed69f7636db6adb05080c32b470950bd40a7810cb2cc6bcf6cfeed5548fa"
]

  8. Now that the web3signer inside Nitro Enclave is up and running, you can configure the validator client to use remote signing. Note that validator client provisioning and setup are not part of this CDK application and you should follow the instruction of your preferred validator client software such as Lighthouse. The following is an example of Lighthouse remote-signing validator_definitions.yml configuration file:

    - enabled: true
voting_public_key: "INSERT_PUBLIC_KEY"
type: web3signer
url: "INSERT_URL_OF_WEB3SIGNER_LOAD_BALANCER"
root_certificate_path: /path/to/web3signer-tls-cert.pem

    To get the web3signer load balancer FQDN, issue:

    cat output.json | jq -r ".${CF_STACK_NAME}.SignerELBFQDN"

    You can download the TLS root certificate from the TLS key DynamoDB table. You can find the table name by issuing the following command

    cat output.json | jq -r ".${CF_STACK_NAME}.TLSKeysTableName"

Clean up

To avoid incurring future charges, delete the resources using the AWS CDK with the following command:

cdk destroy prodNitroValidator