Skip to content

Commit

Permalink
Update vault.md
Browse files Browse the repository at this point in the history
  • Loading branch information
romdalf authored Sep 24, 2024
1 parent 6102644 commit df7fc8a
Showing 1 changed file with 150 additions and 55 deletions.
205 changes: 150 additions & 55 deletions docs/vault.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,36 @@

## Automated deployment

The folder ```scripts/prd/vault``` includes a script ```env4vault.sh``` deploys:
- a HashiCorp Vault dev instance
- a Kind instance running Kubernetes v1.29.2
- the HashiCorp Vault kleidi and Kubernetes Auth configuration
- a kleidi-kms-plugin pod connecting
- a kube-api restart with the ```encryption-config.yaml```configuration
- a series of Secrets test to validate the configuration
The folder ```scripts/prd/vault``` includes a script ```env4vault.sh``` leveraging ```Podman``` and ```Kind``` to:
- clean any previous instances of a previous deployment
- deploy:
- a HashiCorp Vault dev instance
- a Kind instance running Kubernetes v1.29.2
- the latest release of Kleidi
- configure:
- the HashiCorp Vault kleidi and Kubernetes Auth configuration
- a kleidi-kms-plugin system critical pod
- a kube-api restart with the ```encryption-config.yaml```configuration
- validate:
- by creating 1001 pre-deployment secrets
- by creating 1001 post-deployment secrets resulting in encrypted data in etcd
- by replacing the pre-deployment secrets with their encrypted version in etcd
- by rotating the Vault key
- by replacing 1001 pre-deployment encrypted secrets with the rotated key
- If any of the above fails, the script will exit with the related error.

This requires to have the following install:
- Podman
- Kind
- HashiCorp Vault CLI
- Kind
- kubectl

***Depending on the Linux distro and container run time, the safe bet is to run the script in sudo/root mode. The main root cause is related the container to host communication that might be denied.***

## Manual deployment

This implementation includes the initialization of an external HashiCorp Vault with a Transit Key Engine.
Download the HashiCorp Vault binary or install it with ```brew```, then run the following command:
Download the HashiCorp Vault binary or install it with ```brew```.

Run the following command to start of dev/test instance:
```
vault server -dev -dev-root-token-id=kleidi-demo --dev-listen-address=0.0.0.0:8200
```
Expand Down Expand Up @@ -114,7 +124,7 @@ supports_signing false
type aes256-gcm96
```

Create an ACL policy to control access the engine:
Create an ACL policy to control access to the Vault transit key engine:
```
vault policy write kleidi configuration/vault/vault-policy.hcl
```
Expand All @@ -140,6 +150,10 @@ path "transit/keys/kleidi" {
path "auth/token/lookup-self" {
capabilities = ["read"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
```

## Kind Deployment
Expand Down Expand Up @@ -252,7 +266,7 @@ Expected output:

## kleidi Deployment

To provide a secure connectivity between ```kleidi``` running on kubernetes and ```vault```, a ```ServiceAccount``` with a token and RBAC is configured.
To provide secure connectivity between "kleidi" running on Kubernetes and "vault", a "ServiceAccount" with a token and RBAC is configured.

Create the ```ServiceAccount``` including its token and RBAC for Kleidi and Vault:
```
Expand Down Expand Up @@ -318,7 +332,7 @@ k8shost=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clus
```

***Warning***
The k8shost variable might output ```127.0.0.1``` which will result in a dial back failure from Vault when verifying the kubernetes authentication via the provided certificate. Correct value should be a FQDN or ```kubernetes.default.svc.cluster.local```. In the above case ```k8shost=https://kubernetes.default.svc.cluster.local:port``` is a valide option.
The k8shost variable might output ```127.0.0.1```, resulting in a dial-back failure from Vault when verifying the Kubernetes authentication via the provided certificate. The correct value should be an FQDN, the IP address, or ``` Kubernetes.default.svc.cluster.local```. In the above case ```k8shost=https://kubernetes.default.svc.cluster.local:port``` is a valide option.

Injecting the token and certificate to connect to HashiCorp Vault and the Transit engine:
```
Expand Down Expand Up @@ -364,7 +378,7 @@ spec:
hostNetwork: true
containers:
- name: kleidi-kms-plugin
image: ghcr.io/beezy-dev/kleidi-kms-plugin:vault-1283a8e
image: ghcr.io/beezy-dev/kleidi-kms-plugin:vault-781c292
imagePullPolicy: Always
args:
- -provider=hvault
Expand Down Expand Up @@ -398,7 +412,7 @@ spec:
```
***NOTE***
The above provides the last tested images. It is advised not to change it.
The above provides the last released images. Please don't change it.
Then modify the original ```vault-encryption-config.yaml```:
```YAML
Expand Down Expand Up @@ -428,60 +442,141 @@ resources:
- identity: {}
```

This should triggere a restart of the kubernetes API server.
This should trigger a restart of the Kubernetes API server.

## Encryption/Decryption Test

To validate there is now encryption, create a post-deployment secret:
```
kubectl create secret generic postkleidi-secret -n default --from-literal=mykey=mydata
kubectl create secret generic postkleidi -n default --from-literal=mykey=mydata
```
Expected output:
```
secret/postkleidi-secret created
secret/postkleidi created
```
```
kubectl -n kube-system exec etcd-kleidi-vault-control-plane -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/etc/kubernetes/pki/etcd/ca.crt' ETCDCTL_CERT='/etc/kubernetes/pki/etcd/server.crt' ETCDCTL_KEY='/etc/kubernetes/pki/etcd/server.key' ETCDCTL_API=3 etcdctl get /registry/secrets/default/postkleidi-secret" | hexdump -C
kubectl -n kube-system exec etcd-kleidi-vault-prd-control-plane -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/etc/kubernetes/pki/etcd/ca.crt' ETCDCTL_CERT='/etc/kubernetes/pki/etcd/server.crt' ETCDCTL_KEY='/etc/kubernetes/pki/etcd/server.key' ETCDCTL_API=3 etcdctl get /registry/secrets/default/postkleidi" | hexdump -C
```
Expected output:
```
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 70 6f 73 74 6b 6c |s/default/postkl|
00000020 65 69 64 69 2d 73 65 63 72 65 74 0a 6b 38 73 3a |eidi-secret.k8s:|
00000030 65 6e 63 3a 6b 6d 73 3a 76 32 3a 6b 6c 65 69 64 |enc:kms:v2:kleid|
00000040 69 2d 6b 6d 73 2d 70 6c 75 67 69 6e 3a 0a aa 02 |i-kms-plugin:...|
00000050 98 45 a9 2d 64 9c 71 0e ad cb c5 56 22 8a 3a 0e |.E.-d.q....V".:.|
00000060 e9 84 d3 ce 57 24 6b 99 c2 2d 6d 87 bf 67 37 2e |....W$k..-m..g7.|
00000070 71 bd 0c a0 69 a6 56 ae 13 67 f3 fc f2 5c 81 66 |q...i.V..g...\.f|
00000080 2e 8f 62 fd ef ec 71 46 30 05 eb e4 a6 0d 54 fc |..b...qF0.....T.|
00000090 a5 0b 6e 6b 4d 81 a3 ab 5a e6 0c ff 65 3c 0e 16 |..nkM...Z...e<..|
000000a0 c5 4e 6b 0e 3d b4 6e b9 b6 90 7d 53 2b 66 ba d9 |.Nk.=.n...}S+f..|
000000b0 f0 71 00 0e 8a 3a d4 44 d9 3b 78 7c b6 dc dd b0 |.q...:.D.;x|....|
000000c0 ea 53 04 c3 d5 31 f2 10 06 3a 39 e8 d1 8b 37 5d |.S...1...:9...7]|
000000d0 25 fd f5 ee 00 98 e7 45 64 34 a3 3f f8 94 aa 9b |%......Ed4.?....|
000000e0 ea 8e 5f 0b bf b3 84 e4 71 7e 57 b0 50 5a d3 58 |.._.....q~W.PZ.X|
000000f0 61 b4 77 71 9a 1f a1 e9 33 e3 b7 b1 e6 12 32 fd |a.wq....3.....2.|
00000100 97 91 48 84 cc 27 a5 b3 cf 55 d9 45 f7 6f 0f 50 |..H..'...U.E.o.P|
00000110 06 4f ba 59 1b e2 2f 47 3e 9d c8 f7 8d c6 b2 ea |.O.Y../G>.......|
00000120 7e 61 1f 91 c5 44 90 34 5a dc 8b 22 41 2a 8e f0 |~a...D.4Z.."A*..|
00000130 2e db 86 f5 c7 ea 23 07 56 10 d1 9a 89 07 23 58 |......#.V.....#X|
00000140 be cc 0f f3 d0 fd a1 d8 57 74 6a 24 7c 10 91 85 |........Wtj$|...|
00000150 ee 19 f7 ef fd ea 9f 1d a6 93 d6 37 1c 90 6e d3 |...........7..n.|
00000160 e5 9d 84 90 96 18 78 af 3f 24 49 b9 81 2a fc e4 |......x.?$I..*..|
00000170 be b6 bb e4 4d 78 c2 bc cb d4 12 11 6b 6c 65 69 |....Mx......klei|
00000180 64 69 2d 6b 6d 73 2d 70 6c 75 67 69 6e 1a 59 76 |di-kms-plugin.Yv|
00000190 61 75 6c 74 3a 76 31 3a 75 46 4a 52 47 38 68 44 |ault:v1:uFJRG8hD|
000001a0 36 52 4b 69 50 4c 6c 76 6d 52 44 36 75 6b 34 47 |6RKiPLlvmRD6uk4G|
000001b0 6b 4c 36 48 6b 57 73 78 4e 62 37 4c 47 36 69 33 |kL6HkWsxNb7LG6i3|
000001c0 4e 37 54 79 76 33 6b 47 79 4d 51 2b 31 6c 62 5a |N7Tyv3kGyMQ+1lbZ|
000001d0 62 59 6a 6f 36 53 4b 67 45 55 64 38 56 68 47 37 |bYjo6SKgEUd8VhG7|
000001e0 78 67 54 66 30 74 63 76 22 18 0a 13 76 32 2e 6b |xgTf0tcv"...v2.k|
000001f0 6c 65 69 64 69 2e 62 65 65 7a 79 2e 64 65 76 12 |leidi.beezy.dev.|
00000200 01 31 28 01 0a |.1(..|
00000205
```
**The above extract shows an encrypted payload with the header ```enc:kms:v2:kleidi-kms-plugin:```.**
00000020 65 69 64 69 0a 6b 38 73 3a 65 6e 63 3a 6b 6d 73 |eidi.k8s:enc:kms|
00000030 3a 76 32 3a 6b 6c 65 69 64 69 2d 6b 6d 73 2d 70 |:v2:kleidi-kms-p|
00000040 6c 75 67 69 6e 3a 0a a3 02 b0 90 f1 11 d7 38 da |lugin:........8.|
00000050 dd 7e 50 86 52 6e 88 fe 46 78 9d 76 22 d8 f7 0b |.~P.Rn..Fx.v"...|
00000060 f4 96 54 ad ba 9c 45 59 5f 39 be ee e2 14 83 18 |..T...EY_9......|
00000070 78 b3 d5 e1 d0 4f 9c 9a 47 2d 19 e6 56 93 26 82 |x....O..G-..V.&.|
00000080 8b b6 c4 eb ba f5 d0 b1 6b 22 88 55 75 99 25 4c |........k".Uu.%L|
00000090 0e 43 45 87 a4 70 78 b4 26 15 ed c2 6e ad 03 c0 |.CE..px.&...n...|
000000a0 8b 10 56 05 a4 61 c0 41 d5 f9 1b 8b cd 27 d6 32 |..V..a.A.....'.2|
000000b0 71 e7 7c e0 87 fa c8 34 2c 27 26 21 68 a8 e0 0c |q.|....4,'&!h...|
000000c0 70 5c 7b e1 5b 9f 4d ec b4 b0 7b ce 01 d2 8f 80 |p\{.[.M...{.....|
000000d0 25 84 77 78 0c 21 73 48 dc 7c 50 66 6e 00 8b e0 |%.wx.!sH.|Pfn...|
000000e0 08 8f 5d 6c d6 2c 7e 46 e4 cb f9 6c f5 d8 72 00 |..]l.,~F...l..r.|
000000f0 44 dc 23 3f 6d cf 2e 38 b7 03 bd 03 54 30 3b a7 |D.#?m..8....T0;.|
00000100 ba ed 1b 5e 42 c1 47 10 68 79 87 64 31 43 73 87 |...^B.G.hy.d1Cs.|
00000110 e1 c4 ce a0 bc 5c 15 ae b3 30 42 bf f5 fb b8 bc |.....\...0B.....|
00000120 b2 0a bc 38 29 65 e3 8d 81 23 db 92 38 c2 e5 cb |...8)e...#..8...|
00000130 4c 2d 24 3e df ba e6 01 ba 11 cc 16 17 0a 10 bb |L-$>............|
00000140 98 2f 53 90 4a 1a 9e 90 9d 4c a0 34 19 04 91 9c |./S.J....L.4....|
00000150 22 d4 ac 1d 14 01 1b 45 2f d4 ed e0 73 b6 cf a9 |"......E/...s...|
00000160 43 68 1f a3 5c 56 c6 5d 51 5c 75 4d 12 1e 6b 6c |Ch..\V.]Q\uM..kl|
00000170 65 69 64 69 2d 6b 6d 73 2d 70 6c 75 67 69 6e 5f |eidi-kms-plugin_|
00000180 31 5f 31 37 32 37 31 38 38 31 33 39 1a 59 76 61 |1_1727188139.Yva|
00000190 75 6c 74 3a 76 31 3a 36 42 7a 71 53 4d 50 2f 77 |ult:v1:6BzqSMP/w|
000001a0 4b 61 62 4e 6c 79 6d 53 72 59 5a 68 45 7a 55 6c |KabNlymSrYZhEzUl|
000001b0 31 73 4b 76 4c 2f 78 69 63 7a 50 72 61 74 44 71 |1sKvL/xiczPratDq|
000001c0 4a 70 39 59 70 72 49 34 65 38 5a 56 51 63 43 30 |Jp9YprI4e8ZVQcC0|
000001d0 6a 52 4a 44 6c 45 66 4c 42 4c 6d 55 53 51 48 4f |jRJDlEfLBLmUSQHO|
000001e0 48 43 2b 52 77 71 44 22 18 0a 13 76 32 2e 6b 6c |HC+RwqD"...v2.kl|
000001f0 65 69 64 69 2e 62 65 65 7a 79 2e 64 65 76 12 01 |eidi.beezy.dev..|
00000200 31 28 01 0a |1(..|
00000204
```
The above extract shows an encrypted payload with two crucial pieces of information:
- the header ```enc:kms:v2:kleidi-kms-plugin:```.
- the key version ```1_1727188139```
If there is a rotation of the Vault transit key:
```
vault write -f transit/keys/kleidi/rotate
```
then replace the secrets:
```
kubectl get secret postkleidi -o json | kubectl replace -f -
secret/postkleidi replaced
```
then the following payload would expected:
```
kubectl -n kube-system exec etcd-kleidi-vault-prd-control-plane -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/etc/kubernetes/pki/etcd/ca.crt' ETCDCTL_CERT='/etc/kubernetes/pki/etcd/server.crt' ETCDCTL_KEY='/etc/kubernetes/pki/etcd/server.key' ETCDCTL_API=3 etcdctl get /registry/secrets/default/postkleidi" | hexdump -C
```
```
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 70 6f 73 74 6b 6c |s/default/postkl|
00000020 65 69 64 69 0a 6b 38 73 3a 65 6e 63 3a 6b 6d 73 |eidi.k8s:enc:kms|
00000030 3a 76 32 3a 6b 6c 65 69 64 69 2d 6b 6d 73 2d 70 |:v2:kleidi-kms-p|
00000040 6c 75 67 69 6e 3a 0a a3 02 53 67 55 93 db 8b 4e |lugin:...SgU...N|
00000050 ad 83 d9 0e 89 e3 4c 14 f5 24 34 f7 1c c5 ea 37 |......L..$4....7|
00000060 22 8c 01 cd ec 3b 34 0c f8 28 10 17 33 8a f1 a8 |"....;4..(..3...|
00000070 22 ab b8 9c 34 1d 54 f2 6d 75 ac 6c d5 d4 1e 9b |"...4.T.mu.l....|
00000080 9d d0 ab ad 8d 6f 3e dd 7a 8d a7 3e f5 a9 6f a1 |.....o>.z..>..o.|
00000090 77 83 ba 1e 1e b0 25 73 64 c5 b4 91 b9 6e 46 21 |w.....%sd....nF!|
000000a0 fe 8d 4d c7 97 77 64 35 0e f3 96 20 93 12 e8 f2 |..M..wd5... ....|
000000b0 f6 0b 26 fc 61 7d f8 09 7c 08 c3 a5 4b 62 47 3e |..&.a}..|...KbG>|
000000c0 69 59 83 0b c5 3a 9c 32 4c 0e e6 bb c5 38 5a be |iY...:.2L....8Z.|
000000d0 77 8d 0e 5c 87 95 e8 27 65 0b b7 e1 37 d6 4a ed |w..\...'e...7.J.|
000000e0 8b 7c 7b 33 e5 71 e3 20 a5 3b 28 8f 9c 89 73 e9 |.|{3.q. .;(...s.|
000000f0 9a 21 6b 3c 1b 25 c7 61 b1 81 5f 55 59 93 53 ec |.!k<.%.a.._UY.S.|
00000100 d1 2e ca 8b d9 c8 1a d5 2c a3 4f 6e 52 1d 26 28 |........,.OnR.&(|
00000110 70 5f 04 c1 45 58 30 2f 14 b9 a0 7e 1a 50 6a 71 |p_..EX0/...~.Pjq|
00000120 98 64 14 23 df b7 15 41 6e e1 66 88 9d 72 c4 d9 |.d.#...An.f..r..|
00000130 c7 30 63 65 6c e2 23 e4 5f 88 da c4 50 40 cc ce |.0cel.#._...P@..|
00000140 4e 96 91 54 64 07 97 54 63 b3 93 fe dc ae f9 6b |N..Td..Tc......k|
00000150 11 62 50 70 8a 37 ca 7c 78 5e ac 1f d4 53 2d ba |.bPp.7.|x^...S-.|
00000160 13 a6 92 be ed 48 aa 54 06 a7 a3 ee 12 1e 6b 6c |.....H.T......kl|
00000170 65 69 64 69 2d 6b 6d 73 2d 70 6c 75 67 69 6e 5f |eidi-kms-plugin_|
00000180 32 5f 31 37 32 37 31 38 38 37 30 34 1a 59 76 61 |2_1727188704.Yva|
00000190 75 6c 74 3a 76 32 3a 6f 4b 51 46 55 48 57 44 52 |ult:v2:oKQFUHWDR|
000001a0 52 31 30 41 4a 4f 51 74 6b 74 49 55 57 32 4f 69 |R10AJOQtktIUW2Oi|
000001b0 55 56 73 56 57 59 65 6a 79 77 44 56 53 45 52 46 |UVsVWYejywDVSERF|
000001c0 53 41 70 6e 75 76 6b 42 5a 75 65 34 44 61 78 43 |SApnuvkBZue4DaxC|
000001d0 78 45 50 7a 4e 6d 77 48 39 63 75 72 61 4f 4b 34 |xEPzNmwH9curaOK4|
000001e0 6b 6c 6a 7a 77 45 6a 22 18 0a 13 76 32 2e 6b 6c |kljzwEj"...v2.kl|
000001f0 65 69 64 69 2e 62 65 65 7a 79 2e 64 65 76 12 01 |eidi.beezy.dev..|
00000200 31 28 01 0a |1(..|
00000204
```
The version has been updated with ```2_1727188704``` and we can verify the key version in Vault:
```
vault read transit/keys/kleidi
```
```
Key Value
--- -----
allow_plaintext_backup false
auto_rotate_period 0s
deletion_allowed false
derived false
exportable false
imported_key false
keys map[1:1727188139 2:1727188704]
latest_version 2
min_available_version 0
min_decryption_version 1
min_encryption_version 0
name kleidi
supports_decryption true
supports_derivation true
supports_encryption true
supports_signing false
type aes256-gcm96
```
Then let's encrypt the pre-deployment secret too:
Expand Down

0 comments on commit df7fc8a

Please sign in to comment.