Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API-1843: FeatureGate(d) KMS encryption #2035

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
apiVersion: apiextensions.k8s.io/v1 # Hack because controller-gen complains if we don't have this
name: "APIServer"
crdName: apiservers.config.openshift.io
featureGates:
- KMSEncryptionProvider
tests:
onCreate:
swghosh marked this conversation as resolved.
Show resolved Hide resolved
- name: Should be able to create encrypt with KMS for AWS with valid values
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms:
type: AWS
aws:
keyARN: arn:aws:kms:us-east-1:101010101010:key/9a512e29-0d9c-4cf5-8174-fc1a5b22cd6a
region: us-east-1
expected: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
audit:
profile: Default
encryption:
type: KMS
kms:
type: AWS
aws:
keyARN: arn:aws:kms:us-east-1:101010101010:key/9a512e29-0d9c-4cf5-8174-fc1a5b22cd6a
region: us-east-1
- name: Should be able to create encrypt with KMS for AWS without region
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms:
type: AWS
aws:
keyARN: arn:aws:kms:us-east-1:101010101010:key/9a512e29-0d9c-4cf5-8174-fc1a5b22cd6a
expected: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
audit:
profile: Default
encryption:
type: KMS
kms:
type: AWS
aws:
keyARN: arn:aws:kms:us-east-1:101010101010:key/9a512e29-0d9c-4cf5-8174-fc1a5b22cd6a
- name: Should not allow kms config with encrypt aescbc
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: aescbc
kms:
type: AWS
aws:
keyARN: arn:aws:kms:us-east-1:101010101010:key/9a512e29-0d9c-4cf5-8174-fc1a5b22cd6a
expectedError: "kms config is required when encryption type is KMS, and forbidden otherwise"
- name: Should fail to create with an empty KMS config
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms: {}
expectedError: "spec.encryption.kms.type: Required value"
- name: Should fail to create with kms type AWS but without aws config
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms:
type: AWS
expectedError: "aws config is required when kms provider type is AWS, and forbidden otherwise"
- name: Should fail to create AWS KMS without a keyARN
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms:
type: AWS
aws:
region: us-east-1
expectedError: "spec.encryption.kms.aws.keyARN: Required value"
- name: Should fail to create AWS KMS with invalid keyARN format
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms:
type: AWS
aws:
keyARN: not-a-kms-arn
region: us-east-1
expectedError: "keyARN must follow the format `arn:aws:kms:<region>:<account_id>:key/<key_id>`. The account ID must be a 12 digit number and the region and key ID should consist only of lowercase hexadecimal characters and hyphens (-)."
- name: Should fail to create AWS KMS with empty region
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms:
type: AWS
aws:
keyARN: arn:aws:kms:us-east-1:101010101010:key/9a512e29-0d9c-4cf5-8174-fc1a5b22cd6a
region: ""
expectedError: "spec.encryption.kms.aws.region in body should be at least 1 chars long"
- name: Should fail to create AWS KMS with invalid region format
initial: |
apiVersion: config.openshift.io/v1
kind: APIServer
spec:
encryption:
type: KMS
kms:
type: AWS
aws:
keyARN: arn:aws:kms:us-east-1:101010101010:key/9a512e29-0d9c-4cf5-8174-fc1a5b22cd6a
region: "INVALID-REGION"
expectedError: "region must be a valid AWS region, consisting of lowercase characters, digits and hyphens (-) only."
27 changes: 26 additions & 1 deletion config/v1/types_apiserver.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ type APIServerSpec struct {
// server from JavaScript applications.
// The values are regular expressions that correspond to the Golang regular expression language.
// +optional
// +listType=atomic
Copy link
Member Author

@swghosh swghosh Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These +listType=atomic for exisiting fields in APISeverSpec are required to make the verify-crd-schema pass, which fails the test saying lack of SSA tags in the generated CRD. Alternative to adding these is to override the specific test but this retains the API violations as-is today.

AdditionalCORSAllowedOrigins []string `json:"additionalCORSAllowedOrigins,omitempty"`
// encryption allows the configuration of encryption of resources at the datastore layer.
// +optional
Expand Down Expand Up @@ -153,6 +154,7 @@ type APIServerServingCerts struct {
// If no named certificates are provided, or no named certificates match the server name as understood by a client,
// the defaultServingCertificate will be used.
// +optional
// +listType=atomic
NamedCertificates []APIServerNamedServingCert `json:"namedCertificates,omitempty"`
}

Expand All @@ -162,6 +164,7 @@ type APIServerNamedServingCert struct {
// serve secure traffic. If no names are provided, the implicit names will be extracted from the certificates.
// Exact names trump over wildcard names. Explicit names defined here trump over extracted implicit names.
// +optional
// +listType=atomic
Names []string `json:"names,omitempty"`
// servingCertificate references a kubernetes.io/tls type secret containing the TLS cert info for serving secure traffic.
// The secret must exist in the openshift-config namespace and contain the following required fields:
Expand All @@ -170,6 +173,9 @@ type APIServerNamedServingCert struct {
ServingCertificate SecretNameReference `json:"servingCertificate"`
}

// APIServerEncryption is used to encrypt sensitive resources on the cluster.
// +openshift:validation:FeatureGateAwareXValidation:featureGate=KMSEncryptionProvider,rule="has(self.type) && self.type == 'KMS' ? has(self.kms) : !has(self.kms)",message="kms config is required when encryption type is KMS, and forbidden otherwise"
// +union
type APIServerEncryption struct {
// type defines what encryption type should be used to encrypt resources at the datastore layer.
// When this field is unset (i.e. when it is set to the empty string), identity is implied.
Expand All @@ -188,9 +194,23 @@ type APIServerEncryption struct {
// +unionDiscriminator
// +optional
Type EncryptionType `json:"type,omitempty"`

// kms defines the configuration for the external KMS instance that manages the encryption keys,
// when KMS encryption is enabled sensitive resources will be encrypted using keys managed by an
// externally configured KMS instance.
//
// The Key Management Service (KMS) instance provides symmetric encryption and is responsible for
// managing the lifecyle of the encryption keys outside of the control plane.
// This allows integration with an external provider to manage the data encryption keys securely.
//
// +openshift:enable:FeatureGate=KMSEncryptionProvider
// +unionMember
// +optional
KMS *KMSConfig `json:"kms,omitempty"`
swghosh marked this conversation as resolved.
Show resolved Hide resolved
}

// +kubebuilder:validation:Enum="";identity;aescbc;aesgcm
// +openshift:validation:FeatureGateAwareEnum:featureGate="",enum="";identity;aescbc;aesgcm
// +openshift:validation:FeatureGateAwareEnum:featureGate=KMSEncryptionProvider,enum="";identity;aescbc;aesgcm;KMS
type EncryptionType string

const (
Expand All @@ -205,6 +225,11 @@ const (
// aesgcm refers to a type where AES-GCM with random nonce and a 32-byte key
// is used to perform encryption at the datastore layer.
EncryptionTypeAESGCM EncryptionType = "aesgcm"

// kms refers to a type of encryption where the encryption keys are managed
// outside the control plane in a Key Management Service instance,
// encryption is still performed at the datastore layer.
EncryptionTypeKMS EncryptionType = "KMS"
)

type APIServerStatus struct {
Expand Down
55 changes: 55 additions & 0 deletions config/v1/types_kmsencryption.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package v1

// KMSConfig defines the configuration for the KMS instance
// that will be used with KMSEncryptionProvider encryption
// +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'AWS' ? has(self.aws) : !has(self.aws)",message="aws config is required when kms provider type is AWS, and forbidden otherwise"
swghosh marked this conversation as resolved.
Show resolved Hide resolved
// +union
type KMSConfig struct {
swghosh marked this conversation as resolved.
Show resolved Hide resolved
// type defines the kind of platform for the KMS provider.
// Available provider types are AWS only.
//
// +unionDiscriminator
// +required
Type KMSProviderType `json:"type"`
swghosh marked this conversation as resolved.
Show resolved Hide resolved

// aws defines the key config for using an AWS KMS instance
// for the encryption. The AWS KMS instance is managed
// by the user outside the purview of the control plane.
//
// +unionMember
// +optional
AWS *AWSKMSConfig `json:"aws,omitempty"`
}

// AWSKMSConfig defines the KMS config specific to AWS KMS provider
type AWSKMSConfig struct {
// keyARN specifies the Amazon Resource Name (ARN) of the AWS KMS key used for encryption.
// The value must adhere to the format `arn:aws:kms:<region>:<account_id>:key/<key_id>`, where:
// - `<region>` is the AWS region consisting of lowercase letters and hyphens followed by a number.
// - `<account_id>` is a 12-digit numeric identifier for the AWS account.
// - `<key_id>` is a unique identifier for the KMS key, consisting of lowercase hexadecimal characters and hyphens.
//
// +kubebuilder:validation:MaxLength=128
swghosh marked this conversation as resolved.
Show resolved Hide resolved
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:XValidation:rule="self.matches('^arn:aws:kms:[a-z0-9-]+:[0-9]{12}:key/[a-f0-9-]+$')",message="keyARN must follow the format `arn:aws:kms:<region>:<account_id>:key/<key_id>`. The account ID must be a 12 digit number and the region and key ID should consist only of lowercase hexadecimal characters and hyphens (-)."
// +required
KeyARN string `json:"keyARN"`
swghosh marked this conversation as resolved.
Show resolved Hide resolved
swghosh marked this conversation as resolved.
Show resolved Hide resolved
swghosh marked this conversation as resolved.
Show resolved Hide resolved
// region specifies the AWS region where the KMS instance exists, and follows the format
// `<region-prefix>-<region-name>-<number>`, e.g.: `us-east-1`.
// Only lowercase letters and hyphens followed by numbers are allowed.
//
// +kubebuilder:validation:MaxLength=64
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:XValidation:rule="self.matches('^[a-z0-9]+(-[a-z0-9]+)*$')",message="region must be a valid AWS region, consisting of lowercase characters, digits and hyphens (-) only."
// +optional
Region string `json:"region"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to have some validation on the valid characters here, where did it go?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had removed it based on suggestions from @benluddy
#2035 (comment)
Let's re-check if we want to keep that validation rule.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would at least check the character set is lowercase alphanumeric segments separate by hyphens, even if it is not as strict as what you had. We know they use the region as part of the domain name, so it must be a valid domain segment, so below should be accurate.

[a-z0-9]+(-[a-z0-9]+)*

And the maxLength can be 50, per AWS' own docs, based on Ben's link.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

+ // +kubebuilder:validation:XValidation:rule="self.matches('^[a-z0-9]+(-[a-z0-9]+)*$')",message="region must be a valid AWS region, consisting of lowercase characters, digits and hyphens (-) only."

Also, added an integration test case to look for that error message when it is violated.

Copy link
Member Author

@swghosh swghosh Dec 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, on that AWS region on a similar context
found from elsewhere that "aws-global" seems to also be a somewhat valid region for AWS Go SDK v2 trying to connect over AWS STS credentials with some legacy behaviour
xref: https://github.com/cert-manager/cert-manager/blob/537e71ee639a41887e93b0fd151bf063c4730536/pkg/issuer/acme/dns/route53/route53.go#L103
however the said regex, covers this case too so nvm!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xref #2124

Here's an example of our API validation rejecting a value that would have been accepted by an AWS API. Just sharing to support my opinion that it's worth erring on the side of being lax with these. I would be a little surprised if these values were validated consistently across all services within AWS.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just sharing to support my opinion that it's worth erring on the side of being lax with these. I would be a little surprised if these values were validated consistently across all services within AWS.

+1. I'm all for including documented regexes or, perhaps, relatively simple ones, but this is at least the second time I can recall that we (installer team) have had to revise regexes we attempted to reverse engineer the requirements. In the first case we inadvertently excluded disk encryption keys with capital letters, which are allowed in Azure.

Where possible, I prefer validating against the cloud api rather than regexes, but that may just be a luxury we have in the installer.

}

// KMSProviderType is a specific supported KMS provider
// +kubebuilder:validation:Enum=AWS
type KMSProviderType string

const (
// AWSKMSProvider represents a supported KMS provider for use with AWS KMS
AWSKMSProvider KMSProviderType = "AWS"
)
Loading