Skip to content

Commit

Permalink
Add GCP deployment to cdp-deploy module
Browse files Browse the repository at this point in the history
Signed-off-by: Jim Enright <[email protected]>
  • Loading branch information
jimright committed Dec 19, 2023
1 parent 96296b0 commit 4af6521
Show file tree
Hide file tree
Showing 13 changed files with 971 additions and 28 deletions.
23 changes: 19 additions & 4 deletions modules/terraform-cdp-deploy/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- BEGIN_TF_DOCS -->
# Terraform Module for CDP Deployment

This module contains resource files and example variable definition files for deployment of Cloudera Data Platform (CDP) Public Cloud environment and Datalake creation on AWS or Azure.
This module contains resource files and example variable definition files for deployment of Cloudera Data Platform (CDP) Public Cloud environment and Datalake creation on AWS, Azure or GCP.

## Usage

Expand All @@ -11,14 +11,16 @@ The [examples](./examples) directory has example CDP deployments:

* `ex02-azure-basic` creates a basic CDP deployment on Azure. This example makes use of the [terraform-cdp-azure-pre-reqs module](../terraform-cdp-azure-pre-reqs) to create the required cloud resources.

* `ex02-gcp-basic` creates a basic CDP deployment on GCP. This example makes use of the [terraform-cdp-gcp-pre-reqs module](../terraform-cdp-gcp-pre-reqs) to create the required cloud resources.

In each directory an example `terraform.tfvars.sample` values file is included to show input variable values.

## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
| <a name="requirement_cdp"></a> [cdp](#requirement\_cdp) | 0.4.0 |
| <a name="requirement_cdp"></a> [cdp](#requirement\_cdp) | 0.4.1-pre |

## Providers

Expand All @@ -30,6 +32,7 @@ No providers.
|------|--------|---------|
| <a name="module_cdp_on_aws"></a> [cdp\_on\_aws](#module\_cdp\_on\_aws) | ./modules/aws | n/a |
| <a name="module_cdp_on_azure"></a> [cdp\_on\_azure](#module\_cdp\_on\_azure) | ./modules/azure | n/a |
| <a name="module_cdp_on_gcp"></a> [cdp\_on\_gcp](#module\_cdp\_on\_gcp) | ./modules/gcp | n/a |

## Resources

Expand Down Expand Up @@ -77,7 +80,7 @@ No resources.
| <a name="input_cdp_admin_group_name"></a> [cdp\_admin\_group\_name](#input\_cdp\_admin\_group\_name) | Name of the CDP IAM Admin Group associated with the environment. Defaults to '<env\_prefix>-cdp-admin-group' if not specified. | `string` | `null` | no |
| <a name="input_cdp_user_group_name"></a> [cdp\_user\_group\_name](#input\_cdp\_user\_group\_name) | Name of the CDP IAM User Group associated with the environment. Defaults to '<env\_prefix>-cdp-user-group' if not specified. | `string` | `null` | no |
| <a name="input_cdp_xacccount_credential_name"></a> [cdp\_xacccount\_credential\_name](#input\_cdp\_xacccount\_credential\_name) | Name of the CDP Cross Account Credential. Defaults to '<env\_prefix>-xaccount-cred' if not specified. | `string` | `null` | no |
| <a name="input_datalake_custom_instance_groups"></a> [datalake\_custom\_instance\_groups](#input\_datalake\_custom\_instance\_groups) | A set of custom instance groups for the datalake. Only applicable for CDP deployment on AWS. | <pre>list(<br> object({<br> name = string,<br> instance_type = optional(string)<br> })<br> )</pre> | `null` | no |
| <a name="input_datalake_custom_instance_groups"></a> [datalake\_custom\_instance\_groups](#input\_datalake\_custom\_instance\_groups) | A set of custom instance groups for the datalake. Only applicable for CDP deployment on AWS and GCP. | <pre>list(<br> object({<br> name = string,<br> instance_type = optional(string)<br> })<br> )</pre> | `null` | no |
| <a name="input_datalake_image"></a> [datalake\_image](#input\_datalake\_image) | The image to use for the datalake. Can only be used when the 'datalake\_version' parameter is set to null. You can use 'catalog' name and/or 'id' for selecting an image. | <pre>object({<br> id = optional(string)<br> catalog = optional(string)<br> })</pre> | `null` | no |
| <a name="input_datalake_java_version"></a> [datalake\_java\_version](#input\_datalake\_java\_version) | The Java major version to use on the datalake cluster. | `number` | `null` | no |
| <a name="input_datalake_name"></a> [datalake\_name](#input\_datalake\_name) | Name of the CDP datalake. Defaults to '<env\_prefix>-<aw\|az\|gc\|>-dl' if not specified. | `string` | `null` | no |
Expand All @@ -100,12 +103,24 @@ No resources.
| <a name="input_freeipa_instance_type"></a> [freeipa\_instance\_type](#input\_freeipa\_instance\_type) | Instance Type to use for creating FreeIPA instances | `string` | `null` | no |
| <a name="input_freeipa_instances"></a> [freeipa\_instances](#input\_freeipa\_instances) | The number of FreeIPA instances to create in the environment | `number` | `3` | no |
| <a name="input_freeipa_recipes"></a> [freeipa\_recipes](#input\_freeipa\_recipes) | The recipes for the FreeIPA cluster | `set(string)` | `null` | no |
| <a name="input_gcp_cdp_subnet_names"></a> [gcp\_cdp\_subnet\_names](#input\_gcp\_cdp\_subnet\_names) | List of GCP Subnet Names for CDP Resources. Required for CDP deployment on GCP. | `list(any)` | `null` | no |
| <a name="input_gcp_datalake_admin_service_account_email"></a> [gcp\_datalake\_admin\_service\_account\_email](#input\_gcp\_datalake\_admin\_service\_account\_email) | Email id of the service account for Datalake Admin. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_encryption_key"></a> [gcp\_encryption\_key](#input\_gcp\_encryption\_key) | Key Resource ID of the customer managed encryption key to encrypt GCP resources. Only applicable for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_firewall_default_id"></a> [gcp\_firewall\_default\_id](#input\_gcp\_firewall\_default\_id) | Default Firewall for CDP environment. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_firewall_knox_id"></a> [gcp\_firewall\_knox\_id](#input\_gcp\_firewall\_knox\_id) | Knox Firewall for CDP environment. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_idbroker_service_account_email"></a> [gcp\_idbroker\_service\_account\_email](#input\_gcp\_idbroker\_service\_account\_email) | Email id of the service account for IDBroker. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_log_service_account_email"></a> [gcp\_log\_service\_account\_email](#input\_gcp\_log\_service\_account\_email) | Email id of the service account for Log Storage. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_network_name"></a> [gcp\_network\_name](#input\_gcp\_network\_name) | GCP Network VPC name. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_project_id"></a> [gcp\_project\_id](#input\_gcp\_project\_id) | GCP project to deploy CDP environment. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_ranger_audit_service_account_email"></a> [gcp\_ranger\_audit\_service\_account\_email](#input\_gcp\_ranger\_audit\_service\_account\_email) | Email id of the service account for Ranger Audit. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_gcp_report_deployment_logs"></a> [gcp\_report\_deployment\_logs](#input\_gcp\_report\_deployment\_logs) | Flag to enable reporting of additional diagnostic information back to Cloudera. Only applicable for CDP deployment on GCP. | `bool` | `false` | no |
| <a name="input_gcp_xaccount_service_account_private_key"></a> [gcp\_xaccount\_service\_account\_private\_key](#input\_gcp\_xaccount\_service\_account\_private\_key) | Base64 encoded private key of the GCP Cross Account Service Account Key. Required for CDP deployment on GCP. | `string` | `null` | no |
| <a name="input_keypair_name"></a> [keypair\_name](#input\_keypair\_name) | SSH Keypair name in Cloud Service Provider. For CDP deployment on AWS, either 'keypair\_name' or 'public\_key\_text' needs to be set. | `string` | `null` | no |
| <a name="input_multiaz"></a> [multiaz](#input\_multiaz) | Flag to specify that the FreeIPA and DataLake instances will be deployed across multi-availability zones. | `bool` | `true` | no |
| <a name="input_proxy_config_name"></a> [proxy\_config\_name](#input\_proxy\_config\_name) | Name of the proxy config to use for the environment. | `string` | `null` | no |
| <a name="input_public_key_text"></a> [public\_key\_text](#input\_public\_key\_text) | SSH Public key string for the nodes of the CDP environment. Required for CDP deployment on Azure. For CDP deployment on AWS, either 'keypair\_name' or 'public\_key\_text' needs to be set. | `string` | `null` | no |
| <a name="input_s3_guard_table_name"></a> [s3\_guard\_table\_name](#input\_s3\_guard\_table\_name) | Name for the DynamoDB table backing S3Guard. Only applicable for CDP deployment on AWS. | `string` | `null` | no |
| <a name="input_use_public_ips"></a> [use\_public\_ips](#input\_use\_public\_ips) | Use public ip's for the CDP resources created within the Azure network. Required for CDP deployment on Azure. | `bool` | `null` | no |
| <a name="input_use_public_ips"></a> [use\_public\_ips](#input\_use\_public\_ips) | Use public ip's for the CDP resources created within the Cloud network. Required for CDP deployment on Azure and GCP. | `bool` | `null` | no |
| <a name="input_use_single_resource_group"></a> [use\_single\_resource\_group](#input\_use\_single\_resource\_group) | Use a single resource group for all provisioned CDP resources. Required for CDP deployment on Azure. | `bool` | `true` | no |
| <a name="input_workload_analytics"></a> [workload\_analytics](#input\_workload\_analytics) | Flag to specify if workload analytics should be enabled for the CDP environment | `bool` | `true` | no |

Expand Down
4 changes: 3 additions & 1 deletion modules/terraform-cdp-deploy/doc_fragments/header.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Terraform Module for CDP Deployment

This module contains resource files and example variable definition files for deployment of Cloudera Data Platform (CDP) Public Cloud environment and Datalake creation on AWS or Azure.
This module contains resource files and example variable definition files for deployment of Cloudera Data Platform (CDP) Public Cloud environment and Datalake creation on AWS, Azure or GCP.

## Usage

Expand All @@ -10,4 +10,6 @@ The [examples](./examples) directory has example CDP deployments:

* `ex02-azure-basic` creates a basic CDP deployment on Azure. This example makes use of the [terraform-cdp-azure-pre-reqs module](../terraform-cdp-azure-pre-reqs) to create the required cloud resources.

* `ex02-gcp-basic` creates a basic CDP deployment on GCP. This example makes use of the [terraform-cdp-gcp-pre-reqs module](../terraform-cdp-gcp-pre-reqs) to create the required cloud resources.

In each directory an example `terraform.tfvars.sample` values file is included to show input variable values.
71 changes: 71 additions & 0 deletions modules/terraform-cdp-deploy/examples/ex03-gcp-basic/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

provider "google" {
project = var.gcp_project
region = var.gcp_region
}


module "cdp_gcp_prereqs" {
source = "../../../terraform-cdp-gcp-pre-reqs"

env_prefix = var.env_prefix
gcp_region = var.gcp_region

deployment_template = var.deployment_template

ingress_extra_cidrs_and_ports = var.ingress_extra_cidrs_and_ports

# Inputs for BYO-VNet
# create_vnet = var.create_vnet
# cdp_resourcegroup_name = var.cdp_resourcegroup_name
# cdp_vnet_name = var.cdp_vnet_name
# cdp_subnet_names = var.cdp_subnet_names
# cdp_gw_subnet_names = var.cdp_gw_subnet_names

}

module "cdp_deploy" {
source = "../.."

env_prefix = var.env_prefix
infra_type = "gcp"
gcp_project_id = var.gcp_project
region = var.gcp_region
public_key_text = var.public_key_text
deployment_template = var.deployment_template

# From pre-reqs module output
gcp_network_name = module.cdp_gcp_prereqs.gcp_vpc_name
gcp_cdp_subnet_names = module.cdp_gcp_prereqs.gcp_cdp_subnet_names

gcp_firewall_default_id = module.cdp_gcp_prereqs.gcp_firewall_default_name
gcp_firewall_knox_id = module.cdp_gcp_prereqs.gcp_firewall_knox_name

data_storage_location = module.cdp_gcp_prereqs.gcp_data_storage_location
log_storage_location = module.cdp_gcp_prereqs.gcp_log_storage_location
backup_storage_location = module.cdp_gcp_prereqs.gcp_backup_storage_location

gcp_xaccount_service_account_private_key = module.cdp_gcp_prereqs.gcp_xaccount_sa_private_key

gcp_idbroker_service_account_email = module.cdp_gcp_prereqs.gcp_idbroker_service_account_email
gcp_datalake_admin_service_account_email = module.cdp_gcp_prereqs.gcp_datalake_admin_service_account_email
gcp_ranger_audit_service_account_email = module.cdp_gcp_prereqs.gcp_ranger_audit_service_account_email
gcp_log_service_account_email = module.cdp_gcp_prereqs.gcp_log_service_account_email

depends_on = [
module.cdp_gcp_prereqs
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# ------- Global settings -------
env_prefix = "<ENTER_VALUE>" # Required name prefix for cloud and CDP resources, e.g. cldr1

# ------- Cloud Settings -------
gcp_project = "<ENTER_VALUE>" # Change this to specify the GCP Project ID

gcp_region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. europe-west2

public_key_text = "<ENTER_VALUE>" # Change this with the SSH public key text, e.g. ssh-rsa AAA....

# ------- CDP Environment Deployment -------
deployment_template = "<ENTER_VALUE>" # Specify the deployment pattern below. Options are public, semi-private or private

# TODO:
# ------- Optional inputs for BYO-VPC -------
# **NOTE: Uncomment below settings if required

# create_vpc=true # Set to false to use pre-existing VNet

# cdp_resourcegroup_name="<ENTER_EXISTING_RG_NAME>" # Name of pre-existing Resource Group for CDP environment. Required if create_vnet is false
# cdp_vnet_name="<ENTER_EXISTING_VNET_NAME>" # Name of pre-existing VNet for CDP environment. Required if create_vnet is false
# cdp_subnet_names=["<ENTER_EXISTING_SUBNET_NAME>","<ENTER_EXISTING_SUBNET_NAME>"] # List of pre-existing subnet names for CDP resources. Required if create_vnet is false
# cdp_gw_subnet_names=["<ENTER_EXISTING_SUBNET_NAME>"] # List of pre-existing subnet names for CDP Gateway. Required if create_vnet is false
88 changes: 88 additions & 0 deletions modules/terraform-cdp-deploy/examples/ex03-gcp-basic/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# ------- Global settings -------
variable "env_prefix" {
type = string
description = "Shorthand name for the environment. Used in resource descriptions"
}

variable "gcp_project" {
type = string
description = "Region which Cloud resources will be created"
}

variable "gcp_region" {
type = string
description = "Region which Cloud resources will be created"
}

variable "public_key_text" {
type = string

description = "SSH Public key string for the nodes of the CDP environment"
}

# ------- CDP Environment Deployment -------
variable "deployment_template" {
type = string

description = "Deployment Pattern to use for Cloud resources and CDP"
}

# # ------- Network Resources -------
variable "ingress_extra_cidrs_and_ports" {
type = object({
cidrs = list(string)
ports = list(number)
})
description = "List of extra CIDR blocks and ports to include in Security Group Ingress rules"
}

# ------- Optional inputs for BYO-VPC -------
variable "create_vnet" {
type = bool

description = "Flag to specify if the VNet should be created"

default = true
}

variable "cdp_resourcegroup_name" {
type = string
description = "Pre-existing Resource Group for CDP environment. Required if create_vnet is false."

default = null
}

variable "cdp_vnet_name" {
type = string
description = "Pre-existing VNet Name for CDP environment. Required if create_vnet is false."

default = null
}

variable "cdp_subnet_names" {
type = list(any)
description = "List of subnet names for CDP Resources. Required if create_vnet is false."

default = null
}

variable "cdp_gw_subnet_names" {
type = list(any)
description = "List of subnet names for CDP Gateway. Required if create_vnet is false."

default = null
}
63 changes: 63 additions & 0 deletions modules/terraform-cdp-deploy/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -160,3 +160,66 @@ module "cdp_on_azure" {
datalake_java_version = var.datalake_java_version
datalake_recipes = var.datalake_recipes
}

# ------- Call sub-module for GCP Deployment -------
module "cdp_on_gcp" {
count = (var.infra_type == "gcp") ? 1 : 0

source = "./modules/gcp"

# tags = local.env_tags # NOTE: Waiting on provider fix

environment_name = local.environment_name
datalake_name = local.datalake_name
cdp_xacccount_credential_name = local.cdp_xacccount_credential_name
cdp_admin_group_name = local.cdp_admin_group_name
cdp_user_group_name = local.cdp_user_group_name

firewall_default_id = var.gcp_firewall_default_id
firewall_knox_id = var.gcp_firewall_knox_id

datalake_scale = local.datalake_scale
datalake_version = var.datalake_version
enable_ccm_tunnel = var.enable_ccm_tunnel

freeipa_instances = var.freeipa_instances
freeipa_instance_type = var.freeipa_instance_type
freeipa_recipes = var.freeipa_recipes

workload_analytics = var.workload_analytics
endpoint_access_scheme = local.endpoint_access_scheme

environment_polling_timeout = var.environment_polling_timeout
datalake_polling_timeout = var.datalake_polling_timeout

use_public_ips = local.use_public_ips

project_id = var.gcp_project_id

region = var.region
network_name = var.gcp_network_name
cdp_subnet_names = var.gcp_cdp_subnet_names
public_key_text = var.public_key_text

data_storage_location = var.data_storage_location
log_storage_location = var.log_storage_location
backup_storage_location = var.backup_storage_location

xaccount_service_account_private_key = var.gcp_xaccount_service_account_private_key

proxy_config_name = var.proxy_config_name

encryption_key = var.gcp_encryption_key
idbroker_service_account_email = var.gcp_idbroker_service_account_email
ranger_audit_service_account_email = var.gcp_ranger_audit_service_account_email
datalake_admin_service_account_email = var.gcp_datalake_admin_service_account_email
log_service_account_email = var.gcp_log_service_account_email

report_deployment_logs = var.gcp_report_deployment_logs

datalake_custom_instance_groups = var.datalake_custom_instance_groups
datalake_image = var.datalake_image
datalake_java_version = var.datalake_java_version
datalake_recipes = var.datalake_recipes

}
Loading

0 comments on commit 4af6521

Please sign in to comment.