Update CDP TF provider to v0.2.0 (#33)

Signed-off-by: Jim Enright <[email protected]>

README.md

@@ -69,6 +69,19 @@ Configure Terraform Provider for AWS or Azure
   * For **AWS** access keys are required to be able to create the Cloud resources via the Terraform aws provider. See the [AWS Terraform Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration).
   * For **Azure**, authentication with the Azure subscription is required. There are a number of ways to do this outlined in the [Azure Terraform Provider Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure).
 
+### Notes on Azure authentication
+
+* Where you have more than one Azure Subscription the id to use can be passed via the the `ARM_SUBSCRIPTION_ID` environment variable.
+
+* When using a Service Principal (SP) to authenticate with Azure, it is not possible to authenticate with azuread Terraform Provider (the provider used to create the Azure Cross Account AD Application) with the command az login --service-principal. We found the the best way to authenticate using an SP is by setting environment variables. Details of required environment variables are in the [azuread docs](https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/guides/service_principal_client_secret#environment-variables) and [azurerm docs](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret#configuring-the-service-principal-in-terraform) and summarized below.
+```bash
+export ARM_CLIENT_ID="<sp_client_id>"
+export ARM_CLIENT_SECRET="<sp_client_secret>"
+export ARM_TENANT_ID="<sp_tenant_id>"
+export ARM_SUBSCRIPTION_ID="<sp_subscription_id>" 
+```
+
+
 ## Local Development Environment
 
 See the [DEVELOPMENT.md](./DEVELOPMENT.md) file for instructions on how to set up an environment for local development of modules.

modules/terraform-cdp-aws-pre-reqs/examples/ex01-minimal_inputs/main.tf

@@ -38,7 +38,7 @@ terraform {
   required_providers {
     cdp = {
       source  = "cloudera/cdp"
-      version = "0.1.4-pre"
+      version = "0.2.0"
     }
   }
 }

modules/terraform-cdp-aws-pre-reqs/examples/ex02-existing-vpc/main.tf

@@ -51,7 +51,7 @@ terraform {
   required_providers {
     cdp = {
       source  = "cloudera/cdp"
-      version = "0.1.4-pre"
+      version = "0.2.0"
     }
   }
 }

modules/terraform-cdp-aws-pre-reqs/examples/ex03-create-keypair/main.tf

@@ -58,7 +58,7 @@ terraform {
   required_providers {
     cdp = {
       source  = "cloudera/cdp"
-      version = "0.1.4-pre"
+      version = "0.2.0"
     }
   }
 }

modules/terraform-cdp-deploy/README.md

@@ -18,7 +18,7 @@ In each directory an example `terraform.tfvars.sample` values file is included t
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
-| <a name="requirement_cdp"></a> [cdp](#requirement\_cdp) | 0.1.4-pre |
+| <a name="requirement_cdp"></a> [cdp](#requirement\_cdp) | 0.2.0 |
 
 ## Providers
 
@@ -79,7 +79,8 @@ No resources.
 | <a name="input_datalake_image"></a> [datalake\_image](#input\_datalake\_image) | The image to use for the datalake. Can only be used when the 'datalake\_version' parameter is set to null. You can use 'catalog' name and/or 'id' for selecting an image. | <pre>object({<br>    id      = optional(string)<br>    catalog = optional(string)<br>  })</pre> | `null` | no |
 | <a name="input_datalake_java_version"></a> [datalake\_java\_version](#input\_datalake\_java\_version) | The Java major version to use on the datalake cluster. | `number` | `null` | no |
 | <a name="input_datalake_name"></a> [datalake\_name](#input\_datalake\_name) | Name of the CDP datalake. Defaults to '<env\_prefix>-<aw\|az\|gc\|>-dl' if not specified. | `string` | `null` | no |
-| <a name="input_datalake_recipes"></a> [datalake\_recipes](#input\_datalake\_recipes) | Additional recipes that will be attached on the datalake instances | <pre>list(<br>    object({<br>      instance_group_name = string,<br>      recipe_names        = string<br>    })<br>  )</pre> | `null` | no |
+| <a name="input_datalake_polling_timeout"></a> [datalake\_polling\_timeout](#input\_datalake\_polling\_timeout) | Timeout value in minutes for how long to poll for CDP datalake resource creation/deletion | `number` | `90` | no |
+| <a name="input_datalake_recipes"></a> [datalake\_recipes](#input\_datalake\_recipes) | Additional recipes that will be attached on the datalake instances | <pre>set(<br>    object({<br>      instance_group_name = string,<br>      recipe_names        = set(object({}))<br>    })<br>  )</pre> | `null` | no |
 | <a name="input_datalake_scale"></a> [datalake\_scale](#input\_datalake\_scale) | The scale of the datalake. Valid values are LIGHT\_DUTY, ENTERPRISE. | `string` | `null` | no |
 | <a name="input_datalake_version"></a> [datalake\_version](#input\_datalake\_version) | The Datalake Runtime version. Valid values are latest or a semantic version, e.g. 7.2.17 | `string` | `"latest"` | no |
 | <a name="input_enable_ccm_tunnel"></a> [enable\_ccm\_tunnel](#input\_enable\_ccm\_tunnel) | Flag to enable Cluster Connectivity Manager tunnel. If false then access from Cloud to CDP Control Plane CIDRs is required from via SG ingress | `bool` | `true` | no |
@@ -91,6 +92,7 @@ No resources.
 | <a name="input_endpoint_access_scheme"></a> [endpoint\_access\_scheme](#input\_endpoint\_access\_scheme) | The scheme for the workload endpoint gateway. PUBLIC creates an external endpoint that can be accessed over the Internet. PRIVATE which restricts the traffic to be internal to the VPC / Vnet. Relevant in Private Networks. | `string` | `null` | no |
 | <a name="input_env_prefix"></a> [env\_prefix](#input\_env\_prefix) | Shorthand name for the environment. Used in CDP resource descriptions. This will be used to construct the value of where any of the CDP resource variables (e.g. environment\_name, cdp\_iam\_admin\_group\_name) are not defined. | `string` | `null` | no |
 | <a name="input_environment_name"></a> [environment\_name](#input\_environment\_name) | Name of the CDP environment. Defaults to '<env\_prefix>-cdp-env' if not specified. | `string` | `null` | no |
+| <a name="input_environment_polling_timeout"></a> [environment\_polling\_timeout](#input\_environment\_polling\_timeout) | Timeout value in minutes for how long to poll for CDP Environment resource creation/deletion | `number` | `60` | no |
 | <a name="input_freeipa_catalog"></a> [freeipa\_catalog](#input\_freeipa\_catalog) | Image catalog to use for FreeIPA image selection | `string` | `null` | no |
 | <a name="input_freeipa_image_id"></a> [freeipa\_image\_id](#input\_freeipa\_image\_id) | Image ID to use for creating FreeIPA instances | `string` | `null` | no |
 | <a name="input_freeipa_instance_type"></a> [freeipa\_instance\_type](#input\_freeipa\_instance\_type) | Instance Type to use for creating FreeIPA instances | `string` | `null` | no |

modules/terraform-cdp-deploy/examples/ex01-aws-basic/main.tf

@@ -79,7 +79,7 @@ terraform {
   required_providers {
     cdp = {
       source  = "cloudera/cdp"
-      version = "0.1.4-pre"
+      version = "0.2.0"
     }
   }
 }

modules/terraform-cdp-deploy/main.tf

@@ -38,6 +38,9 @@ module "cdp_on_aws" {
   workload_analytics     = var.workload_analytics
   endpoint_access_scheme = local.endpoint_access_scheme
 
+  environment_polling_timeout = var.environment_polling_timeout
+  datalake_polling_timeout    = var.datalake_polling_timeout
+
   # TODO: Will be re-introducted once provider supports other regions
   # cdp_control_plane_region = var.cdp_control_plane_region
 
@@ -103,6 +106,9 @@ module "cdp_on_azure" {
   workload_analytics     = var.workload_analytics
   endpoint_access_scheme = local.endpoint_access_scheme
 
+  environment_polling_timeout = var.environment_polling_timeout
+  datalake_polling_timeout    = var.datalake_polling_timeout
+
   # TODO: Will be re-introducted once provider supports other regions
   # cdp_control_plane_region = var.cdp_control_plane_region
 

modules/terraform-cdp-deploy/modules/aws/main.tf

@@ -61,6 +61,11 @@ resource "cdp_environments_aws_environment" "cdp_env" {
   enable_tunnel       = var.enable_ccm_tunnel
 
   encryption_key_arn = var.encryption_key_arn
+
+  polling_options = {
+    polling_timeout = var.environment_polling_timeout
+  }
+
   # tags               = var.tags # NOTE: Waiting on provider fix
 
   depends_on = [
@@ -119,8 +124,8 @@ resource "cdp_datalake_aws_datalake" "cdp_datalake" {
   datalake_name    = var.datalake_name
   environment_name = cdp_environments_aws_environment.cdp_env.environment_name
 
-  instance_profile        = var.idbroker_instance_profile_arn
-  storage_bucket_location = var.data_storage_location
+  instance_profile      = var.idbroker_instance_profile_arn
+  storage_location_base = var.data_storage_location
 
   runtime           = var.datalake_version == "latest" ? null : var.datalake_version
   scale             = var.datalake_scale
@@ -131,6 +136,11 @@ resource "cdp_datalake_aws_datalake" "cdp_datalake" {
   image                  = var.datalake_image
   java_version           = var.datalake_java_version
   recipes                = var.datalake_recipes
+
+  polling_options = {
+    polling_timeout = var.datalake_polling_timeout
+  }
+
   # tags = var.tags # NOTE: Waiting on provider fix
 
   depends_on = [

modules/terraform-cdp-deploy/modules/aws/provider.tf

@@ -16,7 +16,7 @@ terraform {
   required_providers {
     cdp = {
       source  = "cloudera/cdp"
-      version = "0.1.4-pre"
+      version = "0.2.0"
     }
   }
 

modules/terraform-cdp-deploy/modules/aws/variables.tf

@@ -79,6 +79,13 @@ variable "multiaz" {
 
 }
 
+variable "environment_polling_timeout" {
+  type = number
+
+  description = " Timeout value in minutes for how long to poll for CDP Environment resource creation/deletion"
+
+}
+
 variable "freeipa_instances" {
   type = number
 
@@ -191,15 +198,22 @@ variable "datalake_java_version" {
 }
 
 variable "datalake_recipes" {
-  type = list(
+  type = set(
     object({
       instance_group_name = string,
-      recipe_names        = string
+      recipe_names        = set(object({}))
     })
   )
 
   description = "Additional recipes that will be attached on the datalake instances"
 
+}
+
+variable "datalake_polling_timeout" {
+  type = number
+
+  description = "Timeout value in minutes for how long to poll for CDP datalake resource creation/deletion"
+
 }
 # ------- Cloud Service Provider Settings -------
 variable "region" {

modules/terraform-cdp-deploy/modules/azure/main.tf

@@ -74,6 +74,10 @@ resource "cdp_environments_azure_environment" "cdp_env" {
   encryption_key_resource_group_name = var.encryption_key_resource_group_name
   encryption_key_url                 = var.encryption_key_url
 
+  polling_options = {
+    polling_timeout = var.environment_polling_timeout
+  }
+
   # tags               = var.tags # NOTE: Waiting on provider fix
 
   depends_on = [
@@ -133,8 +137,8 @@ resource "cdp_datalake_azure_datalake" "cdp_datalake" {
   datalake_name    = var.datalake_name
   environment_name = cdp_environments_azure_environment.cdp_env.environment_name
 
-  managed_identity = var.idbroker_identity_id
-  storage_location = var.data_storage_location
+  managed_identity      = var.idbroker_identity_id
+  storage_location_base = var.data_storage_location
 
   runtime           = var.datalake_version == "latest" ? null : var.datalake_version
   scale             = var.datalake_scale
@@ -143,6 +147,11 @@ resource "cdp_datalake_azure_datalake" "cdp_datalake" {
   image        = var.datalake_image
   java_version = var.datalake_java_version
   recipes      = var.datalake_recipes
+
+  polling_options = {
+    polling_timeout = var.datalake_polling_timeout
+  }
+
   # tags = var.tags # NOTE: Waiting on provider fix
 
   depends_on = [

modules/terraform-cdp-deploy/modules/azure/provider.tf

@@ -16,7 +16,7 @@ terraform {
   required_providers {
     cdp = {
       source  = "cloudera/cdp"
-      version = "0.1.4-pre"
+      version = "0.2.0"
     }
   }
 

modules/terraform-cdp-deploy/modules/azure/variables.tf

@@ -72,6 +72,13 @@ variable "enable_raz" {
 
 }
 
+variable "environment_polling_timeout" {
+  type = number
+
+  description = " Timeout value in minutes for how long to poll for CDP Environment resource creation/deletion"
+
+}
+
 variable "freeipa_instances" {
   type = number
 
@@ -187,15 +194,22 @@ variable "datalake_java_version" {
 }
 
 variable "datalake_recipes" {
-  type = list(
+  type = set(
     object({
       instance_group_name = string,
-      recipe_names        = string
+      recipe_names        = set(object({}))
     })
   )
 
   description = "Additional recipes that will be attached on the datalake instances"
 
+}
+
+variable "datalake_polling_timeout" {
+  type = number
+
+  description = "Timeout value in minutes for how long to poll for CDP datalake resource creation/deletion"
+
 }
 # ------- Cloud Service Provider Settings -------
 variable "subscription_id" {

modules/terraform-cdp-deploy/provider.tf

@@ -16,7 +16,7 @@ terraform {
   required_providers {
     cdp = {
       source  = "cloudera/cdp"
-      version = "0.1.4-pre"
+      version = "0.2.0"
     }
   }
 

modules/terraform-cdp-deploy/variables.tf

@@ -118,6 +118,14 @@ variable "enable_raz" {
   default = true
 }
 
+variable "environment_polling_timeout" {
+  type = number
+
+  description = " Timeout value in minutes for how long to poll for CDP Environment resource creation/deletion"
+
+  default = 60
+}
+
 variable "multiaz" {
   type = bool
 
@@ -243,10 +251,10 @@ variable "datalake_java_version" {
 }
 
 variable "datalake_recipes" {
-  type = list(
+  type = set(
     object({
       instance_group_name = string,
-      recipe_names        = string
+      recipe_names        = set(object({}))
     })
   )
 
@@ -255,6 +263,14 @@ variable "datalake_recipes" {
   default = null
 }
 
+variable "datalake_polling_timeout" {
+  type = number
+
+  description = "Timeout value in minutes for how long to poll for CDP datalake resource creation/deletion"
+
+  default = 90
+}
+
 # ------- CDP Environment Deployment - AWS specific -------
 variable "encryption_key_arn" {
   type = string