Name	Name	Last commit message	Last commit date
parent directory ..
images	images
scripts/build	scripts/build
templates	templates
README.md	README.md
budget.tf	budget.tf
elastic_search_deploy.tf	elastic_search_deploy.tf
gke.tf	gke.tf
iam.tf	iam.tf
main.tf	main.tf
network.tf	network.tf
orgpolicy.tf	orgpolicy.tf
outputs.tf	outputs.tf
provider-auth.tf	provider-auth.tf
variables.tf	variables.tf
versions.tf	versions.tf

RAD Lab Application Mordernization Module (w/ Elasticsearch)

Module Overview

This module allows the user to create an Elasticsearch cluster, deployed on a GKE cluster in Google Cloud Platform. It follows the Quickstart-tutorial available on https://elastic.co.

GCP Products/Services

Google Kubernetes Engine
Virtual Private Cloud (VPC)
Billing Budget

Reference Architecture Diagram

Below Architechture Diagram is the base representation of what will be created as a part of RAD Lab Launcher.

API Prerequisites

In the RAD Lab Management Project make sure that Cloud Billing Budget API (billingbudgets.googleapis.com) is enabled. NOTE: This is only required if spinning up Billing Budget for the module.

IAM Permissions Prerequisites

Ensure that the identity executing this module has the following IAM permissions, when creating the project (create_project = true):

Parent: roles/billing.user
Parent: roles/billing.costsManager (OPTIONAL - Only when spinning up Billing Budget for the module)
Parent: roles/resourcemanager.projectCreator
Parent: roles/orgpolicy.policyAdmin (OPTIONAL - Only required if setting any Org policy in modules/[MODULE_NAME]/orgpolicy.tf as part of RAD Lab module)

NOTE: Billing budgets can only be created if you are using a Service Account to deploy the module via Terraform, User account cannot be used.

When deploying in an existing project, ensure the identity has the following permissions on the project:

roles/compute.admin
roles/container.admin
roles/logging.admin
roles/monitoring.admin
roles/iam.serviceAccountAdmin
roles/iam.serviceAccountUser
roles/resourcemanager.projectIamAdmin
roles/serviceusage.serviceUsageAdmin
roles/billing.costsManager (OPTIONAL - Only when spinning up Billing Budget for the module)

Deployments via Service Account

Create a Terraform Service Account in RAD Lab Management Project to execute / deploy the RAD Lab module. Ensure that the Service Account has the above mentioned IAM permissions. NOTE: Make sure to set the resource_creator_identity variable to the Service Account ID in terraform.tfvars file and pass it in module deployment. Example content of terraform.tfvars:

resource_creator_identity = <sa>@<projectID>.iam.gserviceaccount.com

The User, Group, or Service Account who will be deploying the module should have access to impersonate and grant it the roles, roles/iam.serviceAccountTokenCreator on the Terraform Service Account’s IAM Policy. NOTE: This is not a Project IAM Binding; this is a Service Account IAM Binding.

NOTE: Additional permissions are required when deploying the RAD Lab modules via RAD Lab Launcher. Use --disable-perm-check or -dc arguments when using RAD lab Launcher for the module deployment.

Usage:

python3 radlab.py --disable-perm-check --varfile /<path_to_file>/<file_with_terraform.tfvars_contents>

Deploy Elasticsearch

The module deploys both the ECK CRDs and Operators. As this module can be used to demo Elasticsearch, it also deploys an ES and Kibana pod in the cluster. This behaviour can be switched off by setting deploy_elastic_search to false. This will only deploy the CRDs and Operators.

Access Deployment

If the user follows the standard flow of the installation, the module will deploy a pod with Kibana and a pod with ElasticSearch. Kibana exposes a UI that can be accessed via a web browser and ElasticSearch exposes an API server that can be queried via curl.

General Commands

Regardless of which service you are trying to access, these commands will have to be run in a Terminal or Cloud Shell.

# Retrieve credentials to query the Kubernetes API server.  Replace REGION and PROJECTID with the actual values.  You can copy/paste this command from the Terraform output.
gcloud container clusters get-credentials elastic-search-cluster --region REGION --project PROJECTID

# Check status Elasticsearch. The health column should show status green.  Takes around 5 minutes to complete 
kubectl get elasticsearch -n elastic-search-demo

# Check status Kibana.  The health column should show status green.  It can take a while for the pod to become available.
kubectl get kibana -n elastic-search-demo

# Retrieve password.  Not that this stores the password as an environment variable, so only use this for testing purposes.
ES_PWD=$(kubectl get secret elastic-search-es-elastic-user -n elastic-search-demo -o go-template='{{.data.elastic | base64decode}}')

ElasticSearch

ElasticSearch itself exposes an API server which can't be accessed in a web browser. The commands listed below can be run in both a local terminal or Cloud Shell, provided that the commands listed in General Commands.

# Start port-forwarding tunnel in the background
kubectl port-forward -n elastic-search-demo service/elastic-search-es-http 9200 &> /tmp/elastic_search_fwd &

# Access the API server via command line
curl -X GET -k -u elastic:$ES_PWD https://localhost:9200

The Elastic Search API server is now accessible on https://localhost:9200, so you can run additional commands against this endpoint, if desired.

Kibana

It's currently not possible to run kubectl port-forward and access the endpoint via the web preview in Cloud Shell. The commands below have to be run from a local terminal instead. If you use the RAD Lab launcher from Cloud Shell, you will have to execute the following commands in a terminal on your local machine. Make sure that you are logged in with the same user locally, as the one you used to run the launcher. You can do this by running gcloud auth login.

Ensure that all commands listed in General Commands have been run before running the command below.

# If pbcopy is not installed, only run echo $ES_PWD and copy the password manually
echo $ES_PWD | pbcopy

# Start port-forwarding tunnel
kubectl port-forward -n elastic-search-demo service/kibana-kb-http 5601

# Open a browser window and point it to https://localhost:5601. Login with username elastic and the password copied from the command above.

Using Terraform module

Here are a couple of examples to directly use the Terraform module, as opposed to using the RAD Lab Launcher.

Simple

module "elastic_search_simple" {
  source = "./app_mod_elastic"

  billing_account_id = "123456-123456-123456"
  organization_id    = "12345678901"
  folder_id          = "1234567890"
}

Use existing project

Replace pref-project-id with an existing project ID.

module "elastic_search_project" {
  source = "./app_mod_elastic"

  billing_account_id = "123456-123456-123456"
  organization_id    = "12345678901"
  folder_id          = "1234567890"
  create_project     = false
  project_name       = "pref-project-id"
}

Use existing network

Both the project and the network has to exist already for this to work. Additionally, if all the resources for egress traffic have already been created, set enable_internet_egress_traffic to false.

module "elastic_search_project" {
  source = "./app_mod_elastic"

  billing_account_id = "123456-123456-123456"
  organization_id    = "12345678901"
  folder_id          = "1234567890"
  create_project     = false
  project_name       = "pref-project-id"
  create_network     = false
  network_name       = "network-name"
  subnet_name        = "subnet-name"
}

Variables

name	description	type	required	default
billing_account_id	Billing account ID that will be linked to the project	`string`	✓
billing_budget_alert_spend_basis	The type of basis used to determine if spend has passed the threshold	`string`		`CURRENT_SPEND`
billing_budget_alert_spent_percents	A list of percentages of the budget to alert on when threshold is exceeded	`list(number)`		`[0.5, 0.7, 1]`
billing_budget_amount	The amount to use as the budget in USD	`number`		`500`
billing_budget_amount_currency_code	The 3-letter currency code defined in ISO 4217 (https://cloud.google.com/billing/docs/resources/currency#list_of_countries_and_regions). It must be the currency associated with the billing account	`string`		`USD`
billing_budget_credit_types_treatment	Specifies how credits should be treated when determining spend for threshold calculations	`string`		`INCLUDE_ALL_CREDITS`
billing_budget_labels	A single label and value pair specifying that usage from only this set of labeled resources should be included in the budget	`map(string)`		`...`
billing_budget_notification_email_addresses	A list of email addresses which will be recieving billing budget notification alerts. A maximum of 4 channels are allowed as the first element of `trusted_users` is automatically added as one of the channel	`set(string)`		`...`
billing_budget_pubsub_topic	If true, creates a Cloud Pub/Sub topic where budget related messages will be published. Default is false	`bool`		`false`
billing_budget_services	A list of services ids to be included in the budget. If omitted, all services will be included in the budget. Service ids can be found at https://cloud.google.com/skus/	`list(string)`		`null`
create_budget	If the budget should be created	`bool`		`false`
create_network	Indicate if the deployment has to use a network that already exists	`bool`		`true`
create_project	Create a new project or use an existing project. When set to true, variable project_name has to match the exact project ID of the existing project	`bool`		`true`
deploy_elastic_search	Deploy Elastic Search and Kibana	`bool`		`true`
deployment_id	Random ID that will be used to suffix all resources. Leave blank if you want the module to use a generated one	`string`		`null`
disk_size_gb_nodes	Size of the disks attached to the nodes	`number`		`256`
disk_type_nodes	Type of disks to attach to the nodes	`string`		`pd-standard`
elastic_search_instance_count	Number of instances of the Elastic Search pod	`string`		`1`
elk_version	Version for Elastic Search and Kibana	`string`		`7.15.1`
enable_internet_egress_traffic	Enable egress traffic to the internet. Necessary to download the Elastic Search pods	`bool`		`true`
enable_services	Enable the necessary APIs on the project. When using an existing project, this can be set to false	`bool`		`true`
folder_id	Folder ID where the project should be created. It can be skipped if already setting organization_id. Leave blank if the project should be created directly underneath the Organization node	`string`
gke_cluster_name	Name that will be assigned to the GKE cluster	`string`		`elastic-search-cluster`
gke_version	Version to be used for the GKE cluster. Ensure that the release channel is properly set when updating this variable	`string`		`1.20.10-gke.1600`
kibana_instance_count	Number of Kibana instances deployed in the cluster	`string`		`1`
master_ipv4_cidr_block	IPv4 CIDR block to assign to the Master cluster	`string`		`10.200.0.0/28`
network_cidr_block	CIDR block to be assigned to the network	`string`		`10.0.0.0/16`
network_name	Name to be assigned to the network hosting the GKE cluster	`string`		`elastic-search-nw`
node_pool_machine_type	Machine type for the node pool	`string`		`e2-medium`
node_pool_max_count	Maximum instance count for the custom node pool	`number`		`10`
node_pool_min_count	Minimum instance count for the custom nodepool	`number`		`1`
node_pool_name	Name of the nodepool	`string`		`elastic-search-pool`
organization_id	Organization ID where the project will be created. It can be skipped if already setting folder_id	`string`
owner_groups	List of groups that should be added as the owner of the created project	`list(string)`		`[]`
owner_users	List of users that should be added as owner to the created project	`list(string)`		`[]`
pod_cidr_block	CIDR block to be assigned to pods running in the GKE cluster	`string`		`10.100.0.0/16`
pod_ip_range_name	Range name for the pod IP addresses	`string`		`pod-ip-range`
preemptible_nodes	Use preemptible VMs for the node pools	`bool`		`true`
project_id_prefix	If `create_project` is true, this will be the prefix of the Project ID & name created. If `create_project` is false this will be the actual Project ID, of the existing project where you want to deploy the module	`string`		`radlab-app-mod-elastic`
region	Region where the resources should be created	`string`		`us-west1`
release_channel	Enroll the GKE cluster in this release channel	`string`		`REGULAR`
resource_creator_identity	Terraform Service Account which will be creating the GCP resources. If not set, it will use user credentials spinning up the module	`string`
service_cidr_block	CIDR block to be assigned to services running in the GKE cluster	`string`		`10.150.0.0/16`
service_ip_range_name	Name for the IP range for services	`string`		`service-ip-range`
set_domain_restricted_sharing_policy	Enable org policy to allow all principals to be added to IAM policies	`bool`		`false`
set_shielded_vm_policy	Apply org policy to disable shielded VMs	`bool`		`false`
set_vpc_peering_policy	Enable org policy to VPC Peering	`bool`		`false`
subnet_name	Name to be assigned to the subnet hosting the GKE cluster	`string`		`elastic-search-snw`
trusted_groups	The list of trusted groups (e.g. `[email protected]`)	`set(string)`		`[]`
trusted_users	The list of trusted users (e.g. `[email protected]`)	`set(string)`		`[]`

Outputs

name	description	sensitive
billing_budget_budget_id	Resource name of the budget. Values are of the form `billingAccounts/{billingAccountId}/budgets/{budgetId}`	✓
cluster_credentials_cmd	None
deployment_id	RADLab Module Deployment ID
project_id	None

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

app_mod_elastic

app_mod_elastic

README.md

RAD Lab Application Mordernization Module (w/ Elasticsearch)

Module Overview

GCP Products/Services

Reference Architecture Diagram

API Prerequisites

IAM Permissions Prerequisites

Deployments via Service Account

Deploy Elasticsearch

Access Deployment

General Commands

ElasticSearch

Kibana

Using Terraform module

Simple

Use existing project

Use existing network

Variables

Outputs

Files

app_mod_elastic

Directory actions

More options

Directory actions

More options

Latest commit

History

app_mod_elastic

Folders and files

parent directory

README.md

RAD Lab Application Mordernization Module (w/ Elasticsearch)

Module Overview

GCP Products/Services

Reference Architecture Diagram

API Prerequisites

IAM Permissions Prerequisites

Deployments via Service Account

Deploy Elasticsearch

Access Deployment

General Commands

ElasticSearch

Kibana

Using Terraform module

Simple

Use existing project

Use existing network

Variables

Outputs