-
Notifications
You must be signed in to change notification settings - Fork 45
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update docs to include new probe information along with other cleanup (…
- Loading branch information
Showing
3 changed files
with
74 additions
and
115 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,21 +1,25 @@ | ||
| ### Table of Contents ### | ||
|
|
||
| - [Setup](#setup) | ||
| - [AWS Environment](#aws-environment) | ||
| - [VPC](#vpc) | ||
| - [IAM Permissions](#iam-permissions) | ||
| - [Available tools](#available-tools) | ||
| - [1. Egress Verification](#1-egress-verification) | ||
| - [1.1 Usage](#11-usage) | ||
| - [1.1.1 CLI Executable](#111-cli-executable) | ||
| - [1.1.2 Go Implementation Examples](#112-go-implementation-examples) | ||
| - [1.2 Interpreting Output](#12-interpreting-output) | ||
| - [1.3 Workflow](#13-workflow) | ||
| - [2. VPC DNS Verification](#2-vpc-dns-verification) | ||
| - [2.1 Usage](#21-usage) | ||
| - [2.1.1 CLI Executable](#211-cli-executable) | ||
| - [2.1.2 Golang API](#212-golang-api) | ||
| - [3. BYOVPC Configurations Verification](#3-byovpc-configurations-verification) | ||
| <!-- TOC --> | ||
| * [Table of Contents](#table-of-contents-) | ||
| * [Setup](#setup-) | ||
| * [AWS Environment](#aws-environment-) | ||
| * [VPC](#vpc-) | ||
| * [IAM permissions](#iam-permissions-) | ||
| * [Available Tools](#available-tools-) | ||
| * [1. Egress Verification](#1-egress-verification-) | ||
| * [1.1 Usage](#11-usage-) | ||
| * [1.1.1 CLI Executable](#111-cli-executable-) | ||
| * [Egress Validations Under Proxy](#egress-validations-under-proxy-) | ||
| * [Force Temporary Security Group Creation](#force-temporary-security-group-creation-) | ||
| * [1.1.2 Go implementation Examples](#112-go-implementation-examples-) | ||
| * [1.2 Interpreting Output](#12-interpreting-output-) | ||
| * [1.3 Workflow](#13-workflow-) | ||
| * [2. VPC DNS Verification](#2-vpc-dns-verification-) | ||
| * [2.1 Usage](#21-usage-) | ||
| * [2.1.1 CLI Executable](#211-cli-executable-) | ||
| * [2.1.2 Golang API](#212-golang-api-) | ||
| <!-- TOC --> | ||
|
|
||
| ## Setup ## | ||
| ### AWS Environment ### | ||
|
|
@@ -73,40 +77,16 @@ Ensure that the AWS credentials being used have the following permissions. (This | |
| ] | ||
| } | ||
| ``` | ||
|
|
||
| The SRE only needs below permissions because we should supply Security Group ID by running `./osd-network-verifier egress --security-group-id <SG_ID>`: | ||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "ec2:CreateTags", | ||
| "ec2:RunInstances", | ||
| "ec2:DescribeInstances", | ||
| "ec2:DescribeInstanceTypes", | ||
| "ec2:GetConsoleOutput", | ||
| "ec2:TerminateInstances", | ||
| "ec2:DescribeVpcAttribute", | ||
| ], | ||
| "Resource": "*" | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| ``` | ||
|
|
||
| ## Available Tools ## | ||
|
|
||
| ### 1. Egress Verification ### | ||
| #### 1.1 Usage #### | ||
| The processes below describe different ways of using egress verifier on a single subnet. | ||
| In order to verify entire VPC, | ||
| repeat the verification process for each subnet ID. | ||
| To verify the entire VPC, repeat the verification process for each subnet ID. | ||
|
|
||
| ##### 1.1.1 CLI Executable ##### | ||
| 1. Ensure correct [environment setup](#setup). | ||
| 1. Ensure correct [environment setup](#setup-). | ||
|
|
||
| 2. Clone the source: | ||
| ```shell | ||
|
|
@@ -119,16 +99,8 @@ repeat the verification process for each subnet ID. | |
| This generates `osd-network-verifier` executable in project root directory. | ||
|
|
||
| 4. Obtain params: | ||
| 1. subnet_id: Obtain the subnet id to be verified. | ||
| 2. image_id: Select an optional image id parameter (ami-xxxxxxxxxxxx) to run on ec2 instance. | ||
|
|
||
| You may use the following public image ID as : | ||
| ```bash | ||
| --image-id=resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 | ||
| ``` | ||
| If the image id is not provided, it is defaulted to an image id from [AWS account olm-artifacts-template.yaml](https://github.com/openshift/aws-account-operator/blob/17be7a41036e252d59ab19cc2ad1dcaf265758a2/hack/olm-registry/olm-artifacts-template.yaml#L75), | ||
| for the same region where your subnet is. | ||
| 3. platform: This parameter dictates for which set of endpoints the verifier should test. If testing a subnet that hosts (or will host) a traditional OSD/ROSA cluster, set this to `aws` (or leave blank). If you're instead testing a subnet hosting a HyperShift Hosted Cluster (*not* a hosted control plane/management cluster) on AWS, set this to `hostedcluster`. | ||
| 1. subnet-id: the subnet id to be verified. | ||
| 2. platform: This parameter dictates for which set of endpoints the verifier should test. If testing a subnet that hosts (or will host) a traditional OSD/ROSA cluster, set this to `aws` (or leave blank). If you're instead testing a subnet hosting a HyperShift Hosted Cluster (*not* a hosted control plane/management cluster) on AWS, set this to `hostedcluster`. | ||
| 5. Execute: | ||
|
|
@@ -141,43 +113,15 @@ repeat the verification process for each subnet ID. | |
| ./osd-network-verifier egress --platform aws-hcp --subnet-id $SUBNET_ID | ||
| ``` | ||
| Additional optional flags for overriding defaults: | ||
| ```shell | ||
| --cacert string (optional) path to cacert file to be used upon https requests being made by verifier | ||
| --cloud-tags stringToString (optional) comma-seperated list of tags to assign to cloud resources e.g. --cloud-tags key1=value1,key2=value2 (default []) | ||
| --cpu-arch string (optional) compute instance CPU architecture. Ignored if valid instance-type specified | ||
| --debug (optional) if true, enable additional debug-level logging | ||
| --egress-list-location string (optional) the location of the egress URL list to use. Can either be a local file path or an external URL starting with http(s). This value is ignored for the legacy probe. | ||
| --force-temp-security-group (optional) Enforces creation of Temporary SG creation even if --security-group-ids flag is used | ||
| --http-proxy string (optional) http-proxy to be used upon http requests being made by verifier, format: http://user:[email protected]:8978 | ||
| --https-proxy string (optional) https-proxy to be used upon https requests being made by verifier, format: https://user:[email protected]:8978 | ||
| --image-id string (optional) cloud image for the compute instance | ||
| --import-keypair string (optional) Takes the path to your public key used to connect to Debug Instance. Automatically skips Termination | ||
| --instance-type string (optional) compute instance type | ||
| --kms-key-id string (optional) ID of KMS key used to encrypt root volumes of compute instances. Defaults to cloud account default key | ||
| --no-tls (optional) if true, skip client-side SSL certificate validation | ||
| --platform string (optional) infra platform type, which determines which endpoints to test. Either 'aws-classic', 'gcp-classic', or 'hosted-cp' (hypershift) (default "aws-hcp") | ||
| --probe string (optional) select the probe to be used for egress testing. Either 'Curl' (default) or 'Legacy' (default "Curl") | ||
| --profile string (optional) AWS profile. If present, any credentials passed with CLI will be ignored | ||
| --region string (optional) compute instance region. If absent, environment var AWS_REGION = us-east-2 and GCP_REGION = us-east1 will be used | ||
| --security-group-ids strings (optional) comma-separated list of sec. group IDs to attach to the created EC2 instance. If absent, one will be created | ||
| --skip-termination (optional) Skip instance termination to allow further debugging | ||
| --subnet-id string source subnet ID | ||
| --terminate-debug string (optional) Takes the debug instance ID and terminates it | ||
| --timeout duration (optional) timeout for individual egress verification requests (default 2s) | ||
| --vpc-name string (optional unless --platform='gcp-classic') VPC name where GCP cluster is installed | ||
| ``` | ||
| Get cli help: | ||
| Additional optional flags for overriding defaults can be found with: | ||
| ```shell | ||
| ./osd-network-verifier egress --help | ||
| ``` | ||
| ##### Egress Validations Under Proxy ##### | ||
| * Follow the similar flow above, till execute | ||
| * Pass proxy config to be used to egress subcommand | ||
| * Follow the same flow shown above, until execution | ||
| * Pass the proxy config to the egress subcommand | ||
| ```shell | ||
| ./osd-network-verifier egress \ | ||
|
|
@@ -187,52 +131,46 @@ repeat the verification process for each subnet ID. | |
| --cacert path-to-ca.pem \ | ||
| --no-tls # optional, used to bypass ca.pem validation (https) | ||
| ``` | ||
| ##### Force Temporary Security Group Creation ##### | ||
| * Follow the similar flow above, till execute | ||
| * Use the --force-temp-security-group | ||
| * Use the `--force-temp-security-group` flag | ||
| ```shell | ||
| ./osd-network-verifier egress \ | ||
| --subnet-id <subnet_id> \ | ||
| --force-temp-security-group true \ | ||
| --force-temp-security-group \ | ||
| --security-group-ids=<securityGroupID-1, ..., securityGroupID-N> # To add extra security Groups in addtion to the temporary one. | ||
| ``` | ||
| ##### 1.1.2 Go implementation Examples ##### | ||
| - [AWS Go SDK v1](../../examples/aws/verify_egressv1.go) | ||
| - [AWS Go SDK v2](../../examples/aws/verify_egressv2.go) | ||
| - [Verify Egress Example](../../examples/aws/verify_egress.go) | ||
| #### 1.2 Interpreting Output ### | ||
| (TODO: add errors) | ||
| #### 1.3 Workflow #### | ||
| Pictorial representation of workflow of the egress test tool: | ||
| Pictorial representation of the egress test tool workflow: | ||
|  | ||
| Description: | ||
| The AWS client creates a test EC2 instance in the target VPC/subnet and waits until the instance is ready. | ||
| The actual network verification is automated | ||
| by using the `USERDATA` param [available for ec2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html) | ||
| which is run by ec2 on the instance on creation. | ||
| 1. AWS client creates a test ec2 instance in the target vpc/subnet and wait till the instance gets ready | ||
| 2. The actual network verification is automated by using the `USERDATA` param [available for ec2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html) which is run by ec2 on the instance on creation. | ||
| 3. The [`USERDATA`](../../pkg/helpers/config/userdata.yaml) script is in the form of base64-encoded text, and does the following - | ||
| The instance's console output is redirected to the AWS cloud client SDK. | ||
| The active probe then parses this output before the verifier prints it to the user's terminal. | ||
| If debug logging is enabled, the verifier prints this output is printed in full; otherwise it only prints errors. | ||
| 1. installs docker | ||
| 2. runs [validator's docker image](https://gitlab.cee.redhat.com/service/osd-network-verifier-golden-ami/-/blob/master/build/bin/network-validator.go). Firstly, the image of the validator is tried to be pulled. If it fails, then the docker image baked into the AMI is used. | ||
| (The image is also published at: https://quay.io/repository/app-sre/osd-network-verifier) | ||
| 3. The entry point of the osd-network-verifier docker image then executes the main egress verification script | ||
| ```shell | ||
| network-validator --timeout=2s --config=config/config.yaml | ||
| ``` | ||
| - **This entrypoint is where the actual egress endpoint verification is performed.** `build/bin/network-validator.go` makes `curl` requests to each other endpoint in the [egress list](../../README.md#egress-list) (i.e. list of all essential domains for OSD clusters). | ||
| - During development, the verifier docker image can be tested locally as: | ||
| ```shell | ||
| docker run --env "AWS_REGION=us-east-1" quay.io/app-sre/osd-network-verifier:latest --timeout=2s | ||
| ``` | ||
|
|
||
| 4. `USERDATA` script then redirects the instance's console output to the AWS cloud client SDK. The end of this output message is signified with a special End Verification string. | ||
| 5. If debug logging is enabled, this output is printed in full, otherwise only errors are printed, if any. | ||
| --- | ||
| **NOTE** | ||
| For more information on probes, see [the README](../../README.md#probes). | ||
| --- | ||
| ### 2. VPC DNS Verification ### | ||
| #### 2.1 Usage #### | ||
|
|
||