docs: add Layer 2 example including interconnection and VRF (#232)

This serves as a working example of deploying a Metal device in hybrid mode in a VRF with a Metal-billed VRF interconnection that connects to a VPC in AWS. This example also creates an S3 VPC endpoint in AWS, and demonstrates that the VPC endpoint can be used from the Metal device for `aws s3` commands. Closes #162 --------- Co-authored-by: Marques Johansson <[email protected]> Co-authored-by: Chris Privitere <[email protected]>
equinix · Oct 29, 2024 · 03e43a9 · 03e43a9
1 parent 1944f49
commit 03e43a9
Show file tree

Hide file tree

Showing 4 changed files with 403 additions and 0 deletions.
diff --git a/examples/layer2/README.md b/examples/layer2/README.md
@@ -0,0 +1,122 @@
+# Layer 2 networking with Equinix Metal
+
+This example demonstrates the use of the `equinix.cloud.metal_connection`, `equinix.cloud.metal_device`, and `equinix.cloud.metal_port` modules--as well as a variety of AWS modules--to configure Layer 2 connectivity from an Equinix Metal device to AWS S3 over a Metal-billed Fabric interconnection.
+
+## Overview
+
+The [pre-Fabric playbook](pre_fabric.yml) creates a new project, a VLAN, a VRF, a VRF Metal Gateway, and a device, converts the device to [hybrid bonded mode](https://deploy.equinix.com/developers/docs/metal/layer2-networking/overview/#network-configuration-types), and then creates a Metal-billed VRF interconnection and configures BGP peering settings on the interconnection's virtual circuit.
+
+Manual intervention is needed in order to finish setting up the interconnection and accept the Direct Connect request in AWS.
+
+The [post-Fabric playbook](post_fabric.yml) creates a new VPC, a VPC endpoint for S3, and a Virtual Private Gateway attached to the specified Direct Connect, and configures BGP peering between the Direct Connect and your Metal VRF.
+
+## Prerequisites
+
+Before running the playbook, you will need to have the following:
+
+- [Ansible installed on your local machine.](https://docs.ansible.com/ansible/latest/installation_guide/installation_distros.html)
+- The Community.Aws, and Equinix Ansible Collections installed. You can install them using the following commands:
+  ```bash
+  ansible-galaxy collection install equinix.cloud
+  ansible-galaxy collection install community.aws
+  ```
+- You will also need to ensure that the necessary Python libraries are installed:
+  ```bash
+  # Install Equinix Ansible collection dependencies
+  pip install -r https://raw.githubusercontent.com/equinix/ansible-collection-equinix/v0.11.1/requirements.txt
+  # Install AWS collection and Ansible IP function dependencies
+  pip install boto3 netaddr
+  ```
+- An [Equinix Metal API token](https://deploy.equinix.com/developers/docs/metal/identity-access-management/api-keys/). You can obtain an API token from the Equinix Metal Portal. Set the environment variable METAL_AUTH_TOKEN to your API token:
+  ```bash
+  export METAL_AUTH_TOKEN=your_api_token_here
+  ```
+
+## Variables
+
+You can customize some variables, such as Equinix Metal device hostname and IP ranges for Equinix Metal and AWS, from [vars/vars.yml](vars/vars.yml).
+
+## Running the Playbooks
+
+This example contains multiple playbooks and requires manual intervention between the playbooks.
+
+To create the Equinix Metal infrastructure for this example, navigate to the directory containing the playbook file `pre_fabric.yml` and run the following command:
+
+```bash
+ansible-playbook pre_fabric.yml -extra-vars "bgp_md5_password=<some_value>"
+```
+
+*NOTE:* The API performs some validation on the md5 for BGP.  For the latest rules refer to [the VRF virtual circuit API docs](https://deploy.equinix.com/developers/api/metal/#tag/Interconnections/operation/updateVirtualCircuit). As of this writing, the md5:
+* must be 10-20 characters long
+* may not include punctuation
+* must be a combination of numbers and letters
+* must contain at least one lowercase, uppercase, and digit character
+
+The last task in the `pre_fabric.yml` playbook will print out the service token for your Metal connection:
+
+```bash
+TASK [print service token to redeem in Fabric portal] **************************************************************************
+ok: [localhost] => {
+    "connection.service_tokens[0].id": "<service_token_id>"
+}
+```
+
+After the Equinix Metal infrastructure is created, you will need to redeem the service token for your connection in the [Equinix portal](https://portal.equinix.com). Navigate to Fabric -> Connect to Provider, choose AWS, and finally AWS Direct Connect. Choose Primary, put in your account number, choose the metro, click next, then choose "Service Token" from the drop down, and put in the service token.  You will be prompted to name your connection; **take note of the name you use**, you will need it for the next playbook.
+
+To finish setting up the AWS infrastructure, run the following command which will accept the direct connect request in AWS; wait for the connection to become active; create an AWS VPC, VPC endpoint, and VPN gateway; create a virtual interface connecting the VPC to your direct connect, and configure the Metal side of your interconnection to connect to the virtual interface in AWS:
+
+```bash
+ansible-playbook post_fabric.yml --extra-vars "bgp_md5_password=<some_value>" --extra-vars "aws_connection_name=<your_direct_connect_name>"
+```
+
+The last task in the `post_fabric.yml` playbook will print the DNS hostname for your S3 VPC endpoint:
+
+```bash
+TASK [print DNS name for VPC endpoint] *****************************************************************************************
+ok: [localhost] => {
+    "msg": "vpce-<some_id>.s3.us-west-1.vpce.amazonaws.com"
+}
+```
+
+
+
+## Testing the VPC endpoint and interconnection
+
+The DNS entry for your VPC endpoint is public, so you can look up the corresponding IP address from any Internet-connected computer.  You will see that it resolves to an IP address within your private VPC address space (the example below uses the default VPC CIDR for this module, `172.16.0.0/16`):
+
+```bash
+$ dig vpce-<some_id>.s3.us-west-1.vpce.amazonaws.com
+# ...
+;; ANSWER SECTION:
+vpce-<some_id>.s3.us-west-1.vpce.amazonaws.com. 60 IN A 172.16.94.176
+# ...
+```
+
+Since this address resolves to an IP within your VPC, though, you can only connect to it from an EC2 instance in your VPC or from the Metal device you deployed earlier.
+
+SSH in to the Metal device that was created by the `pre_fabric.yml` playbook.
+
+Install the AWS CLI:
+
+```bash
+$ apt install -y awscli
+```
+
+Configure your [AWS CLI credentials](https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-authentication.html).  For example, if you want to store your credentials in environment variables, it will look something like this:
+
+```bash
+$ export AWS_ACCESS_KEY_ID=<some_aws_key_id>
+$ export AWS_SECRET_ACCESS_KEY=<some_aws_access_key>
+$ export AWS_DEFAULT_REGION=us-west-1
+```
+
+You can now use the AWS CLI with your VPC endpoint to interact with the S3 service by adding the `bucket.` prefix to your VPC endpoint hostname:
+
+```bash
+$ aws s3 ls --endpoint-url https://bucket.vpce-<some_id>.s3.us-west-1.vpce.amazonaws.com
+2021-03-22 11:13:54 <some_bucket>
+2021-03-22 11:13:54 <some_other_bucket>
+...
+```
+
+You can learn about other usages of the S3 VPC endpoint with AWS CLI in [the AWS PrivateLink docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html).
diff --git a/examples/layer2/post_fabric.yml b/examples/layer2/post_fabric.yml
@@ -0,0 +1,156 @@
+---
+# NOTE: this playbook should be run _after_:
+# 1. Running the pre_fabric.yml playbook
+# 2. Redeeming the Fabric service token in the Fabric portal
+- name: Equinix Layer 2 example -- AWS resources
+  hosts: localhost
+  gather_facts: no
+  tasks:
+    - name: Include the required variables
+      include_vars: "vars/vars.yml"
+
+    - name: confirm the direct connect
+      community.aws.directconnect_confirm_connection:
+        region: "{{ aws_region }}"
+        name: "{{ aws_connection_name }}"
+      register: confirm_response
+
+    - name: wait for the direct connect to become available
+      community.aws.directconnect_connection:
+        region: "{{ aws_region }}"
+        name: "{{ aws_connection_name }}"
+        # The below attributes are required by the community.aws.directconnect_connection
+        # module but do not appear to be used, since this works for looking up the connection
+        location: "dummy"
+        bandwidth: "1Gbps"
+        state: present
+      register: connection_response
+      until: connection_response.connection.connection_state == "available"
+      retries: 10
+      delay: 60
+
+    - name: create a VPC
+      amazon.aws.ec2_vpc_net:
+        region: "{{ aws_region }}"
+        name: "{{ vpc_name }}"
+        cidr_block: "{{ aws_network_cidr }}"
+      register: created_vpc
+
+    - name: Create a subnet in the VPC
+      amazon.aws.ec2_vpc_subnet:
+        region: "{{ aws_region }}"
+        vpc_id: "{{ created_vpc.vpc.id }}"
+        cidr: "{{ aws_network_cidr }}"
+      register: created_subnet
+
+    - name: create security group for VPC endpoint
+      amazon.aws.ec2_security_group:
+        region: "{{ aws_region }}"
+        vpc_id: "{{ created_vpc.vpc.id }}"
+        name: "{{ security_group_name }}"
+        description: sg for Equinix Ansible Layer 2 example
+        rules:
+          - proto: all
+            cidr_ip: 0.0.0.0/0
+            rule_desc: allow all traffic
+      register: created_sg
+
+    - name: Create new VPC endpoint for S3
+      amazon.aws.ec2_vpc_endpoint:
+        region: "{{ aws_region }}"
+        vpc_id: "{{ created_vpc.vpc.id }}"
+        service: "com.amazonaws.{{ aws_region }}.s3"
+        vpc_endpoint_type: "Interface"
+        vpc_endpoint_subnets:
+          - "{{ created_subnet.subnet.id }}"
+        vpc_endpoint_security_groups:
+          - "{{ created_sg.group_id }}"
+      register: created_vpc_endpoint
+
+    - name: Create a new VGW attached to the VPC
+      community.aws.ec2_vpc_vgw:
+        region: "{{ aws_region }}"
+        vpc_id: "{{ created_vpc.vpc.id }}"
+        name: "{{ vpc_gateway_name }}"
+      register: created_vgw
+
+    - name: Create an association between VGW and connection
+      community.aws.directconnect_virtual_interface:
+        region: "{{ aws_region }}"
+        state: present
+        name: "{{ directconnect_vif_name }}"
+        public: false
+        connection_id: "{{ connection_response.connection.connection_id }}"
+        vlan: "{{ connection_response.connection.vlan }}"
+        virtual_gateway_id: "{{ created_vgw.vgw.id }}"
+        customer_address: "{{ metal_peering_ip }}/30"
+        amazon_address: "{{ aws_peering_ip }}/30"
+        bgp_asn: "{{ metal_side_asn }}"
+        authentication_key: "{{ bgp_md5_password }}"
+      register: created_vif
+      until: created_vif.amazon_side_asn is defined
+      retries: 10
+      delay: 60
+
+    - name: look up the main route table for our VPC
+      amazon.aws.ec2_vpc_route_table_info:
+        region: "{{ aws_region }}"
+        filters:
+          association.main: true
+          vpc-id: "{{ created_vpc.vpc.id }}"
+      register: route_tables
+
+    - name: Enable VGW route propagation
+      amazon.aws.ec2_vpc_route_table:
+        region: "{{ aws_region }}"
+        lookup: id
+        route_table_id: "{{ route_tables.route_tables[0].route_table_id }}"
+        vpc_id: "{{ created_vpc.vpc.id }}"
+        propagating_vgw_ids:
+          - "{{ created_vgw.vgw.id }}"
+
+    - name: Look up the project we created earlier
+      equinix.cloud.metal_project:
+        name: "{{ project_name }}"
+      register: project
+
+    - name: Look up the VRF we created earlier
+      equinix.cloud.metal_vrf:
+        name: "{{ vrf_name }}"
+        metro: "{{ metro }}"
+        local_asn: "{{ metal_side_asn }}"
+        ip_ranges:
+          - "{{ vrf_peering_ip_range }}"
+          - "{{ vrf_gateway_ip_range }}"
+        project_id: "{{ project.id }}"
+      register: vrf
+
+    - name: look up the Metal-billed VRF interconnection we created earlier
+      equinix.cloud.metal_connection:
+        project_id: "{{ project.id }}"
+        metro: "{{ metro }}"
+        name: "{{ interconnection_name }}"
+        type: "shared"
+        speed: "50Mbps"
+        service_token_type: a_side
+        redundancy: primary
+        vrfs:
+          - "{{ vrf.id }}"
+      register: connection
+
+    - name: Configure BGP for interconnection virtual circuit
+      equinix.cloud.metal_virtual_circuit:
+        id: "{{ connection.ports[0].virtual_circuits[0].id }}"
+        peer_asn: "{{ created_vif.amazon_side_asn }}"
+        customer_ip: "{{ aws_peering_ip }}"
+        metal_ip: "{{ metal_peering_ip }}"
+        subnet: "{{ vrf_vc_peering_ip_range }}"
+        md5: "{{ bgp_md5_password }}"
+        # The metal_virtual_circuit module requires this parameter
+        # in order to know that the circuit is a VRF circuit and
+        # not a VLAN circuit
+        vrf: "{{ vrf.id }}"
+
+    - name: print DNS name for VPC endpoint
+      debug:
+        msg: "{{ created_vpc_endpoint.result.dns_entries[0].dns_name | replace('*.', '') }}"
diff --git a/examples/layer2/pre_fabric.yml b/examples/layer2/pre_fabric.yml
@@ -0,0 +1,102 @@
+---
+- name: Equinix Metal Example Playbook
+  hosts: localhost
+  gather_facts: no
+  tasks:
+    - name: Include the required variables
+      include_vars: "vars/vars.yml"
+
+    # Equinix resources
+    - name: Create a project
+      equinix.cloud.metal_project:
+        name: "{{ project_name }}"
+      register: project
+
+    - name: create a vlan
+      equinix.cloud.metal_vlan:
+        project_id: "{{ project.id }}"
+        metro: "{{ metro }}"
+        vxlan: "1234"
+      register: vlan
+
+    - name: create a VRF
+      equinix.cloud.metal_vrf:
+        name: "{{ vrf_name }}"
+        metro: "{{ metro }}"
+        local_asn: "{{ metal_side_asn }}"
+        ip_ranges:
+          - "{{ vrf_peering_ip_range }}"
+          - "{{ vrf_gateway_ip_range }}"
+        project_id: "{{ project.id }}"
+      register: vrf
+
+    - name: create a VRF IP reservation
+      equinix.cloud.metal_reserved_ip_block:
+        project_id: "{{ project.id }}"
+        vrf_id: "{{ vrf.id }}"
+        type: "vrf"
+        metro: "{{ metro }}"
+        network: "{{ vrf_gateway_ip_range | split('/') | first }}"
+        cidr: "{{ vrf_gateway_ip_range | split('/') | last }}"
+      register: vrf_ip_reservation
+
+    - name: create a VRF Metal Gateway
+      equinix.cloud.metal_gateway:
+        project_id: "{{ project.id }}"
+        ip_reservation_id: "{{ vrf_ip_reservation.id }}"
+        virtual_network_id: "{{ vlan.id }}"
+      register: gateway
+
+    # Create a device
+    - name: Create a device
+      equinix.cloud.metal_device:
+        project_id: "{{ project.id }}"
+        metro: "{{ metro }}"
+        hostname: "{{ device_hostname }}"
+        operating_system: "{{ operating_system }}"
+        plan: "{{ plan }}"
+        state: present
+        userdata: |2
+          #!/bin/bash
+
+          cat <<EOF >> /etc/network/interfaces
+          auto bond0.{{vlan.vxlan}}
+          iface bond0.{{vlan.vxlan}} inet static
+              address {{ vrf_gateway_ip_range | ansible.utils.nthhost(2) }}
+              netmask {{ vrf_ip_reservation.netmask }}
+              post-up route add -net {{ vrf_gateway_ip_range }} gw {{ vrf_ip_reservation.gateway }}
+              post-up route add -net {{ aws_network_cidr }} gw {{ vrf_ip_reservation.gateway }}
+          EOF
+
+          systemctl restart networking
+      register: device
+
+    - name: capture port ids for device
+      set_fact:
+        bond_port_id: "{{ device.network_ports | selectattr('name', 'match', 'bond0') | map(attribute='id') | first }}"
+        eth1_port_id: "{{ device.network_ports | selectattr('name', 'match', 'eth1') | map(attribute='id') | first }}"
+
+    - name: convert bond port to hybrid bonded mode
+      equinix.cloud.metal_port:
+        id: "{{ bond_port_id }}"
+        bonded: true
+        layer2: false
+        vlan_ids:
+          - "{{ vlan.id }}"
+
+    - name: create a Metal-billed VRF interconnection
+      equinix.cloud.metal_connection:
+        project_id: "{{ project.id }}"
+        metro: "{{ metro }}"
+        name: "{{ interconnection_name }}"
+        type: "shared"
+        speed: "50Mbps"
+        service_token_type: a_side
+        redundancy: primary
+        vrfs:
+          - "{{ vrf.id }}"
+      register: connection
+
+    - name: print service token to redeem in Fabric portal
+      debug:
+        var: connection.service_tokens[0].id