This project aims to provide a Terraform module to make it easy to setup Azure permissions needed to install and manage ARO. The problem we see in the field, is that there are overlapping identities that need specific permission sets and this often times results in incorrect permissions and makes for a confusing experience.
WARN: It should be noted that certain product changes will force this to change. This is a community-supported project and you should consult your appropriate product documentation prior to using this in your environment to ensure it is appropriate for your needs.
This section identifies the individual identities that need ARO permissions. These identities will be used in verbiage when describing what permissions are needed.
NOTE: the Install Flags column helps to associate these service principals to actual ARO installation flags using the CLI.
| Identity | Type | Install Flags | Description |
|---|---|---|---|
| Cluster Service Principal | Service Principal | --client-id and --client-secret |
Runs all operations in the cluster and that interacts with the Azure API as part of operations within the cluster. Cluster Autoscaler and the Service Controller for Azure are two key components which leverage these credentials. |
| Installer | User or Service Principal | N/A | Whomever the installation process is run as. This may be a user account or a service principal. This is who is logged in using the az login command. |
| Resource Provider Service Principal | Service Principal | N/A | Azure Resource Provider that represents ARO. This is automatically created in the account when first running the setup step az provider register -n Microsoft.RedHatOpenShift --wait. This service principal can be found by running az ad sp list --filter "displayname eq 'Azure Red Hat OpenShift RP'". |
This section defines the objects which need individual permissions.
NOTE: the Install Flags column helps to associate these objects to actual ARO installation flags using the CLI.
| Object Type | Install Flags | Description |
|---|---|---|
| Subscription | --subscription |
The highest level a permission will be applied. Inherits down to all objects within that subscription. This is not a mandatory flag and the subscription may be set based on how a user has logged in with az login. |
| ARO Resource Group | --resource-group |
Resource group in the above subscription where the actual ARO object is created. |
| Cluster Resource Group | --cluster-resource-group |
Resource group in the above subscription where the underlying ARO objects (e.g. VMs, load balancers) are created. This is created automatically as part of provisioning. |
| Network Resource Group | --vnet-resource-group |
Resource group in the above subscription where network resources (e.g. VNET, NSG) exist. Some organizations will use the Cluster Resource Group for this purpose as well and do not need a dedicated Network Resource Group. |
| VNET | --vnet |
VNET where the ARO cluster will be provisioned. |
| Network Security Group | N/A | Only required for BYO-NSG scenarios. Network security group, applied to the subnets. This is is pre-applied by the user to the subnets prior to installation. |
| Disk Encryption Set | --disk-encryption-set |
The disk encryption set used to encrypt master and worker node disks. |
This section identifies what permissions are needed by each individual identity.
NOTE: row numbers are used to indicate in the code where permissions are aligned.
In many cases, such as separation of duties and where network teams must provide infrastructure to consume, a reduced permission set lower than Network Contributor is required. This is possible, however it should be noted that product documentation indicates higher permissions and the product will be developed against that assumption unless otherwise noted.
The following permission, in place of Network Contributor, have been successful (including links to the code which
validates the permissions). The VNET this applies to equates to the value of the --vnet flag in the az aro create command:
Needed always:
- Microsoft.Network/virtualNetworks/join/action
- Microsoft.Network/virtualNetworks/read
- Microsoft.Network/virtualNetworks/write
- Microsoft.Network/virtualNetworks/subnets/join/action
- Microsoft.Network/virtualNetworks/subnets/read
- Microsoft.Network/virtualNetworks/subnets/write
- Microsoft.Network/networkSecurityGroups/join/action
Needed when provided VNET subnets have route table(s) attached:
- Microsoft.Network/routeTables/join/action
- Microsoft.Network/routeTables/read
- Microsoft.Network/routeTables/write
Needed when provided VNET subnets have NAT gateway(s) attached:
- Microsoft.Network/natGateways/join/action
- Microsoft.Network/natGateways/read
- Microsoft.Network/natGateways/write
In addition to minimizing network permissions, the installer role may need minimal permissions as well. These permissions are as follows:
- Microsoft.RedHatOpenShift/openShiftClusters/read
- Microsoft.RedHatOpenShift/openShiftClusters/write
- Microsoft.RedHatOpenShift/openShiftClusters/delete
- Microsoft.RedHatOpenShift/openShiftClusters/listCredentials/action
- Microsoft.RedHatOpenShift/openShiftClusters/listAdminCredentials/action
In addition to the above, the following other permissions may be needed by specific identities:
Prior to running this module, the following must be satisfied:
-
Must be logged in as an administrator user using the
az logincommand. Because assigning permissions is an administrative task, it is assumed whomever is running this module is an administrator. Alternative to full tenant administrator permissions, a user that has the User Access Administrator role should be able to complete this task. -
Must have the
azCLI installed and configured locally. There are some external commands ran in this module which makes this necessary. It is not ideal but it works for now. -
Must have the
jqCLI installed locally. There are some external commands ran in this module which makes this necessary. It is not ideal but it works for now. -
Must have a VNET architecture pre-deployed and used as an input.
Examples of how to use this module are located in the examples directory. The examples thus far are:
- When ARO will be installed with the Azure
azCLI - examples/cli - When ARO will be installed with an automation tool like Terraform (using the Microsoft API) - examples/api