Azure Kubernetes Metrics Adapter

An implementation of the Kubernetes Custom Metrics API and External Metrics API for Azure Services.

This adapter enables you to scale your application deployment pods running on AKS using the Horizontal Pod Autoscaler (HPA) with External Metrics from Azure Resources (such as Service Bus Queues) and Custom Metrics stored in Application Insights.

Try it out:

• External metric scaling video
• Custom metric scaling video
• Deploy the adapter
• Azure Service Bus queue walkthrough
• Request Per Second (RPS) walkthrough

This was build using the Custom Metric Adapter Server Boilerplate project. Learn more about using an HPA to autoscale with external and custom metrics.

Project Status: Alpha

Walkthrough

Try out scaling with External Metrics using the a Azure Service Bus Queue in this walkthrough.

Try out scaling with Custom Metrics using Requests per Second and Application Insights in this walkthrough

Deploy

Requires some set up on your AKS Cluster and Metric Server deployed to your cluster.

kubectl apply -f https://raw.githubusercontent.com/Azure/azure-k8s-metrics-adapter/master/deploy/adapter.yaml

Deploy a metric configuration:

kubectl apply -f https://raw.githubusercontent.com/Azure/azure-k8s-metrics-adapter/master/samples/resources/externalmetric-example.yaml

There is also a Helm chart available for deployment for those using Helm in their cluster.

Deploy a Horizontal Pod Auto Scaler (HPA) to scale of your external metric of choice:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
 name: consumer-scaler
spec:
 scaleTargetRef:
   apiVersion: extensions/v1beta1
   kind: Deployment
   name: consumer
 minReplicas: 1
 maxReplicas: 10
 metrics:
  - type: External
    external:
      metricName: queuemessages
      targetValue: 30

And your that's it to enable auto scaling on External Metric. Checkout the samples for more examples.

Verifying the deployment

You can also can also query the api to available metrics:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

To Query for a specific custom metric (not currently supported):

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test/pods/*/custom-metric" | jq .

To query for a specific external metric:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/test/queuemessages?labelSelector=resourceProviderNamespace=Microsoft.Servicebus,resourceType=namespaces,aggregation=Total,filter=EntityName_eq_externalq,resourceGroup=sb-external-example,resourceName=sb-external-ns,metricName=Messages" | jq .

External Metrics

Requires k8s 1.10+

See a full list of hundreds of available azure external metrics that can be used.

Common external metrics to use for autoscaling are:

• Azure ServiceBus Queue - Message Count - example

Custom Metrics

Custom metrics are currently retrieved from Application Insights. View a list of basic metrics that come out of the box and see sample values at the AI api explorer.

Common Custom Metrics are:

• Requests per Second (RPS) - example

Azure Setup

Security

Authenticating with Azure Monitor can be achieved via a variety of authentication mechanisms. (full list)

We recommend to use one of the following options:

• Azure Managed Service Identity (MSI)
• Azure AD Pod Identity (aad-pod-identity)
• Azure AD Application ID and Secret
• Azure AD Application ID and X.509 Certificate

The Azure AD entity needs to have Monitoring Reader permission on the resource group that will be queried. More information can be found here.

Using Azure Managed Service Identity (MSI)

Enable Managed Service Identity on each of your AKS vms:

There is a known issue when upgrading a AKS cluster with MSI enabled. After the AKS upgrade you will lose your MSI setting and need to re-enable it. An alternative may be to use aad-pod-identity

export RG=<aks resource group> 
export CLUSTER=<aks cluster name> 

NODE_RG="$(az aks show -n $CLUSTER -g $RG | jq -r .nodeResourceGroup)"
az vm list -g $NODE_RG
VMS="$(az vm list -g $NODE_RG | jq -r '.[] | select(.tags.creationSource | . and contains("aks")) | .name')"

while read -r vm; do
    echo "updating vm $vm..."
    msi="$(az vm identity assign -g $NODE_RG -n $vm | jq -r .systemAssignedIdentity)"
done <<< "$VMS"

Give access to the resource the MSI needs to access for each vm:

export RG=<aks resource group> 
export CLUSTER=<aks cluster name> 
export ACCESS_RG=<resource group with metrics>

NODE_RG="$(az aks show -n $CLUSTER -g $RG | jq -r .nodeResourceGroup)"
az vm list -g $NODE_RG
VMS="$(az vm list -g $NODE_RG | jq -r '.[] | select(.tags.creationSource | . and contains("aks")) | .name')"

while read -r vm; do
    echo "getting vm identity $vm..."
    msi="$(az vm identity show -g $NODE_RG -n $vm | jq -r .principalId)"

    echo "adding access with msi $msi..."
    az role assignment create --role "Monitoring Reader" --assignee-object-id $msi --resource-group $ACCESS_RG
done <<< "$VMS"

Using Azure AD Pod Identity

aad-pod-identity is currently in beta and allows to bind a user managed identity or a service principal to a pod. That means that instead of using the same managed identity for all the pod running on a node like explained above, you are able to get a specific identity with specific RBAC for a specific pod.

Using this project requires to deploy a bit of infrastructure first. You can do it following the Get started page of the project.

Once the aad-pod-identity infrastructure is running, you need to create an Azure identity scoped to the resource group you are monitoring:

az identity create -g {ResourceGroup1} -n custom-metrics-identity

Assign Monitoring Reader to it

az role assignment create --role "Monitoring Reader" --assignee <principalId> --scope /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}

Note: you need to assign this role to all resources groups that you want the identity to be able to read Azure Monitor data.

As documented here aad-pod-identity uses the service principal of your Kubernetes cluster to access the Azure resources. You need to give this service principal the rights to use the managed identity created before:

az role assignment create --role "Managed Identity Operator" --assignee <servicePrincipalId> --scope /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-metrics-identity

Install the Azure Identity to your Kubernetes cluster:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: custom-metrics-identity
spec:
  type: 0
  ResourceID: /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-metrics-identity
  ClientID: <clientid>

Install the Azure Identity Binding on your Kubernetes cluster:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: custom-metrics-identity-binding
spec:
  AzureIdentity: custom-metrics-identity
  Selector: custom-metrics-identity

Note: pay attention to the name of the selector above. You will need to use it to bind the identity to your pod.

If you use the Helm Chart to deploy the custom metrics adapter to your Kubernetes cluster, you can configure Azure AD Pod Identity directly in the config values:

azureAuthentication:
  method: aadPodIdentity
  # if you use aadPodIdentity authentication
  azureIdentityName: "custom-metrics-identity"
  azureIdentityBindingName: "custom-metrics-identity-binding"
  # The full Azure resource id of the managed identity (/subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{IdentityName})
  azureIdentityResourceId: ""
  # The Client Id of the managed identity
  azureIdentityClientId: ""

Switch method to aadPodIdentity a give the value for the Azure Identity resource id and client id, for example:

helm install ./charts/azure-k8s-metrics-adapter --set azureAuthentication.method="aadPodIdentity" \
  --set azureAuthentication.azureIdentityResourceId="/subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{IdentityName}" \
  --set azureAuthentication.azureIdentityClientId="{ClientId}" \
  --name "custom-metrics-adapter"

Using Azure AD Application ID and Secret

See how to create an example deployment.

Create a service principal scoped to the resource group the resource you monitoring and assign Monitoring Reader to it:

az ad sp create-for-rbac -n "adapter-sp" --role "Monitoring Reader" --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CLIENT_SECRET: Specifies the app secret to use.

Deploy the environment variables via secret:

 kubectl create secret generic adapter-service-principal -n custom-metrics \
  --from-literal=azure-tenant-id=<tenantid> \
  --from-literal=azure-client-id=<clientid>  \
  --from-literal=azure-client-secret=<secret>

Azure AD Application ID and X.509 Certificate

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CERTIFICATE_PATH: Specifies the certificate Path to use.
• AZURE_CERTIFICATE_PASSWORD: Specifies the certificate password to use.

Subscription Information

The use the adapter your Azure Subscription must be provided. There are a few ways to provide this information:

• Azure Instance Metadata - If you are running the adapter on a VM in Azure (for instance in an AKS cluster) there is nothing you need to do. The Subscription Id will be automatically picked up from the Azure Instance Metadata endpoint
• Environment Variable - If you are outside of Azure or want full control of the subscription that is used you can set the Environment variable SUBSCRIPTION_ID on the adapter deployment. This takes precedence over the Azure Instance Metadata.
• On each HPA - you can work with multiple subscriptions by supplying the metric selector subscriptionID on each HPA. This overrides Environment variables and Azure Instance Metadata settings.

FAQ

• Can I scale with Azure Storage queues?
  • Not currently. The Azure Storage Queue only reports it's capacity metrics daily. Follow this issue for updates.

Contributing

See Contributing for more information.

Issues

Report any issues in the Github issues.

Roadmap

See the Projects tab for current roadmap.

Reporting Security Issues

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.