[DRAFT][Terraform - AKS] For non production environments, use least-waste expander to maximize resource usage

[yelghali]

You can configure more granular details of the cluster autoscaler by changing the default values in the cluster-wide autoscaler profile. For example, a scale down event happens after nodes are under-utilized after 10 minutes. If you had workloads that ran every 15 minutes, you may want to change the autoscaler profile to scale down under utilized nodes after 15 or 20 minutes. When you enable the cluster autoscaler, a default profile is used unless you specify different settings.

c.f https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile

One of the auto-scaling profile settings, is the use of cluster expanders

When Cluster Autoscaler identifies that it needs to scale up a cluster due to unschedulable pods, it increases the number of nodes in some node group. When there is one node group, this strategy is trivial. When there is more than one node group, it has to decide which to expand.

Currently Cluster Autoscaler has 5 expanders:

random - this is the default expander, and should be used when you don't have a particular need for the node groups to scale differently.

most-pods - selects the node group that would be able to schedule the most pods when scaling up. This is useful when you are using nodeSelector to make sure certain pods land on certain nodes. Note that this won't cause the autoscaler to select bigger nodes vs. smaller, as it can add multiple smaller nodes at once.

least-waste - selects the node group that will have the least idle CPU (if tied, unused memory) after scale-up. This is useful when you have different classes of nodes, for example, high CPU or high memory nodes, and only want to expand those when there are pending pods that need a lot of those resources.

price - select the node group that will cost the least and, at the same time, whose machines would match the cluster size. This expander is described in more details HERE. Currently it works only for GCE, GKE and Equinix Metal (patches welcome.)

priority - selects the node group that has the highest priority assigned by the user. It's configuration is described in more details here

code scan check
If sku == free (meant for non production clusters):

auto_scaler_profile {
expander = 'least-waste'
}

From https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#expander