Allow setting options per worker group

[himanshu-kun]

/kind discussion
/priority 3

What would you like to be added:

Autoscaler now supports configuring certain flags per worker group. Some example flags are:

• ScaleDownUtilizationThreshold
• ScaleDownGpuUtilizationThreshold
• ScaleDownUnneededTime
• ScaleDownUnreadyTime
• MaxNodeProvisionTime

This helps in more fine-grained control over a cluster and how autoscaler works for that cluster.
kubernetes#3583 -> upstream issue which introduced this
kubernetes#3789 -> PR with first implementations

With time, more and more options are being made configurable on worker group level , which makes it essential for gardener to also allow configuring them by our end-users.

Some issue which can be solved by this are:
#227 (by reducing scale-down-unneeded-time for the required nodegroup also)

The change should require changes in shoot API also , some related discussions are already ongoing gardener/gardener#8142

---

Changes/PRs that need to be filed to close this story

• Autoscaler: Read annotations from MCD and add them to NodeGroupAutoscalingOptions > Add options to NodeGroupAutoscalingOptions from machineDeployment annotations #257
• Gardener: Update the shoot API to expose these values to be set per node group > Added autoscaler options to worker groups gardener#9245
• g/Extensions: Deploy MCDs with these node group specific values as annotations
  • AWS Vendor gardener/[email protected], Upgrade k8s.io/* dependencies and controller-runtime gardener-extension-provider-aws#901
  • GCP Add cluster-autoscaler options to machine deployment gardener-extension-provider-gcp#733
  • Azure Add cluster-autoscaler options to machine deployment gardener-extension-provider-azure#831
  • Openstack Add cluster-autoscaler options to machine deployment gardener-extension-provider-openstack#752
  • Alicloud Add cluster-autoscaler options to machine deployment gardener-extension-provider-alicloud#715
  • Equinix Metal > PR not raised as vendored g/g version is 1.81. Required g/g version is at lease 1.91 Cluster Autoscaler settings per worker group not supported  gardener-extension-provider-equinix-metal#308

Why is this needed:
For better configurability of gardener autoscaler

[aaronfern]

A few options were debated, and we finally decided to go with an approach that is similar to the one used by amazon ASG

The following is a proposal of the solution

g/g changes:

• Add spec.provider.workers.clusterAutoscaler to the shoot API that will expose these 5 fields to be set per node group

  example

  spec:
    provider:
      workers:
      - name: test-name
        clusterAutoscaler:
          scaleDownUtilizationThreshold: 0.5
          scaleDownGpuUtilizationThreshold: 0.5 
          scaleDownUnneededTime: 1m
          scaleDownUnreadyTime: 1m
          maxNodeProvisionTime: 1m
  

g/extension changes:

• Update code which creates and deploys machineDeployments to add these node group specific values as annotations on the machineDeployment. If node group specific values are not provided then shoot defaults are to be added as annotations.

autoscaler changes:

• Update the machineDeployment.GetOptions() method in mcm_cloud_provider.go to read the annotations on machineDeployments and add these node group specific options to the NodeGroupAutoscalingOptions struct

The other approached we discussed were the following:

• Overload the --nodes flag to add these node group specific values. This then involves us updating the validation for this flag, and we would then be at a disadvantage should we go down a path of using node-group-auto-discovery
• Adding a new flag (for eg --node-group-options) that contains node group specific options , but this option has similar problems as the above, plus its making change to the core autoscaler which we should avoid.

/cc @timuthy for comments on the g/g changes

[rishabh-11]

/close as all PRs are merged