[bitnami/keycloak] Add HPA Behavior when scaling up and down

[ashktn]

Description of the change

This PR adds the ability to configure the behavior of the HorizontalPodAutoscaler of the Keycloak StatefulSet with new parameter block autoscaling.behavior
ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior

This allows us to set policies that control how the autoscaler scales up/down the pods e.g., the keycloak pods can be scaled by X units or percent of current pods during a periodSeconds window.
stabilizationWindowSeconds can be set to set the duration which past recommendations should be considered while scaling
selectPolicy - The priority of policies that the autoscaler will apply when scaling up/down. Can be Max, Min, Disabled

Benefits

Keycloak pods are stateful due to the embedded Infinispan cache that requires cache key rebalancing across all live nodes when a pod joins or leaves the cluster. (This process happens on pod startup and shutdown).

The cache keys have a set number of owners (pods) that can survive one node failure with its default config (https://www.keycloak.org/server/caching#_configuring_caches).

Adding the HPA behavior when scaling down to 1 pod every 300s allows the cache keys to be rebalanced across other pods before terminating, hence avoiding potential data loss if too many pods were to be terminated at the same time by the autoscaler's default behaviour.

Possible drawbacks

The default values set in values.yaml slows down the scaleDown of pods.

Applicable issues

• fixes [bitnami/keycloak] - Allow configuring HPA Behaviour for scaleUp / scaleDown #25651

Additional information

The default HPA scaleDown behaviour is now one pod every 300s.

If anybody wishes to go back to the previous behaviour, they can set the values as follows:

  behavior:
    scaleDown:
      policies: []

@javsalgar - I'm happy to set the default value to the above so that this release doesn't change the default behaviour. But i figured the default value I'm proposing are reasonable for an HA deployment of Keycloak.
These are also the default values set in the Codecentric Keycloak Helm Chart

Checklist

• Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
• Variables are documented in the values.yaml and added to the README.md using readme-generator-for-helm
• Title of the pull request follows this pattern [bitnami/<name_of_the_chart>] Descriptive title
• All commits signed off and in agreement of Developer Certificate of Origin (DCO)

Tests

pods start fine using default values

# HPA Enabled but no behavior set

# values.yaml
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPU: 70

---
# applied yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/component: keycloak
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 24.0.4
    helm.sh/chart: keycloak-21.1.3
  name: keycloak
  namespace: default
spec:
  behavior:
    scaleDown:
      policies:
      - periodSeconds: 300
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 300
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: keycloak

---

Autoscaling enabled - scaleDown policies set to empty to remove the default scaleDown policy

# values.yaml
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPU: 70
  behavior:
    scaleDown:
      policies: []

--- 
# applied yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/component: keycloak
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 24.0.4
    helm.sh/chart: keycloak-21.1.3
  name: keycloak
  namespace: default
spec:
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: keycloak

---

Autoscaling enabled - scaleUp policy added and some values changes from default

values.yaml

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPU: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 1337
      selectPolicy: Min
      policies:
        - type: Percent
          value: 30
          periodSeconds: 123
    scaleDown:
      stabilizationWindowSeconds: 60

--- 
# applied yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/component: keycloak
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 24.0.4
    helm.sh/chart: keycloak-21.1.3
  name: keycloak
  namespace: default
spec:
  behavior:
    scaleDown:
      policies:
      - periodSeconds: 300
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 60
    scaleUp:
      policies:
      - periodSeconds: 123
        type: Percent
        value: 30
      selectPolicy: Min
      stabilizationWindowSeconds: 1337
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: keycloak

[dgomezleon]

LGTM. Thanks for contributing

[ashktn]

thanks for the quick reviews!