feat(prometheus): Always enable Prometheus

[jsirianni]

Description of Changes

• Removed prometheus.enable option
• Removed prometheus.enableSidecar option

The intention is to always configure BindPlane to use Prometheus.

Migration path

Configuration

The following options can be removed from a user's values configuration:

• prometheus.enable
• prometheus.enableSidecar

StatefulSet

Single instance StatefulSet deployments can be upgraded without issue regardless of their Prometheus status.

Existing StatefulSets without Prometheus will be upgraded without issue. The new pod will contain a sidecar (additional container within the bindplane pod).

Existing StatefulSets with prometheus.enableSidecar set will be upgraded without issue and continue to use their sidecar.

Deployment (High Availability)

When running as a Deployment, the chart will not deploy Prometheus as a sidecar, because this would result in one Prometheus container per pod in the deployment. It is expected that Deployment users will configure a remote Prometheus server on their own, separate from this chart.

Existing Deployments without Prometheus will start up fine but will not function until Prometheus is correctly configured.

Existing Deployments that contain Prometheus will continue to use Prometheus without issue.

Testing

Single Instance

Prometheus Previously Enabled

Single instance testing is easy because the output from helm is the same as before if Prometheus was already enabled.

1. Checkout always-on-prometheus
2. Create values.yaml with the Prometheus options

image:
  tag: 1.43.0
config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
prometheus:
  enable: true
  enableSideCar: true
dev:
  collector:
    create: true

3. Write the helm output to a file:

helm template --values ./values.yaml charts/bindplane > old.yaml

4. Checkout this branch
5. Remove the prometheus options from the values file
6. Write the helm output to a file:

helm template --values ./values.yaml charts/bindplane > new.yaml

Diff the results

diff new.yaml old.yaml

Or checksum them:

$ sha256sum new.yaml old.yaml

33cf5913fcaff8270571a29d250f7379f67f84c69b1326350a4e94505a8037a8  new.yaml
33cf5913fcaff8270571a29d250f7379f67f84c69b1326350a4e94505a8037a8  old.yaml

Prometheus Not Enabled

If Prometheus was not enabled previously, you can use the following values file to deploy main and then upgrade to this branch:

image:
  tag: 1.43.0
config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true

Deploy the epic/always-on-prometheus branch to minikube (the branch we are merging into)

git checkout epic/always-on-prometheus
helm template --values ./values.yaml charts/bindplane | kubectl apply -f -

Note that bindplane has a single container (does not have a Prometheus sidecar container).

$ kubectl get pod

NAME                                                      READY   STATUS    RESTARTS   AGE
bindplane-test-agent-5bc749fb67-9vwr6                     1/1     Running   0          21m
release-name-bindplane-0                                  1/1     Running   0          21m
release-name-bindplane-transform-agent-59d7657d8f-gl5b2   1/1     Running   0          21m

Upgrading can be done by deleting the statefulset and re-deploying. The delete is necessary due to the addition of a sidecar container. See this previous PR for details on why this is safe.

git checkout single-instance-prometheus-always-on
kubectl delete sts release-name-bindplane   
template --values ./values.yaml charts/bindplane | kubectl apply -f -

Once deployed, the bindplane pod will contain two containers.

NAME                                                      READY   STATUS    RESTARTS   AGE
bindplane-test-agent-5bc749fb67-9vwr6                     1/1     Running   0          25m
release-name-bindplane-0                                  2/2     Running   0          78s
release-name-bindplane-transform-agent-59d7657d8f-gl5b2   1/1     Running   0          25m

High Availability

WIP

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• CI passes