feat(prometheus): Deploy Prometheus StatefulSet by default

[jsirianni]

Description of Changes

BindPlane v1.46.0 will require Prometheus because measurement support for bolt store and postgres have been removed.

This PR implements a Prometheus StatefulSet that is deployed by default. Support for Prometheus side car and self managed Prometheus is retained. This allows the chart to be compatible with all existing deployments. If sidecar or self managed prometheus are in use, the new statefulset will not be deployed.

Changes

• Removed prometheus.enable bool option
• Added prometheus.remote bool option
  • When true, the chart will not deploy Prometheus. This is for self managed Prometheus.
• Change default value for prometheus.host to ""
  • This field should only be set when connecting to a self managed Prometheus
  • Chart will use Prometheus clusterIP service name by default
  • Chart will use localhost when Prometheus side car is enabled
  • Chart will use the prometheus.host when prometheus.remote is enabled
• Moved prometheus.sidecar.resources to prometheus.resources
  • This block will be shared between the StatefulSet Prometheus and the SideCar, whichever is in use
• Moved prometheus.storageClass and prometheus.volumeSize to prometheus.storage block

When this change is released, the release will contain upgrade notes. The same upgrade notes will be added to the 1.46 bindplane release notes.

Testing

I have already performed significant testing. Feel free to run through a few scenario or two.

Test method

• Upgrade from main branch (if applicable)
• Ensure BindPlane and Prometheus storage is working by deleting the pods and checking to see if data exists on the new pods

Test matrix used to ensure all upgrade paths are possible. See values files used below. GKE tends to be more strict w/ volume mounts, so we test against both.

Test Case	Minikube	GKE
new deployments	pass	pass
main without Prometheus	pass	pass
main w/ sidecar	pass	pass
main w/ remote Prometheus, not managed by the chart	pass	pass

New deployment.

Brand new deployment.

We expect bindplane and prometheus to be deployed.

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true

Main without Prometheus configured

Previous deployment from main, without Prometheus explicitly enabled.

We expect the upgrade to this branch to deploy prometheus.

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true

Main w/ Sidecar

Previous deployment from main, with Prometheus sidecar explicitly configured.

prometheus:
  enable: true
  enableSideCar: true
config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true

When upgrading, the prometheus.enable field can be removed.

Main w/ remote Prometheus

Previous deployment from main, with a remote Prometheus configured. This is for Prometheus that is unmanaged by this chart.

In this case, Prometheus is deployed to my Linux system using the packages released w/ BindPlane.

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true
prometheus:
  enable: true
  host: "10.99.1.10"

Before upgrading, I update the values config to look like this:

prometheus.enable --> prometheus.remote

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
dev:
  collector:
    create: true
prometheus:
  remote: true
  host: "10.99.1.10"

Helm outputs the same config in both cases, you can compare like this. If I write the output to file and check the hash, they are identical.

f3aa3b375db7dee4cbc7d655a7b4cfa9d111464dbc9800da3466929fe2e4dde3  new.yaml
f3aa3b375db7dee4cbc7d655a7b4cfa9d111464dbc9800da3466929fe2e4dde3  old.yaml

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• CI passes