feat: Support OTLP and Prometheus APM metrics

[jsirianni]

Description of Changes

Added a metrics config section, allowing users to enable OTLP or Prometheus metrics. The Prometheus metrics include the require annotations for being scraped by the Kubernetes Prometheus Node source.

When using the Prometheus metrics, users can configure the BindPlane node agent to autodetect and scrape the BindPlane pods for metrics.

When using OTLP, users can configure BindPlane to push metrics to any OTLP receiver, such as the Node or Gateway agents.

Testing

Deploy minikube

minikube start
minikube addons enable ingress 
kubectl apply -f test/helper/postgres/postgres.yaml 

Create a license secret from $BINDPLANE_LICENSE

kubectl create secret generic bindplane \
  --from-literal=license=$BINDPLANE_LICENSE

Create a values.yaml file in root of the helm repo.

config:
  username: bpuser
  password: bppass
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
  server_url: http://bindplane.local:80
  licenseUseSecret: true

ingress:
  enable: true
  host: bindplane.local
  class: nginx

backend:
  type: postgres
  postgres:
    host: postgres.postgres.svc.cluster.local
    database: bindplane
    username: postgres
    password: password
    maxConnections: 20
    
eventbus:
  type: nats

replicas: 2

resources:
  requests:
    memory: 100Mi
    cpu: 100m
  limits:
    memory: 100Mi

nats:
  resources:
    requests:
      memory: 100Mi
      cpu: 100m
    limits:
      memory: 100Mi

metrics:
  #type: "prometheus"
  type: "otlp"
  otlp:
    endpoint: bindplane-node-agent.bindplane-agent.svc.cluster.local:4317
    insecure: true

Checkout this branch and deploy BindPlane

helm upgrade \
  --install bindplane-ha \
  ./charts/bindplane \
  --values values.yaml

Wait for the pods to come up:

kubectl get pod

Make sure you have the following /etc/hosts entry

127.0.0.1 bindplane.local

Run minikube tunnel in a new terminal, and then go to http://bindplane.local.

Sign in with bpuser / bppass.

1. Create a Kubernetes Node configuration
2. Attach OTLP source
3. Attach Prometheus source
4. Attach Dev Null destination
5. Add "Filter by Metric Name" processor to Prometheus, and include regex bindplane_.*
6. Install the Kubernetes Node agent by downloading the YAML and applying it to minikube

kubectl apply -f ~/Downloads/bindplane-agent.yaml

Once the agent is up, roll the config out and verify that OTLP metrics show up in Live Preview. You will need to wait up to 60 seconds.

Edit values.yaml change metrics.type: otlp to metrics.type: prometheus. Re-deploy.

Checkout this branch and deploy BindPlane

```bash
helm upgrade \
  --install bindplane-ha \
  ./charts/bindplane \
  --values values.yaml

Once the pods are up, wait 60 seconds and check Prometheus live preview. You should be getting a handful of metrics. Notice that the metrics are prefixed with bindplane_, this is by design.

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• CI passes
• Changes to ports, services, or other networking have been tested with istio