Brigade Metrics: Monitoring for Brigade 2

Brigade Metrics adds monitoring capabilities to a Brigade 2 installation. It utilizes Brigade APIs to export time series metrics to Prometheus and makes visualizations of those metrics available through a Grafana dashboard.

Installation

Prerequisites:

• A Kubernetes cluster:

  • For which you have the admin cluster role
  • That is already running Brigade 2
  • Capable of provisioning a public IP address for a service of type LoadBalancer. (This means you won't have much luck running the gateway locally in the likes of kind or minikube unless you're able and willing to mess with port forwarding settings on your router, which we won't be covering here.)

• kubectl, helm (commands below require Helm 3.7.0+), and brig (the Brigade 2 CLI)

1. Create a Service Account

Note: To proceed beyond this point, you'll need to be logged into Brigade 2 as the "root" user (not recommended) or (preferably) as a user with the ADMIN role. Further discussion of this is beyond the scope of this documentation. Please refer to Brigade's own documentation.

Using Brigade 2's brig CLI, create a service account:

$ brig service-account create \
    --id brigade-metrics \
    --description brigade-metrics

Make note of the token returned. This value will be used in another step. It is your only opportunity to access this value, as Brigade does not save it.

Authorize this service account with read-only access to Brigade:

$ brig role grant READER \
    --service-account brigade-metrics

2. Installing Brigade Metrics

For now, we're using the GitHub Container Registry (which is an OCI registry) to host our Helm chart. Helm 3.7 has experimental support for OCI registries. In the event that the Helm 3.7 dependency proves troublesome for users, or in the event that this experimental feature goes away, or isn't working like we'd hope, we will revisit this choice before going GA.

First, be sure you are using Helm 3.7.0 or greater and enable experimental OCI support:

$ export HELM_EXPERIMENTAL_OCI=1

Use the following command to extract the full set of configuration options from the chart. Here we're storing a copy at ~/brigade-metrics-values.yaml:

$ helm inspect values oci://ghcr.io/brigadecore/brigade-metrics \
    --version v0.4.1 > ~/brigade-metrics-values.yaml

Edit the configuration (~/brigade-metrics-values.yaml in this example). At minimum, you will need to make the following changes:

• Set the value of exporter.brigade.apiAddress to the address of your Brigade 2 API server. This should utilize the internal DNS hostname by which that API server is reachable within your Kubernetes cluster. This value is defaulted to https://brigade-apiserver.brigade.svc.cluster.local, but may need to be updated if you installed Brigade 2 in a different namespace.

• Set the value of exporter.brigade.apiToken to the service account token that was generated earlier.

• grafana.host: Set this to the host name where you'd like the dashboard (Grafana) to be accessible.

• Specify a username and password for the metrics dashboard by setting values for grafana.auth.username and grafana.auth.password.

• grafana.service.type: If you plan to enable ingress (advanced), you can leave this as its default -- ClusterIP. If you do not plan to enable ingress, you probably will want to change this value to LoadBalancer.

Install Brigade Metrics, referencing your edited configuration:

$ helm install brigade-metrics \
    oci://ghcr.io/brigadecore/brigade-metrics \
    --version v0.4.1 \
    --create-namespace \
    --namespace brigade-metrics \
    --values ~/brigade-metrics-values.yaml \
    --wait \
    --timeout 300s

3. (RECOMMENDED) Create a DNS Entry

If you overrode defaults and set grafana.service.type to LoadBalancer, use this command to find the gateway's public IP address:

$ kubectl get svc brigade-metrics-grafana \
    --namespace brigade-metrics \
    --output jsonpath='{.status.loadBalancer.ingress[0].ip}'

If you overrode defaults and enabled support for an ingress controller, you probably know what you're doing well enough to track down the correct IP without our help. 😉

With this public IP in hand, edit your name servers and add an A record pointing your domain to the public IP.

4. Accessing the Dashboard

If you overrode defaults and set grafana.service.type to LoadBalancer, then the dashboard should be accessible over HTTPS at the public IP address or DNS hostname.

If you kept the default setting of ClusterIP for grafana.service.type, then use port forwarding to expose the dashboard on your local network interface:

$ kubectl port-forward \
    service/brigade-metrics-grafana \
    --namespace brigade-metrics \
    8443:443

In this case, the dashboard should be accessible at https://localhost:8443. Expect to receive a cert warning.

Log in using the username and password you selected in the previous section.

Contributing

The Brigade project accepts contributions via GitHub pull requests. The Contributing document outlines the process to help get your contribution accepted.

Support & Feedback

We have a slack channel! Kubernetes/#brigade Feel free to join for any support questions or feedback, we are happy to help. To report an issue or to request a feature open an issue here

Code of Conduct

Participation in the Brigade project is governed by the CNCF Code of Conduct.