Skip to content

Commit

Permalink
Add MySQL integration (#817)
Browse files Browse the repository at this point in the history

Signed-off-by: Pete Wall <[email protected]>
  • Loading branch information
petewall authored Nov 6, 2024
1 parent aa2ae9d commit 48a01f8
Show file tree
Hide file tree
Showing 74 changed files with 2,814 additions and 109 deletions.
1 change: 1 addition & 0 deletions .github/workflows/platform-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ jobs:
name: Platform Test
needs: list-tests
runs-on: ubuntu-latest
if: ${{ needs.list-tests.outputs.tests != '[]' }}
strategy:
matrix:
test: ${{ fromJson(needs.list-tests.outputs.tests) }}
Expand Down
3 changes: 1 addition & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,9 @@ Each component has its own guide for contributing:
<!-- [Main Helm chart](./charts/k8s-monitoring/README.md#Contributing) -->

- [Feature subcharts](./charts/k8s-monitoring/docs/Features.md#Contributing)
- [Integrations](./charts/feature-integrations/README.md#Contributing)
- [Destinations](./charts/k8s-monitoring/docs/destinations/README.md#Contributing)

<!-- [Integrations](./charts/feature-integrations/README.md#Contributing) -->

## Building

Lots of files in this repository are generated in order to reduce duplication and ensure consistency. To build these
Expand Down
47 changes: 47 additions & 0 deletions charts/feature-integrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,3 +105,50 @@ Actual integration testing in a live environment should be done in the main [k8s
| global.alloyModules.source | string | `"git"` | The source of the Alloy modules. The valid options are "configMap" or "git" |
| global.maxCacheSize | int | `100000` | Sets the max_cache_size for every prometheus.relabel component. ([docs](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.relabel/#arguments)) This should be at least 2x-5x your largest scrape target or samples appended rate. |
| global.scrapeInterval | string | `"60s"` | How frequently to scrape metrics. |

### Integration: MySQL

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| mysql | object | `{"instances":[]}` | Scrape metrics from MySQL |

## Contributing

To contribute integrations to this feature, there are a few files that need to be created or modified:

* `values.yaml` - The main feature chart's values file. Add a section for your integration. It must contain an
`instance` array and any settings that apply to every instance of the integration. For example:

```yaml
<slug>:
instances: []
globalSetting: value
```

* `integrations/<slug>-values.yaml` - The values that will be used for each instance. This must include `name` to
differentiate it from other instances, and any other settings that are specific to that instance. For example:

```yaml
name: ""
labelSelectors:
app.kubernetes.io/name: my-service
protocol: http
...
```

* `templates/_integration<=_<slug>.tpl` - The file that contains template functions that build the configuration to
discover, gather, process, and deliver the telemetry data. This file is required to implement the following template
functions:
* `integrations.<slug>.type.metrics` - Returns true if this integration scrapes metrics.
* `integrations.<slug>.type.logs` - Returns true if this integration gathers logs.
* `integrations.<slug>.module` - Returns the configuration that is included once if this integration is used. This
is typically the module definition.
* `integrations.<slug>.include.metrics` - Returns the configuration that is included for each instance of the
integration that scrapes metrics.
* `integrations.<slug>.include.logs` - Returns the configuration that is included for each instance of the
integration that gathers logs.
* `integrations.<slug>.exclude.logs` - Returns a rule that can be used by other Log-gathering features to ensure
that logs that are gathered from this integration are not collected twice. Typically the inverse of a rule in
the `integrations.<slug>.include.logs` function.
* `default-allow-lists/<slug>.yaml` - If the integration scrapes metrics, a common pattern is to provide a list of
metrics that should be allowed. This minimizes the amount of metrics delivered to a useful minimal set.
41 changes: 41 additions & 0 deletions charts/feature-integrations/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,44 @@ Actual integration testing in a live environment should be done in the main [k8s
<!-- markdownlint-enable no-bare-urls -->
{{ template "chart.requirementsSection" . }}
{{ template "chart.valuesSection" . }}

## Contributing

To contribute integrations to this feature, there are a few files that need to be created or modified:

* `values.yaml` - The main feature chart's values file. Add a section for your integration. It must contain an
`instance` array and any settings that apply to every instance of the integration. For example:

```yaml
<slug>:
instances: []
globalSetting: value
```

* `integrations/<slug>-values.yaml` - The values that will be used for each instance. This must include `name` to
differentiate it from other instances, and any other settings that are specific to that instance. For example:

```yaml
name: ""
labelSelectors:
app.kubernetes.io/name: my-service
protocol: http
...
```

* `templates/_integration<=_<slug>.tpl` - The file that contains template functions that build the configuration to
discover, gather, process, and deliver the telemetry data. This file is required to implement the following template
functions:
* `integrations.<slug>.type.metrics` - Returns true if this integration scrapes metrics.
* `integrations.<slug>.type.logs` - Returns true if this integration gathers logs.
* `integrations.<slug>.module` - Returns the configuration that is included once if this integration is used. This
is typically the module definition.
* `integrations.<slug>.include.metrics` - Returns the configuration that is included for each instance of the
integration that scrapes metrics.
* `integrations.<slug>.include.logs` - Returns the configuration that is included for each instance of the
integration that gathers logs.
* `integrations.<slug>.exclude.logs` - Returns a rule that can be used by other Log-gathering features to ensure
that logs that are gathered from this integration are not collected twice. Typically the inverse of a rule in
the `integrations.<slug>.include.logs` function.
* `default-allow-lists/<slug>.yaml` - If the integration scrapes metrics, a common pattern is to provide a list of
metrics that should be allowed. This minimizes the amount of metrics delivered to a useful minimal set.
2 changes: 1 addition & 1 deletion charts/feature-integrations/docs/integrations/alloy.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| scrapeInterval | string | `60s` | How frequently to scrape metrics from Windows Exporter. |
| scrapeInterval | string | `60s` | How frequently to scrape metrics from Alloy. |
55 changes: 55 additions & 0 deletions charts/feature-integrations/docs/integrations/mysql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# mysql

## Values

### Exporter Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| exporter.collectors | list | `["heartbeat","mysql.user"]` | The list of collectors to enable for the MySQL Exporter ([Documentation](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.mysql/#supported-collectors)). |
| exporter.dataSource | object | `{"auth":{"password":"","passwordFrom":"","passwordKey":"password","username":"","usernameFrom":"","usernameKey":"username"},"host":"","port":3306}` | The data source to use for the MySQL Exporter. |
| exporter.dataSource.auth.password | string | `""` | The password to use for the MySQL connection. |
| exporter.dataSource.auth.passwordFrom | string | `""` | Raw config for accessing the password. |
| exporter.dataSource.auth.passwordKey | string | `"password"` | The key for storing the password in the secret. |
| exporter.dataSource.auth.username | string | `""` | The username to use for the MySQL connection. |
| exporter.dataSource.auth.usernameFrom | string | `""` | Raw config for accessing the username. |
| exporter.dataSource.auth.usernameKey | string | `"username"` | The key for storing the username in the secret. |
| exporter.dataSource.host | string | `""` | The MySQL host to connect to. |
| exporter.dataSource.port | int | `3306` | The MySQL port to connect to. |
| exporter.dataSourceName | string | `""` | The data source string to use for the MySQL Exporter. |

### Logs Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| logs.enabled | bool | `true` | Whether to enable special processing of MySQL pod logs. |
| logs.labelSelectors | object | `{}` | Label selectors to choose the set of pods to gather logs from for this MySQL Instance. Example: `app.kubernetes.io/name: mysql` |

### Metric Processing Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| maxCacheSize | string | `100000` | Sets the max_cache_size for prometheus.relabel component. This should be at least 2x-5x your largest scrape target or samples appended rate. ([docs](https://grafana.com/docs/alloy/latest/reference/components/prometheus.relabel/#arguments)) Overrides global.maxCacheSize |
| metricsTuning.excludeMetrics | list | `[]` | Metrics to drop. Can use regular expressions. |
| metricsTuning.includeMetrics | list | `[]` | Metrics to keep. Can use regular expressions. |

### General Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| name | string | `""` | Name for this MySQL instance. |

### Scrape Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| scrapeInterval | string | `60s` | How frequently to scrape metrics from MySQL Exporter. |

### Secret

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| secret.create | bool | `true` | Whether to create a secret to store credentials for this MySQL integration instance. |
| secret.embed | bool | `false` | If true, skip secret creation and embed the credentials directly into the configuration. |
| secret.name | string | `""` | The name of the secret to create. |
| secret.namespace | string | `""` | The namespace for the secret. |
2 changes: 1 addition & 1 deletion charts/feature-integrations/integrations/alloy-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ name: ""
labelSelectors:
app.kubernetes.io/name: alloy

# -- How frequently to scrape metrics from Windows Exporter.
# -- How frequently to scrape metrics from Alloy.
# @default -- `60s`
# @section -- Scrape Settings
scrapeInterval:
Expand Down
94 changes: 94 additions & 0 deletions charts/feature-integrations/integrations/mysql-values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
---
# -- Name for this MySQL instance.
# @section -- General Settings
name: ""

# Settings for log gathering using the Pod Logs feature
logs:
# -- Whether to enable special processing of MySQL pod logs.
# @section -- Logs Settings
enabled: true

# -- Label selectors to choose the set of pods to gather logs from for this MySQL Instance.
# Example:
# `app.kubernetes.io/name: mysql`
# @section -- Logs Settings
labelSelectors: {}

# Settings for the Alloy embedded MySQL Exporter
exporter:
# -- The data source string to use for the MySQL Exporter.
# @section -- Exporter Settings
dataSourceName: ""

# -- The data source to use for the MySQL Exporter.
# @section -- Exporter Settings
dataSource:
# -- The MySQL host to connect to.
# @section -- Exporter Settings
host: ""
# -- The MySQL port to connect to.
# @section -- Exporter Settings
port: 3306
auth:
# -- The username to use for the MySQL connection.
# @section -- Exporter Settings
username: ""
# -- The key for storing the username in the secret.
# @section -- Exporter Settings
usernameKey: "username"
# -- Raw config for accessing the username.
# @section -- Exporter Settings
usernameFrom: ""
# -- The password to use for the MySQL connection.
# @section -- Exporter Settings
password: ""
# -- The key for storing the password in the secret.
# @section -- Exporter Settings
passwordKey: "password"
# -- Raw config for accessing the password.
# @section -- Exporter Settings
passwordFrom: ""

# -- The list of collectors to enable for the MySQL Exporter ([Documentation](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.mysql/#supported-collectors)).
# @section -- Exporter Settings
collectors:
- heartbeat
- mysql.user

secret:
# -- (bool) Whether to create a secret to store credentials for this MySQL integration instance.
# @default -- `true`
# @section -- Secret
create:
# -- If true, skip secret creation and embed the credentials directly into the configuration.
# @section -- Secret
embed: false
# -- The name of the secret to create.
# @section -- Secret
name: ""
# -- The namespace for the secret.
# @section -- Secret
namespace: ""

# -- How frequently to scrape metrics from MySQL Exporter.
# @default -- `60s`
# @section -- Scrape Settings
scrapeInterval:

# Adjustments to the scraped metrics to filter the amount of data sent to storage.
metricsTuning:
# -- Metrics to keep. Can use regular expressions.
# @section -- Metric Processing Settings
includeMetrics: []
# -- Metrics to drop. Can use regular expressions.
# @section -- Metric Processing Settings
excludeMetrics: []

# -- Sets the max_cache_size for prometheus.relabel component.
# This should be at least 2x-5x your largest scrape target or samples appended rate.
# ([docs](https://grafana.com/docs/alloy/latest/reference/components/prometheus.relabel/#arguments))
# Overrides global.maxCacheSize
# @default -- `100000`
# @section -- Metric Processing Settings
maxCacheSize:
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
{
"type": "object",
"properties": {
"exporter": {
"type": "object",
"properties": {
"collectors": {
"type": "array",
"items": {
"type": "string"
}
},
"dataSource": {
"type": "object",
"properties": {
"auth": {
"type": "object",
"properties": {
"password": {
"type": "string"
},
"passwordFrom": {
"type": "string"
},
"passwordKey": {
"type": "string"
},
"username": {
"type": "string"
},
"usernameFrom": {
"type": "string"
},
"usernameKey": {
"type": "string"
}
}
},
"host": {
"type": "string"
},
"port": {
"type": "integer"
}
}
},
"dataSourceName": {
"type": "string"
}
}
},
"logs": {
"type": "object",
"properties": {
"enabled": {
"type": "boolean"
},
"labelSelectors": {
"type": "object"
}
}
},
"maxCacheSize": {
"type": "null"
},
"metricsTuning": {
"type": "object",
"properties": {
"excludeMetrics": {
"type": "array"
},
"includeMetrics": {
"type": "array"
}
}
},
"name": {
"type": "string"
},
"scrapeInterval": {
"type": "null"
},
"secret": {
"type": "object",
"properties": {
"create": {
"type": "null"
},
"embed": {
"type": "boolean"
},
"name": {
"type": "string"
},
"namespace": {
"type": "string"
}
}
},
"type": {
"type": "string",
"const": "mysql"
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
"anyOf": [
{ "$ref": "#/definitions/alloy-integration"},
{ "$ref": "#/definitions/cert-manager-integration"},
{ "$ref": "#/definitions/etcd-integration"}
{ "$ref": "#/definitions/etcd-integration"},
{ "$ref": "#/definitions/mysql-integration"}
]
}
}
Expand Down
Loading

0 comments on commit 48a01f8

Please sign in to comment.