-
Notifications
You must be signed in to change notification settings - Fork 28
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
18 changed files
with
418 additions
and
56 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| apiVersion: access.strimzi.io/v1alpha1 | ||
| kind: KafkaAccess | ||
| metadata: | ||
| name: mobu-kafka | ||
| spec: | ||
| kafka: | ||
| name: sasquatch | ||
| namespace: sasquatch | ||
| listener: tls | ||
| user: | ||
| kind: KafkaUser | ||
| apiGroup: kafka.strimzi.io | ||
| name: mobu | ||
| namespace: sasquatch |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| apiVersion: v2 | ||
| name: app-metrics | ||
| version: 1.0.0 | ||
| appVersion: "1.0.0" | ||
| description: Kafka topics, users, and a telegraf connector for metrics events. | ||
| type: application |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # app-metrics | ||
|
|
||
| Kafka topics, users, and a telegraf connector for metrics events. | ||
|
|
||
| ## Values | ||
|
|
||
| | Key | Type | Default | Description | | ||
| |-----|------|---------|-------------| | ||
| | affinity | object | `{}` | Affinity for pod assignment | | ||
| | apps | list | `[]` | A list of applications that will publish metrics events, and the keys that should be ingested into InfluxDB as tags. The names should be the same as the app names in Phalanx. | | ||
| | args | list | `[]` | Arguments passed to the Telegraf agent containers | | ||
| | cluster.name | string | `"sasquatch"` | | | ||
| | debug | bool | false | Run Telegraf in debug mode. | | ||
| | env | list | See `values.yaml` | Telegraf agent enviroment variables | | ||
| | envFromSecret | string | `""` | Name of the secret with values to be added to the environment. | | ||
| | globalInfluxTags | list | `["service"]` | Keys in an every event sent by any app that should be recorded in InfluxDB as "tags" (vs. "fields"). | | ||
| | image.pullPolicy | string | `"Always"` | Image pull policy | | ||
| | image.repo | string | `"docker.io/library/telegraf"` | Telegraf image repository | | ||
| | image.tag | string | `"1.30.2-alpine"` | Telegraf image tag | | ||
| | imagePullSecrets | list | `[]` | Secret names to use for Docker pulls | | ||
| | influxdb.url | string | `"http://sasquatch-influxdb.sasquatch:8086"` | URL of the InfluxDB v1 instance to write to | | ||
| | nodeSelector | object | `{}` | Node labels for pod assignment | | ||
| | podAnnotations | object | `{}` | Annotations for telegraf-kafka-consumers pods | | ||
| | podLabels | object | `{}` | Labels for telegraf-kafka-consumer pods | | ||
| | replicaCount | int | `1` | Number of Telegraf replicas. Increase this value to increase the consumer throughput. | | ||
| | resources | object | See `values.yaml` | Kubernetes resources requests and limits | | ||
| | tolerations | list | `[]` | Tolerations for pod assignment | |
15 changes: 15 additions & 0 deletions
15
applications/sasquatch/charts/app-metrics/templates/kafka-topics.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| {{- range .Values.apps }} | ||
| --- | ||
| apiVersion: kafka.strimzi.io/v1beta2 | ||
| kind: KafkaTopic | ||
| metadata: | ||
| name: "lsst.square.app-metrics.events.{{ . }}" | ||
| labels: | ||
| strimzi.io/cluster: {{ $.Values.cluster.name }} | ||
| spec: | ||
| partitions: 10 | ||
| replicas: 3 | ||
| config: | ||
| # http://kafka.apache.org/documentation/#topicconfigs | ||
| retention.ms: 86400000 # 1 day | ||
| {{- end }} |
31 changes: 31 additions & 0 deletions
31
applications/sasquatch/charts/app-metrics/templates/kafka-users.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| {{- range .Values.apps }} | ||
| --- | ||
| apiVersion: kafka.strimzi.io/v1beta2 | ||
| kind: KafkaUser | ||
| metadata: | ||
| name: square-app-metrics-{{ . }} | ||
| labels: | ||
| strimzi.io/cluster: {{ $.Values.cluster.name }} | ||
| spec: | ||
| authentication: | ||
| type: tls | ||
| authorization: | ||
| type: simple | ||
| acls: | ||
| - resource: | ||
| type: group | ||
| name: app-metrics-events | ||
| patternType: prefix | ||
| operations: | ||
| - "Read" | ||
| host: "*" | ||
| - resource: | ||
| type: topic | ||
| name: "lsst.square.app-metrics.events.{{ . }}" | ||
| patternType: literal | ||
| operations: | ||
| - "Describe" | ||
| - "Read" | ||
| - "Write" | ||
| host: "*" | ||
| {{- end }} |
65 changes: 65 additions & 0 deletions
65
applications/sasquatch/charts/app-metrics/templates/telegraf-configmap.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: sasquatch-telegraf-app-metrics | ||
| labels: | ||
| app.kubernetes.io/name: sasquatch-telegraf | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| app.kubernetes.io/part-of: sasquatch | ||
| data: | ||
| telegraf.conf: |+ | ||
| [agent] | ||
| metric_batch_size = 5000 | ||
| metric_buffer_limit = 100000 | ||
| collection_jitter = "0s" | ||
| flush_interval = "10s" | ||
| flush_jitter = "0s" | ||
| debug = {{ default false .Values.debug }} | ||
| omit_hostname = true | ||
| [[outputs.influxdb]] | ||
| urls = [ | ||
| {{ .Values.influxdb.url | quote }} | ||
| ] | ||
| database = "telegraf-kafka-app-metrics-consumer" | ||
| username = "${INFLUXDB_USER}" | ||
| password = "${INFLUXDB_PASSWORD}" | ||
| [[outputs.influxdb]] | ||
| namepass = ["telegraf_*"] | ||
| urls = [ | ||
| {{ .Values.influxdb.url | quote }} | ||
| ] | ||
| database = "telegraf" | ||
| username = "${INFLUXDB_USER}" | ||
| password = "${INFLUXDB_PASSWORD}" | ||
| {{- range $index, $app := .Values.apps }} | ||
| [[inputs.kafka_consumer]] | ||
| brokers = [ | ||
| "sasquatch-kafka-brokers.sasquatch:9092" | ||
| ] | ||
| consumer_group = "telegraf-kafka-consumer-app-metrics" | ||
| sasl_mechanism = "SCRAM-SHA-512" | ||
| sasl_password = "$TELEGRAF_PASSWORD" | ||
| sasl_username = "telegraf" | ||
| data_format = "avro" | ||
| avro_schema_registry = "http://sasquatch-schema-registry.sasquatch:8081" | ||
| avro_timestamp = "timestamp_ns" | ||
| avro_timestamp_format = "unix_ns" | ||
| avro_union_mode = "nullable" | ||
| avro_tags = {{ include "helpers.toTomlArray" (index $.Values.global "app-metrics-config" $app "influxTags") }} | ||
| topics = [ | ||
| "lsst.square.app-metrics.events.{{ $app }}", | ||
| ] | ||
| max_processing_time = "5s" | ||
| consumer_fetch_default = "5MB" | ||
| max_undelivered_messages = 10000 | ||
| compression_codec = 3 | ||
| {{- end }} | ||
| [[inputs.internal]] | ||
| name_prefix = "telegraf_" | ||
| collect_memstats = true | ||
| tags = { instance = "app-metrics" } |
78 changes: 78 additions & 0 deletions
78
applications/sasquatch/charts/app-metrics/templates/telegraf-deployment.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: sasquatch-telegraf-app-metrics | ||
| labels: | ||
| app.kubernetes.io/name: sasquatch-telegraf | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| app.kubernetes.io/part-of: sasquatch | ||
| spec: | ||
| replicas: {{ default 1 .Values.replicaCount }} | ||
| selector: | ||
| matchLabels: | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| annotations: | ||
| checksum/config: {{ include (print $.Template.BasePath "/telegraf-configmap.yaml") $ | sha256sum }} | ||
| {{- if .Values.podAnnotations }} | ||
| {{- toYaml .Values.podAnnotations | nindent 8 }} | ||
| {{- end }} | ||
| spec: | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| runAsUser: 1000 | ||
| runAsGroup: 1000 | ||
| containers: | ||
| - name: telegraf | ||
| securityContext: | ||
| capabilities: | ||
| drop: | ||
| - all | ||
| readOnlyRootFilesystem: true | ||
| allowPrivilegeEscalation: false | ||
| image: "{{ .Values.image.repo }}:{{ .Values.image.tag }}" | ||
| imagePullPolicy: {{ default "IfNotPresent" .Values.image.pullPolicy | quote }} | ||
| {{- if .Values.resources }} | ||
| resources: | ||
| {{- toYaml .Values.resources | nindent 10 }} | ||
| {{- end }} | ||
| {{- if .Values.args }} | ||
| args: | ||
| {{- toYaml .Values.args | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.env }} | ||
| env: | ||
| {{- toYaml .Values.env | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.envFromSecret }} | ||
| envFrom: | ||
| - secretRef: | ||
| name: {{ .Values.envFromSecret }} | ||
| {{- end }} | ||
| volumeMounts: | ||
| - name: config | ||
| mountPath: /etc/telegraf | ||
| {{- if .Values.imagePullSecrets }} | ||
| imagePullSecrets: | ||
| {{- toYaml .Values.imagePullSecrets | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.nodeSelector }} | ||
| nodeSelector: | ||
| {{- toYaml .Values.nodeSelector | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.affinity }} | ||
| affinity: | ||
| {{- toYaml .Values.affinity | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.tolerations }} | ||
| tolerations: | ||
| {{- toYaml .Values.tolerations | nindent 8 }} | ||
| {{- end }} | ||
| volumes: | ||
| - name: config | ||
| configMap: | ||
| name: sasquatch-telegraf-app-metrics |
Oops, something went wrong.