-
Notifications
You must be signed in to change notification settings - Fork 28
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
17 changed files
with
430 additions
and
56 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| apiVersion: access.strimzi.io/v1alpha1 | ||
| kind: KafkaAccess | ||
| metadata: | ||
| name: mobu-kafka | ||
| spec: | ||
| kafka: | ||
| name: sasquatch | ||
| namespace: sasquatch | ||
| listener: tls | ||
| user: | ||
| kind: KafkaUser | ||
| apiGroup: kafka.strimzi.io | ||
| name: mobu | ||
| namespace: sasquatch |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| apiVersion: v2 | ||
| name: app-metrics | ||
| version: 1.0.0 | ||
| appVersion: "1.0.0" | ||
| description: Kafka topics, users, and a telegraf connector for metrics events. | ||
| type: application |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # app-metrics | ||
|
|
||
| Kafka topics, users, and a telegraf connector for metrics events. | ||
|
|
||
| ## Values | ||
|
|
||
| | Key | Type | Default | Description | | ||
| |-----|------|---------|-------------| | ||
| | affinity | object | `{}` | Affinity for pod assignment | | ||
| | args | list | `[]` | Arguments passed to the Telegraf agent containers | | ||
| | cluster.name | string | `"sasquatch"` | | | ||
| | collection_jitter | string | "0s" | Data collection jitter. This is used to jitter the collection by a random amount. Each plugin will sleep for a random time within jitter before collecting. | | ||
| | debug | bool | false | Run Telegraf in debug mode. | | ||
| | env | list | See `values.yaml` | Telegraf agent enviroment variables | | ||
| | envFromSecret | string | `""` | Name of the secret with values to be added to the environment. | | ||
| | flush_interval | string | "10s" | Data flushing interval for all outputs. Don’t set this below interval. Maximum flush_interval is flush_interval + flush_jitter | | ||
| | flush_jitter | string | "0s" | Jitter the flush interval by a random amount. This is primarily to avoid large write spikes for users running a large number of telegraf instances. | | ||
| | image.pullPolicy | string | `"Always"` | Image pull policy | | ||
| | image.repo | string | `"docker.io/library/telegraf"` | Telegraf image repository | | ||
| | image.tag | string | `"1.30.2-alpine"` | Telegraf image tag | | ||
| | imagePullSecrets | list | `[]` | Secret names to use for Docker pulls | | ||
| | influxdb.database | string | `"telegraf-kafka-app-metrics-consumer-v1"` | Name of the InfluxDB v1 database to write to | | ||
| | influxdb.url | string | `"http://sasquatch-influxdb.sasquatch:8086"` | URL of the InfluxDB v1 instance to write to | | ||
| | metric_batch_size | int | 5000 | Sends metrics to the output in batches of at most metric_batch_size metrics. | | ||
| | metric_buffer_limit | int | 100000 | Caches metric_buffer_limit metrics for each output, and flushes this buffer on a successful write. This should be a multiple of metric_batch_size and could not be less than 2 times metric_batch_size. | | ||
| | metricsApps[0] | string | `"mobu"` | | | ||
| | metricsApps[1] | string | `"some-other-app"` | | | ||
| | metricsApps[2] | string | `"yet-another-app"` | | | ||
| | metricsTags | string | `"[ \"service\", \"username\" ]\n"` | | | ||
| | nodeSelector | object | `{}` | Node labels for pod assignment | | ||
| | podAnnotations | object | `{}` | Annotations for telegraf-kafka-consumers pods | | ||
| | podLabels | object | `{}` | Labels for telegraf-kafka-consumer pods | | ||
| | replicaCount | int | `1` | Number of Telegraf replicas. Increase this value to increase the consumer throughput. | | ||
| | resources | object | See `values.yaml` | Kubernetes resources requests and limits | | ||
| | tolerations | list | `[]` | Tolerations for pod assignment | |
15 changes: 15 additions & 0 deletions
15
applications/sasquatch/charts/app-metrics/templates/kafka-topics.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| {{- range .Values.metricsApps }} | ||
| --- | ||
| apiVersion: kafka.strimzi.io/v1beta2 | ||
| kind: KafkaTopic | ||
| metadata: | ||
| name: "lsst.square.app-metrics.events.{{ . }}" | ||
| labels: | ||
| strimzi.io/cluster: {{ $.Values.cluster.name }} | ||
| spec: | ||
| partitions: 10 | ||
| replicas: 3 | ||
| config: | ||
| # http://kafka.apache.org/documentation/#topicconfigs | ||
| retention.ms: 86400000 # 1 day | ||
| {{- end }} |
31 changes: 31 additions & 0 deletions
31
applications/sasquatch/charts/app-metrics/templates/kafka-users.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| {{- range .Values.metricsApps }} | ||
| --- | ||
| apiVersion: kafka.strimzi.io/v1beta2 | ||
| kind: KafkaUser | ||
| metadata: | ||
| name: square-app-metrics-{{ . }} | ||
| labels: | ||
| strimzi.io/cluster: {{ $.Values.cluster.name }} | ||
| spec: | ||
| authentication: | ||
| type: tls | ||
| authorization: | ||
| type: simple | ||
| acls: | ||
| - resource: | ||
| type: group | ||
| name: app-metrics-events | ||
| patternType: prefix | ||
| operations: | ||
| - "Read" | ||
| host: "*" | ||
| - resource: | ||
| type: topic | ||
| name: "lsst.square.app-metrics.events.{{ . }}" | ||
| patternType: literal | ||
| operations: | ||
| - "Describe" | ||
| - "Read" | ||
| - "Write" | ||
| host: "*" | ||
| {{- end }} |
61 changes: 61 additions & 0 deletions
61
applications/sasquatch/charts/app-metrics/templates/telegraf-configmap.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: sasquatch-telegraf-app-metrics | ||
| labels: | ||
| app.kubernetes.io/name: sasquatch-telegraf | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| app.kubernetes.io/part-of: sasquatch | ||
| data: | ||
| telegraf.conf: |+ | ||
| [agent] | ||
| metric_batch_size = 5000 | ||
| metric_buffer_limit = 100000 | ||
| collection_jitter = "0s" | ||
| flush_interval = "10s" | ||
| flush_jitter = "0s" | ||
| debug = {{ default false .Values.debug }} | ||
| omit_hostname = true | ||
| [[outputs.influxdb]] | ||
| urls = [ | ||
| {{ .Values.influxdb.url | quote }} | ||
| ] | ||
| database = {{ .Values.influxdb.database | quote }} | ||
| username = "${INFLUXDB_USER}" | ||
| password = "${INFLUXDB_PASSWORD}" | ||
| [[outputs.influxdb]] | ||
| namepass = ["telegraf_*"] | ||
| urls = [ | ||
| {{ .Values.influxdb.url | quote }} | ||
| ] | ||
| database = "telegraf" | ||
| username = "${INFLUXDB_USER}" | ||
| password = "${INFLUXDB_PASSWORD}" | ||
| [[inputs.kafka_consumer]] | ||
| brokers = [ | ||
| "sasquatch-kafka-brokers.sasquatch:9092" | ||
| ] | ||
| consumer_group = "telegraf-kafka-consumer-app-metrics" | ||
| sasl_mechanism = "SCRAM-SHA-512" | ||
| sasl_password = "$TELEGRAF_PASSWORD" | ||
| sasl_username = "telegraf" | ||
| data_format = "avro" | ||
| avro_schema_registry = "http://sasquatch-schema-registry.sasquatch:8081" | ||
| avro_timestamp = "timestamp_ns" | ||
| avro_timestamp_format = "unix_ns" | ||
| avro_union_mode = "nullable" | ||
| avro_tags = {{ .Values.metricsTags }} | ||
| topic_regexps = [ "lsst.square.app-metrics.events.*" ] | ||
| max_processing_time = "5s" | ||
| consumer_fetch_default = "5MB" | ||
| max_undelivered_messages = 10000 | ||
| compression_codec = 3 | ||
| [[inputs.internal]] | ||
| name_prefix = "telegraf_" | ||
| collect_memstats = true | ||
| tags = { instance = "app-metrics" } |
78 changes: 78 additions & 0 deletions
78
applications/sasquatch/charts/app-metrics/templates/telegraf-deployment.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: sasquatch-telegraf-app-metrics | ||
| labels: | ||
| app.kubernetes.io/name: sasquatch-telegraf | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| app.kubernetes.io/part-of: sasquatch | ||
| spec: | ||
| replicas: {{ default 1 .Values.replicaCount }} | ||
| selector: | ||
| matchLabels: | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app.kubernetes.io/instance: sasquatch-telegraf-app-metrics | ||
| annotations: | ||
| checksum/apps: {{ .Values.metricsApps | toString | sha256sum }} | ||
| {{- if .Values.podAnnotations }} | ||
| {{- toYaml .Values.podAnnotations | nindent 8 }} | ||
| {{- end }} | ||
| spec: | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| runAsUser: 1000 | ||
| runAsGroup: 1000 | ||
| containers: | ||
| - name: telegraf | ||
| securityContext: | ||
| capabilities: | ||
| drop: | ||
| - all | ||
| readOnlyRootFilesystem: true | ||
| allowPrivilegeEscalation: false | ||
| image: "{{ .Values.image.repo }}:{{ .Values.image.tag }}" | ||
| imagePullPolicy: {{ default "IfNotPresent" .Values.image.pullPolicy | quote }} | ||
| {{- if .Values.resources }} | ||
| resources: | ||
| {{- toYaml .Values.resources | nindent 10 }} | ||
| {{- end }} | ||
| {{- if .Values.args }} | ||
| args: | ||
| {{- toYaml .Values.args | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.env }} | ||
| env: | ||
| {{- toYaml .Values.env | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.envFromSecret }} | ||
| envFrom: | ||
| - secretRef: | ||
| name: {{ .Values.envFromSecret }} | ||
| {{- end }} | ||
| volumeMounts: | ||
| - name: config | ||
| mountPath: /etc/telegraf | ||
| {{- if .Values.imagePullSecrets }} | ||
| imagePullSecrets: | ||
| {{- toYaml .Values.imagePullSecrets | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.nodeSelector }} | ||
| nodeSelector: | ||
| {{- toYaml .Values.nodeSelector | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.affinity }} | ||
| affinity: | ||
| {{- toYaml .Values.affinity | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.tolerations }} | ||
| tolerations: | ||
| {{- toYaml .Values.tolerations | nindent 8 }} | ||
| {{- end }} | ||
| volumes: | ||
| - name: config | ||
| configMap: | ||
| name: sasquatch-telegraf-app-metrics |
Oops, something went wrong.