Skip to content

Commit

Permalink
feat: add extraProcessor for kubelet and systemd (#3251)
Browse files Browse the repository at this point in the history 
* feat: add extraProcessor for kubelet and systemd

Signed-off-by: Dominik Rosiek <[email protected]>

* chore: changelog

Signed-off-by: Dominik Rosiek <[email protected]>

* fix: add tests and fix

Signed-off-by: Dominik Rosiek <[email protected]>

* chore: format

Signed-off-by: Dominik Rosiek <[email protected]>

* Update docs/collecting-systemd-logs.md

Co-authored-by: Mikołaj Świątek <[email protected]>

* Update docs/collecting-systemd-logs.md

Co-authored-by: Mikołaj Świątek <[email protected]>

---------

Signed-off-by: Dominik Rosiek <[email protected]>
Co-authored-by: Mikołaj Świątek <[email protected]>
  • Loading branch information
@sumo-drosiek @swiatekm
sumo-drosiek and swiatekm authored Sep 8, 2023
1 parent fc40217 commit 9262326
Show file tree
Hide file tree
Showing 9 changed files with 402 additions and 0 deletions.
1 change: 1 addition & 0 deletions .changelog/3251.added.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
feat: add extraProcessor for kubelet and systemd
2 changes: 2 additions & 0 deletions deploy/helm/sumologic/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `sumologic.logs.systemd.excludeHostRegex` | A regular expression for hosts. Matching hosts will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `Nil` |
| `sumologic.logs.systemd.excludePriorityRegex` | A regular expression for priority. Matching priority will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.systemd.excludeUnitRegex` | A regular expression for unit. Matching unit will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.systemd.otelcol.extraProcessors` | Extra processors for systemd logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details. | `[]` |
| `sumologic.logs.kubelet.sourceName` | Set the \_sourceName metadata field in Sumo Logic. | `"k8s_kubelet"` |
| `sumologic.logs.kubelet.sourceCategory` | Set the \_sourceCategory metadata field in Sumo Logic. | `"kubelet"` |
| `sumologic.logs.kubelet.sourceCategoryPrefix` | Set the prefix, for \_sourceCategory metadata. | `"kubernetes/"` |
Expand All @@ -84,6 +85,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `sumologic.logs.kubelet.excludeHostRegex` | A regular expression for hosts. Matching hosts will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.kubelet.excludePriorityRegex` | A regular expression for priority. Matching priority will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.kubelet.excludeUnitRegex` | A regular expression for unit. Matching unit will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.kubelet.otelcol.extraProcessors` | Extra processors for systemd logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details. | `[]` |
| `sumologic.logs.defaultFluentd.sourceName` | Configuration option only for Fluentd, set the \_sourceName metadata field in Sumo Logic. | `k8s_default` |
| `sumologic.logs.defaultFluentd.sourceCategory` | Configuration option only for Fluentd, set the \_sourceCategory metadata field in Sumo Logic. | `default` |
| `sumologic.logs.defaultFluentd.sourceCategoryPrefix` | Configuration option only for Fluentd, set the prefix, for \_sourceCategory metadata. | `"kubernetes/"` |
Expand Down
32 changes: 32 additions & 0 deletions deploy/helm/sumologic/conf/logs/otelcol/config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,18 @@ processors:
{{- end }}
{{- end }}

{{- if .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{ toYaml $processor | indent 2}}
{{- end }}
{{- end }}

{{- if .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{ toYaml $processor | indent 2}}
{{- end }}
{{- end }}

receivers:
{{ if eq (include "logs.collector.fluentbit.enabled" .) "true" }}
fluentforward:
Expand Down Expand Up @@ -514,6 +526,11 @@ service:
- resource/add_cluster
- source/kubelet
- transform/add_timestamp
{{- if .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- fluentforward
Expand All @@ -536,6 +553,11 @@ service:
- resource/add_cluster
- source/systemd
- transform/add_timestamp
{{- if .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- fluentforward
Expand Down Expand Up @@ -603,6 +625,11 @@ service:
- transform/remove_attributes
- transform/flatten
- transform/add_timestamp
{{- if .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- otlp
Expand Down Expand Up @@ -630,6 +657,11 @@ service:
- transform/remove_attributes
- transform/flatten
- transform/add_timestamp
{{- if .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- otlp
Expand Down
8 changes: 8 additions & 0 deletions deploy/helm/sumologic/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,10 @@ sumologic:
# units:
# - docker.service

otelcol:
## Extra processors for systemd logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details.
extraProcessors: []

## Set the _sourceName metadata field in Sumo Logic.
sourceName: "%{_sourceName}"
## Set the _sourceCategory metadata field in Sumo Logic.
Expand All @@ -395,6 +399,10 @@ sumologic:
excludeUnitRegex: ""

kubelet:
otelcol:
## Extra processors for kubelet logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details.
extraProcessors: []

## Set the _sourceName metadata field in Sumo Logic.
sourceName: "k8s_kubelet"
## Set the _sourceCategory metadata field in Sumo Logic.
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Documentation links:
- [Examples](/docs/configuration-examples.md)
- Logs
- [Collecting container logs](/docs/collecting-container-logs.md)
- [Collecting Systemd logs](/docs/collecting-systemd-logs.md)
- Metrics
- [Collecting Kubernetes metrics](/docs/collecting-kubernetes-metrics.md)
- [Collecting application metrics](/docs/collecting-application-metrics.md)
Expand Down
17 changes: 17 additions & 0 deletions docs/collecting-container-logs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,23 @@

<!-- TOC -->

- [Collecting Container Logs](#collecting-container-logs)
- [Configuration](#configuration)
- [Multiline log parsing](#multiline-log-parsing)
- [Conditional multiline log parsing](#conditional-multiline-log-parsing)
- [Log format](#log-format)
- [fields log format](#fields-log-format)
- [json_merge log format](#json_merge-log-format)
- [text log format](#text-log-format)
- [Setting source name and other built-in metadata](#setting-source-name-and-other-built-in-metadata)
- [Filtering](#filtering)
- [Modifying log records](#modifying-log-records)
- [Adding custom fields](#adding-custom-fields)
- [Persistence](#persistence)
- [Advanced Configuration](#advanced-configuration)
- [Direct configuration](#direct-configuration)
- [Disabling container logs](#disabling-container-logs)

<!-- /TOC -->

By default, log collection is enabled. This includes both container logs and systemd logs. This document covers container logs.
Expand Down
Loading

0 comments on commit 9262326

Please sign in to comment.