Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add extraProcessor for kubelet and systemd #3251

Merged
merged 6 commits into from
Sep 8, 2023
Merged

feat: add extraProcessor for kubelet and systemd #3251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a93eb01
feat: add extraProcessor for kubelet and systemd
Sep 7, 2023
485042b
chore: changelog
Sep 7, 2023
c01c0d1
fix: add tests and fix
Sep 7, 2023
a69d8a3
chore: format
Sep 7, 2023
d6a4cf1
Update docs/collecting-systemd-logs.md
sumo-drosiek Sep 8, 2023
5940250
Update docs/collecting-systemd-logs.md
sumo-drosiek Sep 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .changelog/3251.added.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
feat: add extraProcessor for kubelet and systemd
2 changes: 2 additions & 0 deletions deploy/helm/sumologic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `sumologic.logs.systemd.excludeHostRegex` | A regular expression for hosts. Matching hosts will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `Nil` |
| `sumologic.logs.systemd.excludePriorityRegex` | A regular expression for priority. Matching priority will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.systemd.excludeUnitRegex` | A regular expression for unit. Matching unit will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.systemd.otelcol.extraProcessors` | Extra processors for systemd logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details. | `[]` |
| `sumologic.logs.kubelet.sourceName` | Set the \_sourceName metadata field in Sumo Logic. | `"k8s_kubelet"` |
| `sumologic.logs.kubelet.sourceCategory` | Set the \_sourceCategory metadata field in Sumo Logic. | `"kubelet"` |
| `sumologic.logs.kubelet.sourceCategoryPrefix` | Set the prefix, for \_sourceCategory metadata. | `"kubernetes/"` |
Expand All @@ -84,6 +85,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `sumologic.logs.kubelet.excludeHostRegex` | A regular expression for hosts. Matching hosts will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.kubelet.excludePriorityRegex` | A regular expression for priority. Matching priority will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.kubelet.excludeUnitRegex` | A regular expression for unit. Matching unit will be excluded from Sumo. The logs will still be sent to logs metadata provider (FluentD/otelcol). | `""` |
| `sumologic.logs.kubelet.otelcol.extraProcessors` | Extra processors for systemd logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details. | `[]` |
| `sumologic.logs.defaultFluentd.sourceName` | Configuration option only for Fluentd, set the \_sourceName metadata field in Sumo Logic. | `k8s_default` |
| `sumologic.logs.defaultFluentd.sourceCategory` | Configuration option only for Fluentd, set the \_sourceCategory metadata field in Sumo Logic. | `default` |
| `sumologic.logs.defaultFluentd.sourceCategoryPrefix` | Configuration option only for Fluentd, set the prefix, for \_sourceCategory metadata. | `"kubernetes/"` |
Expand Down
32 changes: 32 additions & 0 deletions deploy/helm/sumologic/conf/logs/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,18 @@ processors:
{{- end }}
{{- end }}

{{- if .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{ toYaml $processor | indent 2}}
{{- end }}
{{- end }}

{{- if .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{ toYaml $processor | indent 2}}
{{- end }}
{{- end }}

receivers:
{{ if eq (include "logs.collector.fluentbit.enabled" .) "true" }}
fluentforward:
Expand Down Expand Up @@ -514,6 +526,11 @@ service:
- resource/add_cluster
- source/kubelet
- transform/add_timestamp
{{- if .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- fluentforward
Expand All @@ -536,6 +553,11 @@ service:
- resource/add_cluster
- source/systemd
- transform/add_timestamp
{{- if .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- fluentforward
Expand Down Expand Up @@ -603,6 +625,11 @@ service:
- transform/remove_attributes
- transform/flatten
- transform/add_timestamp
{{- if .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.kubelet.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- otlp
Expand Down Expand Up @@ -630,6 +657,11 @@ service:
- transform/remove_attributes
- transform/flatten
- transform/add_timestamp
{{- if .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.systemd.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- batch
receivers:
- otlp
Expand Down
8 changes: 8 additions & 0 deletions deploy/helm/sumologic/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,10 @@ sumologic:
# units:
# - docker.service

otelcol:
## Extra processors for systemd logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details.
extraProcessors: []

## Set the _sourceName metadata field in Sumo Logic.
sourceName: "%{_sourceName}"
## Set the _sourceCategory metadata field in Sumo Logic.
Expand All @@ -390,6 +394,10 @@ sumologic:
excludeUnitRegex: ""

kubelet:
otelcol:
## Extra processors for kubelet logs. See [/docs/collecting-systemd-logs.md](/docs/collecting-systemd-logs.md) for details.
extraProcessors: []

## Set the _sourceName metadata field in Sumo Logic.
sourceName: "k8s_kubelet"
## Set the _sourceCategory metadata field in Sumo Logic.
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Documentation links:
- [Examples](/docs/configuration-examples.md)
- Logs
- [Collecting container logs](/docs/collecting-container-logs.md)
- [Collecting Systemd logs](/docs/collecting-systemd-logs.md)
- Metrics
- [Collecting Kubernetes metrics](/docs/collecting-kubernetes-metrics.md)
- [Collecting application metrics](/docs/collecting-application-metrics.md)
Expand Down
17 changes: 17 additions & 0 deletions docs/collecting-container-logs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,23 @@

<!-- TOC -->

- [Collecting Container Logs](#collecting-container-logs)
- [Configuration](#configuration)
- [Multiline log parsing](#multiline-log-parsing)
- [Conditional multiline log parsing](#conditional-multiline-log-parsing)
- [Log format](#log-format)
- [fields log format](#fields-log-format)
- [json_merge log format](#json_merge-log-format)
- [text log format](#text-log-format)
- [Setting source name and other built-in metadata](#setting-source-name-and-other-built-in-metadata)
- [Filtering](#filtering)
- [Modifying log records](#modifying-log-records)
- [Adding custom fields](#adding-custom-fields)
- [Persistence](#persistence)
- [Advanced Configuration](#advanced-configuration)
- [Direct configuration](#direct-configuration)
- [Disabling container logs](#disabling-container-logs)

<!-- /TOC -->

By default, log collection is enabled. This includes both container logs and systemd logs. This document covers container logs.
Expand Down
Loading