[receiver/googlepubsub] Add Google Cloud Logging encoding #29299
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
2 times, most recently
from
42c1a0b to
5a628bf
Compare
alexvanboxel
added a commit
to alexvanboxel/opentelemetry-specification
that referenced
this pull request
Nov 19, 2023
## Changed * Clarify that the JSON representation is used for the LogEntry * Change to Pascal-case for representing the field in LogEntry * Map MonitoredResource to resource attributes with gcp prefix * Clarify the type of the Body for jsonPayload, protoPayload, and textPayload ## Added * Mapping section for Severity from Cloud Logging to OpenTelemetry * Mapping section for Severity from OpenTelemetry to Cloud Logging * Semantic Mapping section (only includes log.record.uid) * Clarified that the JSON representation is used for the LogEntry * Map LogEntryOperation to resource attribute with gcp.operation (KVList) * Map LogEntrySourceLocation to resource attribute with gcp.source_location (KVList) * Map LogSplit to resource attribute with gcp.log_split (KVList) ## Why With this draft PR (open-telemetry/opentelemetry-collector-contrib#29299), the PubsubReceiver will be able to handle the native translation of Google Cloud LogEntry's. But translations are opinionated. I prefer that we agree on the target mapping before merging the PR.
alexvanboxel
added a commit
to alexvanboxel/opentelemetry-specification
that referenced
this pull request
Nov 19, 2023
## Changed * Clarify that the JSON representation is used for the LogEntry * Change to Pascal-case for representing the field in LogEntry * Map MonitoredResource to resource attributes with gcp prefix * Clarify the type of the Body for jsonPayload, protoPayload, and textPayload ## Added * Mapping section for Severity from Cloud Logging to OpenTelemetry * Mapping section for Severity from OpenTelemetry to Cloud Logging * Semantic Mapping section (only includes log.record.uid) * Clarified that the JSON representation is used for the LogEntry * Map LogEntryOperation to resource attribute with gcp.operation (KVList) * Map LogEntrySourceLocation to resource attribute with gcp.source_location (KVList) * Map LogSplit to resource attribute with gcp.log_split (KVList) ## Why With this [draft collector PR](open-telemetry/opentelemetry-collector-contrib#29299), the PubsubReceiver will be able to handle the native translation of Google Cloud LogEntry's. But translations are opinionated. I prefer that we agree on the target mapping before merging the PR.
alexvanboxel
added a commit
to alexvanboxel/opentelemetry-specification
that referenced
this pull request
Nov 19, 2023
* Clarify that the JSON representation is used for the LogEntry * Change to Pascal-case for representing the field in LogEntry * Map MonitoredResource to resource attributes with gcp prefix * Clarify the type of the Body for jsonPayload, protoPayload, and textPayload * Mapping section for Severity from Cloud Logging to OpenTelemetry * Mapping section for Severity from OpenTelemetry to Cloud Logging * Semantic Mapping section (only includes log.record.uid) * Clarified that the JSON representation is used for the LogEntry * Map LogEntryOperation to resource attribute with gcp.operation (KVList) * Map LogEntrySourceLocation to resource attribute with gcp.source_location (KVList) * Map LogSplit to resource attribute with gcp.log_split (KVList) With this [draft collector PR](open-telemetry/opentelemetry-collector-contrib#29299), the PubsubReceiver will be able to handle the native translation of Google Cloud LogEntry's. But translations are opinionated. I prefer that we agree on the target mapping before merging the PR.
alexvanboxel
added a commit
to alexvanboxel/opentelemetry-specification
that referenced
this pull request
Nov 27, 2023
* Clarify that the JSON representation is used for the LogEntry * Change to Pascal-case for representing the field in LogEntry * Map MonitoredResource to resource attributes with gcp prefix * Clarify the type of the Body for jsonPayload, protoPayload, and textPayload * Mapping section for Severity from Cloud Logging to OpenTelemetry * Mapping section for Severity from OpenTelemetry to Cloud Logging * Semantic Mapping section (only includes log.record.uid) * Clarified that the JSON representation is used for the LogEntry * Map LogEntryOperation to resource attribute with gcp.operation (KVList) * Map LogEntrySourceLocation to resource attribute with gcp.source_location (KVList) * Map LogSplit to resource attribute with gcp.log_split (KVList) With this [draft collector PR](open-telemetry/opentelemetry-collector-contrib#29299), the PubsubReceiver will be able to handle the native translation of Google Cloud LogEntry's. But translations are opinionated. I prefer that we agree on the target mapping before merging the PR.
|
I would be interested in this functionality too. Any way I could assist? |
The PR is ready implementation-wise, but let's see if we agree on the spec first see: open-telemetry/opentelemetry-specification#3775 and open-telemetry/opentelemetry-specification#3774 |
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
from
5a628bf to
5862268
Compare
alexvanboxel
added a commit
to alexvanboxel/opentelemetry-specification
that referenced
this pull request
Dec 11, 2023
* Clarify that the JSON representation is used for the LogEntry * Change to Pascal-case for representing the field in LogEntry * Map MonitoredResource to resource attributes with gcp prefix * Clarify the type of the Body for jsonPayload, protoPayload, and textPayload * Mapping section for Severity from Cloud Logging to OpenTelemetry * Mapping section for Severity from OpenTelemetry to Cloud Logging * Semantic Mapping section (only includes log.record.uid) * Clarified that the JSON representation is used for the LogEntry * Map LogEntryOperation to resource attribute with gcp.operation (KVList) * Map LogEntrySourceLocation to resource attribute with gcp.source_location (KVList) * Map LogSplit to resource attribute with gcp.log_split (KVList) With this [draft collector PR](open-telemetry/opentelemetry-collector-contrib#29299), the PubsubReceiver will be able to handle the native translation of Google Cloud LogEntry's. But translations are opinionated. I prefer that we agree on the target mapping before merging the PR.
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
7 times, most recently
from
4fab86e to
c08836e
Compare
alexvanboxel
changed the title
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
from
c08836e to
e411fba
Compare
|
I'm the receiver's CODEOWNER. Note for reviewers: this is mainly an implementation by @kamalmarhubi; I've changed some conversions to match the |
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
from
e411fba to
4c07eb1
Compare
|
@atoulme, I've rebased (so as not to conflict with the new trace/metric imports). Could you give the PR a ready-to-merge? The |
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
from
4c07eb1 to
4a1dd54
Compare
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
2 times, most recently
from
8338602 to
2654244
Compare
|
Lovely! Exciting to try this out to forward Cloud Logging logs via Dataflow to otel-collector :) |
I'll try to fix the merge conflict by next Collector SIG meeting, because it's hard to get momentum on this. BTW, you don't need Dataflow for this, just Log routing. |
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
from
2654244 to
5d23e0e
Compare
|
Any news on this PR? |
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
from
5d23e0e to
69d8003
Compare
This commit addresses the need to accept logs from Google Cloud Platform into an OpenTelemetry pipeline. The goal is to introduce the `cloud_logging` encoding option for the receiver, allowing users to seamlessly integrate Cloud Logging logs into their OpenTelemetry pipeline. issue open-telemetry#23184 Co-authored-by: Alex Van Boxel <[email protected]>
alexvanboxel
force-pushed
the
pubsub-cloudlog
branch
from
69d8003 to
a4121c0
Compare
|
I also cannot merge anything. @open-telemetry/collector-contrib-maintainer can. |
Maybe you can accept the code review; the ready-for-merge label doesn't appear because I've pushed this? |
djaglowski
approved these changes
Feb 1, 2024
cparkins
pushed a commit
to AmadeusITGroup/opentelemetry-collector-contrib
that referenced
this pull request
Feb 1, 2024
…etry#29299) **Description:** This commit addresses the need to accept logs from Google Cloud Platform into an OpenTelemetry pipeline. The goal is to introduce the `cloud_logging` encoding option for the receiver, allowing users to seamlessly integrate Cloud Logging logs into their OpenTelemetry pipeline. **Link to tracking Issue:** open-telemetry#23184 **Testing:** Added tests for parsing LogEntries **Documentation:** Added the option to the README.md Co-authored-by: Kamal Al Marhubi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This commit addresses the need to accept logs from Google Cloud Platform
into an OpenTelemetry pipeline. The goal is to introduce the
cloud_loggingencoding option for the receiver, allowing users toseamlessly integrate Cloud Logging logs into their OpenTelemetry
pipeline.
Link to tracking Issue:
#23184
Testing:
Added tests for parsing LogEntries
Documentation:
Added the option to the README.md