Skip to content

Commit

Permalink
receiver/googlecloudpubsub: Add support for Cloud Logging messages
Browse files Browse the repository at this point in the history 
Add a new `cloud_logging` encoding, which allows the receiver to
consume messages from a Pub/Sub topic that's used as the destination
of a Cloud Logging sink.

refs open-telemetry#23184
  • Loading branch information
@kamalmarhubi
kamalmarhubi committed Oct 8, 2023
1 parent fa88837 commit 4e84241
Show file tree
Hide file tree
Showing 8 changed files with 806 additions and 11 deletions.
16 changes: 12 additions & 4 deletions receiver/googlecloudpubsubreceiver/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ The following configuration options are supported:
* `subscription` (Required): The subscription name to receive OTLP data from. The subscription name should be a
fully qualified resource name (eg: `projects/otel-project/subscriptions/otlp`).
* `encoding` (Optional): The encoding that will be used to received data from the subscription. This can either be
`otlp_proto_trace`, `otlp_proto_metric`, `otlp_proto_log`, or `raw_text` (see `encoding`). This will only be used as
a fallback, when no `content-type` attribute is present.
`otlp_proto_trace`, `otlp_proto_metric`, `otlp_proto_log`, `cloud_logging`, or `raw_text` (see `encoding`). This will
only be used as a fallback, when no `content-type` attribute is present.
* `compression` (Optional): The compression that will be used on received data from the subscription. When set it can
only be `gzip`. This will only be used as a fallback, when no `content-encoding` attribute is present.
* `endpoint` (Optional): Override the default Pubsub Endpoint, useful when connecting to the PubSub emulator instance
Expand Down Expand Up @@ -54,12 +54,20 @@ must the `encoding` field in the configuration be set.
| - | - | otlp_proto_trace | Decode OTLP trace message |
| - | - | otlp_proto_metric | Decode OTLP trace message |
| - | - | otlp_proto_log | Decode OTLP trace message |
| - | - | cloud_logging | Decode [Cloud Logging]'s [LogEntry] message type
| - | - | raw_text | Wrap in an OTLP log message |

When the `encoding` configuration is set, the attributes on the message are ignored.

The receiver can be used for ingesting arbitrary text message on a Pubsub subscription and wrap them in OTLP Log
message, making it a convenient way to ingest log lines from Pubsub.
With `cloud_logging`, the receiver can be used to bring Cloud Logging messages into an OpenTelemetry pipeline. You'll
first need to [set up a logging sink][sink-docs] with a Pub/Sub topic as its destination.

With `raw_text`, the receiver can be used for ingesting arbitrary text message on a Pubsub subscription, wrapping them
in OTLP Log messages, making it a convenient way to ingest raw log lines from Pubsub.

[Cloud Logging]: https://cloud.google.com/logging
[LogEntry]: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
[sink-docs]: https://cloud.google.com/logging/docs/export/configure_export_v2#creating_sink

## Pubsub subscription

Expand Down
3 changes: 2 additions & 1 deletion receiver/googlecloudpubsubreceiver/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ func (config *Config) validateForLog() error {
case "otlp_proto_log":
case "raw_text":
case "raw_json":
case "cloud_logging":
default:
return fmt.Errorf("log encoding %v is not supported. supported encoding formats include [otlp_proto_log,raw_text,raw_json]", config.Encoding)
return fmt.Errorf("log encoding %v is not supported. supported encoding formats include [otlp_proto_log,raw_text,raw_json,cloud_logging]", config.Encoding)
}
return nil
}
Expand Down
15 changes: 9 additions & 6 deletions receiver/googlecloudpubsubreceiver/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,34 +3,41 @@ module github.com/open-telemetry/opentelemetry-collector-contrib/receiver/google
go 1.20

require (
cloud.google.com/go/logging v1.7.0
cloud.google.com/go/pubsub v1.33.0
github.com/google/go-cmp v0.5.9
github.com/iancoleman/strcase v0.2.0
github.com/json-iterator/go v1.1.12
github.com/stretchr/testify v1.8.4
go.opentelemetry.io/collector/component v0.86.1-0.20231006161201-d364ad61c4d7
go.opentelemetry.io/collector/confmap v0.86.1-0.20231006161201-d364ad61c4d7
go.opentelemetry.io/collector/consumer v0.86.1-0.20231006161201-d364ad61c4d7
go.opentelemetry.io/collector/exporter v0.86.1-0.20231006161201-d364ad61c4d7
go.opentelemetry.io/collector/pdata v1.0.0-rcv0015.0.20231006161201-d364ad61c4d7
go.opentelemetry.io/collector/receiver v0.86.1-0.20231006161201-d364ad61c4d7
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.26.0
google.golang.org/api v0.143.0
google.golang.org/genproto v0.0.0-20230913181813-007df8e322eb
google.golang.org/genproto/googleapis/api v0.0.0-20230913181813-007df8e322eb
google.golang.org/grpc v1.58.2
google.golang.org/protobuf v1.31.0
)

require (
cloud.google.com/go v0.110.7 // indirect
cloud.google.com/go/compute v1.23.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.1 // indirect
cloud.google.com/go/longrunning v0.5.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.1 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/knadh/koanf v1.5.0 // indirect
github.com/knadh/koanf/v2 v2.0.1 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
Expand All @@ -47,18 +54,14 @@ require (
go.opentelemetry.io/otel v1.19.0 // indirect
go.opentelemetry.io/otel/metric v1.19.0 // indirect
go.opentelemetry.io/otel/trace v1.19.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.13.0 // indirect
golang.org/x/net v0.15.0 // indirect
golang.org/x/oauth2 v0.12.0 // indirect
golang.org/x/sync v0.3.0 // indirect
golang.org/x/sys v0.12.0 // indirect
golang.org/x/text v0.13.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230920204549-e6e6cdab5c13 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

Expand Down
6 changes: 6 additions & 0 deletions receiver/googlecloudpubsubreceiver/go.sum
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 19 additions & 0 deletions receiver/googlecloudpubsubreceiver/internal/common_protos.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package internal // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudpubsubreceiver/internal"

import (
_ "google.golang.org/genproto/googleapis/cloud/audit"
)
Loading

0 comments on commit 4e84241

Please sign in to comment.