Skip to content

Commit

Permalink
feat: depot httpv2 sink (#21)
Browse files Browse the repository at this point in the history 
* feat: create common http client (#53)

* feat: create bare minimum http sink (#56)

* feat: create bare minimum http sink

* feat: add unit test

* test: fix test

* refactor: refactor request creation into factory class

* refactor: remove comments

* chore: minor fixes

* chore: disable parameters in configs

* chore: minor fixes

* fix: fix metadata to also support generic value

* chore: abstract metadata util

* chore: http sink config extend to http client config

* chore: refactor config class definition

* chore: query param as map

* chore: remove null check

* chore: refactor

* chore: minor refactoring

* chore: create converter for header config

Co-authored-by: lavkesh <[email protected]>
Co-authored-by: jesry.pandawa <[email protected]>

* fix: fix single request (raystack#64)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: qa and review fixes

* chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true

* feat: Bigtable record parser (#39)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* feat: create bigtable records using InputOutputFieldMapping provided as configuration

* refactor: fix checkstyle and add unit tests

* review: minor refactor

* refactor: add BigTableSchemaTest and fix BigTableRecordParserTest

* refactor: fix checkstyle

* tests: add few more tests

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: schema refactoring

* chore: naming conventions

* feat: add functionality to create rowkey from configured template (#44)

* chore: refactor Template validation

* chore: change exception message in Template

* feat: add bigtable sink metrics and logging (#51)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: qa and review fixes

* chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true

* feat: Bigtable record parser (#39)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* feat: create bigtable records using InputOutputFieldMapping provided as configuration

* refactor: fix checkstyle and add unit tests

* review: minor refactor

* refactor: add BigTableSchemaTest and fix BigTableRecordParserTest

* refactor: fix checkstyle

* tests: add few more tests

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: schema refactoring

* chore: naming conventions

* feat: add functionality to create rowkey from configured template (#44)

* chore: refactor Template validation

* chore: change exception message in Template

* feat: add bigtable sink metrics and logging (#51)

* feat: Bigtable error parsing (#55)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: qa and review fixes

* chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true

* feat: Bigtable record parser (#39)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* feat: create bigtable records using InputOutputFieldMapping provided as configuration

* refactor: fix checkstyle and add unit tests

* review: minor refactor

* refactor: add BigTableSchemaTest and fix BigTableRecordParserTest

* refactor: fix checkstyle

* tests: add few more tests

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: schema refactoring

* chore: naming conventions

* feat: add functionality to create rowkey from configured template (#44)

* chore: refactor Template validation

* chore: change exception message in Template

* feat: add bigtable sink metrics and logging (#51)

* feat: parse bigtable errors and create odpf response

* feat: capture error metrics with predefined tags

* chore: add tests for bigtable response parser

* chore: remove deprecated jcenter repo from build.gradle

* chore: comment out classpath dependencies

* chore: remove unused classpath dependencies

* chore: minor refactor

* chore: remove unused classpath dependencies

* refactor: change BigtableRecord,Response contracts, improved logging, added tests

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: minor fixes (raystack#59)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: qa and review fixes

* chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true

* feat: Bigtable record parser (#39)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* feat: create bigtable records using InputOutputFieldMapping provided as configuration

* refactor: fix checkstyle and add unit tests

* review: minor refactor

* refactor: add BigTableSchemaTest and fix BigTableRecordParserTest

* refactor: fix checkstyle

* tests: add few more tests

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: schema refactoring

* chore: naming conventions

* feat: add functionality to create rowkey from configured template (#44)

* chore: refactor Template validation

* chore: change exception message in Template

* feat: add bigtable sink metrics and logging (#51)

* feat: parse bigtable errors and create odpf response

* feat: capture error metrics with predefined tags

* chore: add tests for bigtable response parser

* chore: remove deprecated jcenter repo from build.gradle

* chore: add bq error logs (#57)

* chore: comment out classpath dependencies

* chore: remove unused classpath dependencies

* chore: minor refactor

* chore: remove unused classpath dependencies

* refactor: change BigtableRecord,Response contracts, improved logging, added tests

* chore: small fixes

Co-authored-by: Mayur Gubrele <[email protected]>
Co-authored-by: mayur.gubrele <[email protected]>

* docs: adds bigtable sink documentation (#58)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: qa and review fixes

* chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true

* feat: Bigtable record parser (#39)

* feat: Bigtable Sink (#33)

* feat: add bigtable sink with stringified odpf messages as values

* feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud

* feat: add wrapper class on bigtable client

* refactor: fix checkstyle

* feat: add bigtable parser tests

* feat: add bigtable sink tests

* feat: add bigtable client tests

* chore: revert version bump

* chore: revert version change in build.gradle

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* feat: create bigtable records using InputOutputFieldMapping provided as configuration

* refactor: fix checkstyle and add unit tests

* review: minor refactor

* refactor: add BigTableSchemaTest and fix BigTableRecordParserTest

* refactor: fix checkstyle

* tests: add few more tests

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: schema refactoring

* chore: naming conventions

* feat: add functionality to create rowkey from configured template (#44)

* chore: refactor Template validation

* chore: change exception message in Template

* feat: add bigtable sink metrics and logging (#51)

* docs: add bigtable sink documentation

Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* chore: rc version

* fix: fix uri and exception handler

* feat: exclude io.grpc from bigtable library and bump the version to 0.3.4-beta.4

* chore: add iterator in http record and toString in Template

* chore: add more exception handler

* Revert "chore: add more exception handler"

This reverts commit 0edb3b7.

* Revert "chore: add iterator in http record and toString in Template"

This reverts commit a18bad6.

* fix: fix template, uri and http exception handling

* fix: fix exception handling on response and request body

* fix: fix template exception

* Revert "fix: fix template exception"

This reverts commit 746754a.

* chore: move IOException handling for request and response body

Co-authored-by: Mayur Gubrele <[email protected]>
Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: lavkesh <[email protected]>

* feat: add batch request (raystack#68)

Co-authored-by: lavkesh <[email protected]>

* Http batch request (raystack#70)

* feat: add batch request

* chore: minor refactor

Co-authored-by: jesrypandawa <[email protected]>

* feat: header builder generates headers from template (raystack#63)

* feat: header builder generates headers from template

* refactor: fix for the bug where baseheaders were getting updated with template ones

* add a maven repo in build.gradle

* feat: add messageContainer class and handle exceptions for headerBuilder flow

* refactor: add tests for headers in singleRequestTest

* refactor: resolve review comments

* refactor: minor refactoring as per review comments

* feat: add http query parameter builder (raystack#71)

* feat: add query param builder

* chore: add minor changes

* refactor: refactor builder class to cache schema, header,queryparam,and url

* feat: get sink_retryable_error for configured retry http response code (raystack#75)

* feat: get sink_retryable_error for configured retry http response code ranges

* refactor: changes based on review comments

* refactor: getMapping on ParsedMessage

* feat: add ProtoJsonProvider to work with JSONPath

* feat: use ProtoJsonProvider for getFieldByName on ProtoParsedMessage

* chore: remove unused imports

* feat: add config to log http request based on response code (raystack#74)

* feat: add config to log http request based on response code

* refactor: remove unused import

* refactor: fix error after merge

* refactor: refactor log request method name

* refactor: refactor http request for parsedMessage design changes

* refactor: refactor request body (#77)

* feat: add http templatized json body (#79)

* refactor: refactor request body

* feat: add templatized json body

* chore: explicit the exception in http sink factory, add more test for templatized json

* feat: moving to gotocompany

* chore: cleanup schema class references

* feat: implement getMapping using ParsedMessage interface


* wip: add proto implementation

* feat: add json implementation

* chore: add tests for schema implementation

* chore: add tests for generic json schema implementation

* chore: remoeve getMapping from parsedMessage and clean up test

* chore: add tests for byte and enum types in RecordConverter

* chore: clean up unused protoField and jsonfield class

* chore: add tests for getFields ProtoParsedMessage

* feat: include default fields in Proto message

* feat: add float point check for Double

* refactor: inline proto logical type parse

* refactor: ProtoLogicalValue constructor

---------

Co-authored-by: jesry.pandawa <[email protected]>

* feat: add toJson in parsedMessage and implementation (raystack#6)

* feat: add toJson in parsedMessage and implementation

* chore: fix config and unit test

* chore: fix checkstyle

* chore: fix test

* chore: remove preserve proto field names config

* feat: add http sink message payload type (#8)

* feat: add http sink json payload type (#7)

* feat: add http sink json payload type

* fix: http sink payload key cases

---------

Co-authored-by: Harikrishna <[email protected]>

* feat: add http delete method (#9)

* feat: add http delete method

* refactor: remove comments

---------

Co-authored-by: Harikrishna <[email protected]>

* chore: requestutils refactor

* refactor: http to httpv2 prefix (#20)

* refactor: http to httpv2 prefix

* refactor: http to httpv2 prefix

* refactor: http to httpv2 prefix

* refactor: http to httpv2 prefix

* refactor: http to httpv2 prefix

* refactor: http to httpv2 prefix

* refactor: http to httpv2 prefix

* fix: add SINK prefix in HttpClientConfigs (#17)

* fix: add SINK prefix in HttpClientConfigs

* fix add SINK prefix in HttpClientConfigs

* fix: add httpv2 prefix in http configs

* tests: fix tests

* tests: fix tests

* tests: fix tests

* fix: fix configs

* fix: fix checkstyle

* feat: http sink metrics (#14)

* feat: http sink metrics

* test: add http metrics tests

* chore: beta release version

* refactor: update exception message

* chore: version bump

* chore: version bump

* fix: remove simple date format (#23)

* fix: throw exception when template is used in batch mode for url , query and header

* fix: log request payload body in debug mode

* chore: fix build

* test: log request payload body in debug mode

* test: log request payload body in debug mode

* fix: add config for default value for empty fields (#25)

* fix: add config for default value for empty fields

* fix: add config for default value for empty fields

* refactor: rename variables which are not final anymore

* fix: default value behaviour for empty fields

* test: fix tests

* test: fix tests

* test: fix tests

* test: fix tests

* test: fix tests

* feat: add support for JSON array and primitives in Json body template (#40)

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* feat: add support for JSON array and primitives in Json body template

* tests: add unit tests for json parser

* refactor: removed extra checks

* fix: fix check style

* fix: fix check style

* fix: fix check style

* fix: fix check style

* fix: fix check style

* tests: fix unit tests

* tests: fix unit tests

* refactor: change implementation to use jackson library instead of gson

* refactor: change implementation to use jackson library instead of gso

* refactor: move json parsing logic to JsonParserUtils

* tests: add unit tests for JsonParserUtil

* fix: fix check style

* fix: fix check style

* fix: fix bugs and unit tests

* fix: fix check style

* fix: fix check style

* fix: fix check style

* fix: fix check style

* fix: fix check style

* fix: fix PR review comments

* docs: add java doc for utility methods

* test: add unit tests for JsonParserUtils

* test: add unit tests for JsonParserUtils

* test: add unit tests for JsonParserUtils

* test: add unit tests for JsonParserUtils

---------

Co-authored-by: lavkesh <[email protected]>

* docs: add docs for Httpv2 sink (#41)

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* docs: add docs for Httpv2 sink

* fix: remove messageschema get schema method

* fix: remove default values of oauth configs

* refactor: revert bq storage sink changes (#42)

* fix: fix http connection pool error (#43)

* fix: fix http connection pool error

* fix: fix build error

* fix: fix response code logic and consuming response body

---------

Co-authored-by: gagandhand <[email protected]>
Co-authored-by: gdgagan696 <[email protected]>

* docs: add docs for httpv2 sink metrics

* docs: add docs for httpv2 sink metrics

* fix: change default field value enable to true

---------

Co-authored-by: jesrypandawa <[email protected]>
Co-authored-by: lavkesh <[email protected]>
Co-authored-by: jesry.pandawa <[email protected]>
Co-authored-by: Mayur Gubrele <[email protected]>
Co-authored-by: mayur.gubrele <[email protected]>
Co-authored-by: jesrypandawa <[email protected]>
Co-authored-by: Harikrishna <[email protected]>
Co-authored-by: gagandhand <[email protected]>
Co-authored-by: gdgagan696 <[email protected]>
  • Loading branch information
@sumitaich1998 @jesrypandawa
@lavkesh @MayurGubrele @h4rikris @gdgagangeek @gdgagan696
10 people authored Apr 24, 2024
1 parent faba3ac commit fbc5791
Show file tree
Hide file tree
Showing 160 changed files with 7,458 additions and 1,940 deletions.
5 changes: 3 additions & 2 deletions build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ plugins {
}

group 'com.gotocompany'
version '0.8.2'
version '0.9.0'

repositories {
mavenLocal()
Expand Down Expand Up @@ -51,7 +51,8 @@ dependencies {
implementation 'com.fasterxml.jackson.core:jackson-databind:2.15.2'
implementation 'org.json:json:20230227'
implementation group: 'com.jayway.jsonpath', name: 'json-path', version: '2.8.0'

implementation group: 'com.squareup.okhttp3', name: 'okhttp', version: '4.9.1'
implementation group: 'joda-time', name: 'joda-time', version: '2.10.2'
testImplementation group: 'junit', name: 'junit', version: '4.13.1'
testImplementation 'org.hamcrest:hamcrest-all:1.3'
testImplementation 'org.mockito:mockito-core:4.5.1'
Expand Down
7 changes: 7 additions & 0 deletions docs/SUMMARY.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,20 @@
## Sink
* [Bigquery](sinks/bigquery.md)
* [Redis](sinks/redis.md)
* [Bigtable](sinks/bigtable.md)
* [HTTP](sinks/http-sink.md)


## Reference

* [Configuration](reference/configuration/README.md)
* [Generic](reference/configuration/generic.md)
* [Stencil Client](reference/configuration/stencil-client.md)
* [Bigquery Sink](reference/configuration/bigquery-sink.md)
* [Redis Sink](reference/configuration/redis.md)
* [Bigtable Sink](reference/configuration/bigtable.md)
* [HTTP Sink](reference/configuration/http-sink.md)

* [Metrics](reference/metrics.md)

## Contribute
Expand Down
1 change: 1 addition & 0 deletions docs/reference/configuration/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ This page contains reference for all the configurations for sink connectors.
* [Bigquery Sink](bigquery-sink.md)
* [Redis Sink](redis.md)
* [Bigtable Sink](bigtable.md)
* [HTTP Sink](http-sink.md)

22 changes: 22 additions & 0 deletions docs/reference/configuration/generic.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,28 @@ Allow unknown fields in proto schema
* Type: `required`
* Default: `false`

## `SINK_ADD_METADATA_ENABLED`

Defines whether to add Kafka metadata fields like topic, partition, offset, timestamp in the input proto messge.

* Example value: `true`
* Type: `optional`
* Default: `false`

## `SINK_METADATA_COLUMNS_TYPES`

Defines which Kafka metadata fields to add in the input proto message.
* Example value: `message_offset`
* Type: `optional`

## `SINK_DEFAULT_FIELD_VALUE_ENABLE`

Defines whether to send the default values in the request body for fields which are not present or null in the input Proto message

* Example value: `false`
* Type: `optional`
* Default value: `false`

## `METRIC_STATSD_HOST`

URL of the StatsD host
Expand Down
180 changes: 180 additions & 0 deletions docs/reference/configuration/http-sink.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
# HTTP Sink

An HTTP sink requires the following variables to be set along with Generic ones

## `SINK_HTTPV2_SERVICE_URL`


The HTTP endpoint of the service to which this consumer should PUT/POST/PATCH/DELETE data. This can be configured as per the requirement, a constant or a dynamic one \(which extract given field values from each message and use that as the endpoint\)
If service url is constant, messages will be sent as batches while in case of dynamic one each message will be sent as a separate request \(Since they’d be having different endpoints\).

- Example value: `http://http-service.test.io`
- Example value: `http://http-service.test.io/test-field/%s,order_number` This will take the value with field name `order_number` from proto and create the endpoint as per the template
- Type: `required`

## `SINK_HTTPV2_REQUEST_METHOD`

Defines the HTTP verb supported by the endpoint, Supports PUT, POST, PATCH and DELETE verbs as of now.

- Example value: `post`
- Type: `required`
- Default value: `put`

## `SINK_HTTPV2_HEADERS`

Defines the HTTP headers required to push the data to the above URL.

- Example value: `Authorization:auth_token, Accept:text/plain`
- Type: `optional`

## `SINK_HTTPV2_HEADERS_TEMPLATE`

Defines a template for creating custom headers from the fields of a protobuf message. This should be a valid JSON itself.

* Example value: `{"%s,order_id":"Order"}`
* Type: `optional`

## `SINK_HTTPV2_HEADERS_PARAMETER_SOURCE`

Defines the source from which the fields should be parsed. This field should be present in order to use this feature.

- Example value: `Key`
- Type: `optional`
- Default value: `MESSAGE`

## `SINK_HTTPV2_QUERY_TEMPLATE`

Defines a template for creating a custom query from the fields of a protobuf message. This should be a valid JSON itself.

* Example value: `{"%s,order_id":"Order"}`
* Type: `optional`

## `SINK_HTTPV2_QUERY_PARAMETER_SOURCE`

Defines the source from which the fields should be parsed. This field should be present in order to use this feature.

- Example value: `Key`
- Type: `optional`
- Default value: `Message`

## `SINK_HTTPV2_REQUEST_MODE`

Defines Single or Batch request mode which corresponds to sending a single message payload or sending a batch payload of messages in the body of each http request.

* Example value: `BATCH`
* Type: `optional`
* Default value: `SINGLE`


## `SINK_HTTPV2_REQUEST_BODY_MODE`

Defines the type of format of the request body in the payload. This can be either of the following types-
* `RAW` - The raw Protobuf/ JSON encoded byte string containing both key and message
* `TEMPLATIZED_JSON` - JSON payload constructed by the provided JSON body template
* `JSON` - JSON payload containing all the fields of the Proto/JSON message
* `MESSAGE` - Only the Protobuf/ JSON encoded Message , excluding the key


* Example value: `TEMPLATIZED_JSON`
* Type: `optional`
* Default value: `RAW`


## `SINK_HTTPV2_REQUEST_LOG_STATUS_CODE_RANGES`

Defines the range of HTTP status codes for which request body will be logged.

- Example value: `400-600`
- Type: `optional`
- Default value: `400-600`


## `SINK_HTTPV2_RETRY_STATUS_CODE_RANGES`

Defines the range of HTTP status codes for which retry will be attempted. Please remove 404 from retry code range in case of HTTP DELETE otherwise it might try to retry to delete already deleted resources.

- Example value: `400-600`
- Type: `optional`
- Default value: `400-600`

## `SINK_HTTPV2_JSON_BODY_TEMPLATE`


Defines a template for creating a custom request body from the fields of a protobuf message. This should be a valid JSON itself.

- Example value: `{"test":"%s,routes[0]", "%s,order_number" : "xxx"}`
- Type: `optional`

## `SINK_HTTPV2_DEFAULT_FIELD_VALUE_ENABLE`

Defines whether to send the default values in the request body for fields which are not present or null in the input Proto message

* Example value: `false`
* Type: `optional`
* Default value: `true`

## `SINK_HTTPV2_DELETE_BODY_ENABLE`

This config if set to true will allow body for the HTTP DELETE method, otherwise no payload will be sent with DELETE request.

- Example value: `false`
- Type: `optional`
- Default value: `true`


## `SINK_HTTPV2_MAX_CONNECTIONS`

Defines the maximum number of HTTP connections.

- Example value: `10`
- Type: `required`
- Default value: `10`

## `SINK_HTTPV2_REQUEST_TIMEOUT_MS`

Defines the connection timeout for the request in millis.

- Example value: `10000`
- Type: `required`
- Default value: `10000`

## `SINK_HTTPV2_OAUTH2_ENABLE`

Enable/Disable OAuth2 support for HTTP sink.

- Example value: `true`
- Type: `optional`
- Default value: `false`

## `SINK_HTTPV2_OAUTH2_ACCESS_TOKEN_URL`

Defines the OAuth2 Token Endpoint.

- Example value: `https://sample-oauth.my-api.com/oauth2/token`
- Type: `optional`

## `SINK_HTTPV2_OAUTH2_CLIENT_NAME`


Defines the OAuth2 identifier issued to the client.

- Example value: `client-name`
- Type: `optional`


## `SINK_HTTPV2_OAUTH2_CLIENT_SECRET`


Defines the OAuth2 secret issued for the client.

- Example value: `client-secret`
- Type: `optional`


## `SINK_HTTPV2_OAUTH2_SCOPE`


Space-delimited scope overrides. If scope override is not provided, no scopes will be granted to the token.

- Example value: `User:read, sys:info`
- Type: `optional`
8 changes: 8 additions & 0 deletions docs/reference/configuration/redis.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,14 @@ The Redis deployment you are using. At present, we support `Standalone` and `Clu
- Type: `required`
- Default value: `Standalone`

### `SINK_REDIS_DEFAULT_FIELD_VALUE_ENABLE`

Defines whether to send the default values for fields which are not present or null in the input Proto message

* Example value: `false`
* Type: `optional`
* Default value: `true`

### `SINK_REDIS_SOCKET_TIMEOUT_MS`

The max time in milliseconds that the Redis client will wait for response from the Redis server.
Expand Down
6 changes: 6 additions & 0 deletions docs/reference/metrics.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Sinks can have their own metrics, and they will be emmited while using sink conn
* [Bigquery Sink](metrics.md#bigquery-sink)
* [Bigtable Sink](metrics.md#bigtable-sink)
* [Redis Sink](metrics.md#redis-sink)
* [Http Sink](metrics.md#http-sink)


## Bigquery Sink
Expand Down Expand Up @@ -52,3 +53,8 @@ Total number of records which could not be pushed to the Redis server due to bro

Total number of attempts to recreate the connection to Redis server from the Jedis client.

## Http Sink

### `Http Response Code Total`

Total count of each response code, i.e the total number of Kafka records under each response code. So in case of batch request mode, each message in the batch will be counted individually.
Loading

0 comments on commit fbc5791

Please sign in to comment.