Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enforce request maximum size and number of logs #2033

Merged
merged 4 commits into from
Dec 11, 2024

Conversation

mrsillydog
Copy link
Contributor

@mrsillydog mrsillydog commented Dec 9, 2024

Proposed Change
Google SecOps (Chronicle) limits API log ingestion in two different ways - first, in uncompressed request size, and secondly in number of logs per request. This PR changes the implementation of the chronicle exporter to respect these limits by checking against them before sending, and then splitting up the batched request logging data if either limit is exceeded. These limits are configurable on the SecOps backend, and therefore now are also configurable in the exporter.

Possible concerns:

  • Is proto.Size() an accurate measurement for the HTTP protocol workflow? It was confirmed as accurate for the GRPC protocol workflow.
  • What is a reasonable default size limit for HTTP and GRPC? The documentation lists it as 1MB for the GRPC endpoint, undocumented for the HTTP endpoint. Manual testing confirmed our configured limit is 5 MiB via GRPC, but substantially higher (unclear exactly what the limit is) via the v1alpha HTTP endpoint.
Checklist
  • Changes are tested
  • CI has passed

@mrsillydog mrsillydog requested review from dpaasman00 and a team as code owners December 9, 2024 18:15
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation requests and questions but that's it. Looking good.

exporter/chronicleexporter/config.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Show resolved Hide resolved
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mrsillydog mrsillydog merged commit 24d60e5 into release/v1.67.0 Dec 11, 2024
15 checks passed
@mrsillydog mrsillydog deleted the feat/enforce-request-maximum branch December 11, 2024 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants