Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enforce request maximum size and number of logs #2032

Closed
wants to merge 2 commits into from

Conversation

mrsillydog
Copy link
Contributor

Proposed Change

Google SecOps (Chronicle) limits API log ingestion in two different ways - first, in uncompressed request size, and secondly in number of logs per request. This PR changes the implementation of the chronicle exporter to respect these limits by checking against them before sending, and then splitting up the batched request logging data if either limit is exceeded. These limits are configurable on the SecOps backend, and therefore now are also configurable in the exporter.

Possible concerns:

  • Is proto.Size() an accurate measurement for the HTTP protocol workflow? It was confirmed as accurate for the GRPC protocol workflow.
  • What is a reasonable default size limit for HTTP and GRPC? The documentation lists it as 1MB for the GRPC endpoint, undocumented for the HTTP endpoint. Manual testing confirmed our configured limit is 5 MiB via GRPC, but substantially higher (unclear exactly what the limit is) via the v1alpha HTTP endpoint.
Checklist
  • Changes are tested
  • CI has passed

@mrsillydog mrsillydog requested review from dpaasman00 and a team as code owners December 9, 2024 18:10
@mrsillydog mrsillydog closed this Dec 9, 2024
@mrsillydog mrsillydog deleted the feat/enforce-request-maximums branch December 9, 2024 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant