Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bumping codec limit to 20mb #258

Merged
merged 2 commits into from
Nov 25, 2024
Merged

Conversation

cbb330
Copy link
Collaborator

@cbb330 cbb330 commented Nov 25, 2024

Summary

this is a permanent solution (in the sense that, we don't risk this happening again in the next few months) to resolve

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 10240000

which can occur on large listing APIs such as /v1/databases which resolves up to 18.5k tables at most which exceeds client side http request of 10MB

this will continue to grow higher (projecting to about 37k tables without any new dimensions added)

and as stated below, the only long term fix is pagination in list apis.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

will monitor the acceptance tests

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

@cbb330 cbb330 merged commit 3f041c5 into linkedin:main Nov 25, 2024
1 check passed
@sumedhsakdeo
Copy link
Collaborator

List requests need to be paginated. That's the right way to fix this. Fine with this as short term mitigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants