Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Batching EventHorizon requests #773

Open
mhelleborg opened this issue Aug 21, 2024 · 0 comments
Open

Support Batching EventHorizon requests #773

mhelleborg opened this issue Aug 21, 2024 · 0 comments
Assignees

Comments

@mhelleborg
Copy link
Member

Description

In order to support higher throughput for EventHorizon, it can be rewritten to support batching.

This requires updating the contract to support it, and leveraging them in the consumer & producer infrastructure in the runtime.

Design requirements

Backwards compatibility

New versions of the runtime should be able to communicate as normal with older ones and vice versa. This can be solved by adding optional fields to the subscription request, which will only be recognized by runtimes running the updated version. Thus it can select the batch mode if the consumer asks for it, and publish the old message types if not.

Resource limits

Since we are making the producer a lot faster with sending batches, the consumer will need to be able to limit work in progress. This can be done by specifying batch sizes & using bounded channels to prevent a slow DB from causing OOM-issues.

@mhelleborg mhelleborg self-assigned this Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant