Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

message batching issues #16

Open
eapugo opened this issue Nov 15, 2018 · 0 comments
Open

message batching issues #16

eapugo opened this issue Nov 15, 2018 · 0 comments

Comments

@eapugo
Copy link

eapugo commented Nov 15, 2018

It seems in the docs you have listed an option for high volume by increasing the message_count_threshold.

High Volume
If you find that uploads are going too slowly, you can increase the message batching:

output {
google_pubsub {
project_id => "my_project"
topic => "my_topic"
json_key_file => "service_account_key.json"

# Options for configuring the upload
message_count_threshold => 10000
delay_threshold_secs => 10
request_byte_threshold => 5MB

}
}

Yet, in the description of message_count_threshold you declare that limit must be below 1000. It doesn't operate over 1000 message count
: INVALID_ARGUMENT: The value for 1107 is too large. You passed message_count in the request, but the maximum value is 1000."

The output engine is much slower than expected even with 1000

Your feedback would be much appreciated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant