Eventing Trigger - dispatcher rate limiter

[lamluongbinh]

Hi Team,

I am currently working on applying a rate limiter to control the rate of messages process by a Trigger.

Based on my tests, this can be achieved by:

1. Enabling the feature flag: dispatcher-rate-limiter: enabled.
2. Creating a new broker with 1 partition.
3. Adding an annotation to the Trigger to ensure it is ordered: kafka.eventing.knative.dev/delivery.order: ordered.
4. The limit is determined by the following equation: LimitedNumber = NumberOfPartitions * 20

I would like to ask if there is another way to control the LimitedNumber.

From my reading of the code, the bucket is calculated a: link

bucket = vReplica * (MaxPollRecords or NumberOfPartitions)

Alternatively, is there a way to apply a configuration or annotation to set the limit number for a Trigger?

[pierDipi]

Hi @lamluongbinh

I would like to ask if there is another way to control the LimitedNumber.

From my reading of the code, the bucket is calculated a: link

bucket = vReplica * (MaxPollRecords or NumberOfPartitions)

Alternatively, is there a way to apply a configuration or annotation to set the limit number for a Trigger?

currently, there is not, however, I'm wondering if the way is configurable is working for you

Based on my tests, this can be achieved by:

1. Enabling the feature flag: dispatcher-rate-limiter: enabled.
2. Creating a new broker with 1 partition.
3. Adding an annotation to the Trigger to ensure it is ordered: kafka.eventing.knative.dev/delivery.order: ordered.
4. The limit is determined by the following equation: LimitedNumber = NumberOfPartitions * 20

from where did you get that 20 ? I think you should see 1 event per second in this case

[lamluongbinh]

Hi @pierDipi,

Thanks for your reply.

The number is showing in my test, but I'm not sure where the value of 20 comes from. Could it be the default worker pool size for Vert.x (20 threads)? Initially, I tested with 1 partition, and then increased it to 3 partitions. The behavior is as follows:

1. With 1 partition: The rate limit is set to 20 messages per second.
2. After increasing to 3 partitions (without sending new messages): No messages are allocated to partitions 2 and 3. As a result, the rate limit remains at 20 messages per second.
3. When new messages are sent: They start filling partitions 2 and 3, causing the rate limit to increase to 60 messages per second as more partitions contribute to processing.
4. Once partitions 2 and 3 are processed (and no new messages arrive): The rate limit drops back to 20 messages per second, aligning with the load on the active partitions.

[pierDipi]

which version are you using?