Add ack and fail to Streamlet API (#2909) #3217
Conversation
Update the streamlet API to allow ack/fail for messages.
|
Is there a way to verify the behavior with unit tests? |
|
I believe I will have to find a way to test via integration tests in order to verify programmatically. Even with integration tests I'm not sure how to test tuple failures. I'm open to ideas. While running my heron-examples there are situation where tuples fail and are re-emitted successfully. I'm just not sure how to force that behaviour in testing. |
|
Most operators don't have "failure" case (for example filter, flatmap). Only ack I remember. The only way to trigger fail() I can think of now is to write a simple bolt that has failure cases and then use the applyOperator() function. I was thinking of catching exceptions as failures but it might have some unwanted side effects so it is not decided yet. |
Added unit tests for SupplierSource and ComplexSource. Added two helper classes for the the tests. Updated several other classes to use clearer variable names. Also updated location of ackingEnabled check and cache creation.
This PR updates Heron to allow for ack and fail behaviour with the Streamlet API.
Would request feedback, especially as to the caching of id's and tuples.
These updates have been tested against a collection of streamlet examples. The repo located at https://github.com/jmark99/heron-examples contains a collection of many streamlet examples. The modifications have been run successfully against this collection. The README file for that repo describes the steps taken to exercise the changes.
This commit is intended to fix #2909