New serverless pattern - kinesis-lambda-error-handling #1638
Conversation
… Kinesis Data Stream anomalies
|
@marakere your 'example-pattern.json' is missing some key fields, please review below and address any errors you have
If you need any help, take a look at the example-pattern file. Make the changes, and push your changes back to this pull request. When all automated checks are successful, the Serverless DA team will process your pull request. |
|
Replaced the value from framework from "AWS SAM" to "SAM". |
| Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing. | ||
|
|
||
| ## How it works | ||
|  |
There was a problem hiding this comment.
image not loading in the readme
| We highly recommend to use [Amazon Kinesis Data Generator(KDG)](https://aws.amazon.com/blogs/big-data/test-your-streaming-data-solution-with-the-new-amazon-kinesis-data-generator/) for the high volume testing. The KDG makes it simple to send test data to your Amazon Kinesis stream or Amazon Kinesis Firehose delivery stream. | ||
|
|
||
| ## Cleanup | ||
| sam delete |
| ./kinesis-producer/without-poison-pill-put-records.sh | ||
| ``` | ||
|
|
||
| - Navigate to AWS Console, and then to Cloudwatch, Log groups, select the log group and the latest Log stream |
There was a problem hiding this comment.
what about using sam logs to do this in the CLI? As these instructions are not quite clear of what we should be looking at.
| ./kinesis-producer/with-poison-pill-put-records.sh | ||
| ``` | ||
|
|
||
| - Cloudwatch logs shows that there was one invalid or poison message, so here the bisectonfailure was applied which lead Lambda to split the batch in half and resume each half separately. Maximum retry attempts and maximum record age limit the number of retries on a failed batch. As the retry limit is 5, it will retry 5 times, before the message is put to AWS SQS. |
There was a problem hiding this comment.
same here, use sam logs to get the logs
|
Few small comments, after those are fixed then i can merge it |
|
Thanks for the review comments. |
|
Thanks for submitting your pattern. https://serverlessland.com/patterns/kinesis-lambda-error-handling You can share your pattern using this link :) |
Effective consumer strategies for handling Kinesis Data Stream anomalies
The purpose of this pattern is to showcase on how to handle the consumer(AWS Lambda) failure when reading/processing the records from the Amazon Kinesis data stream.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.