Amazon Marketing Stream reference implementation using AWS CDK

This project contains an example implementation and infrastructure code to both destinations SQS as well as firehose:

Provisions necessary AWS infrastructure to receive and store Amazon Marketing Stream data, as well as confirm Stream dataset subscriptions for SQS. The Stream subscription process has been simplified now, with Kinesis Data Firehose, you no longer need to confirm your subscription.
Subscribe to datasets and manage subscriptions using a CLI.

Disclaimer

This is a reference implementation, and not the only definitive way to consume Amazon Marketing Stream data. Note that this implementation is subject to change and future releases may not be backwards compatible.

SQS Solution architecture

Firehose Solution architecture

This application, developed using Python and the AWS Cloud Development Kit (CDK), supports two deployment options: one for SQS and another for the new destination Firehose. However, SQS is the default deployment.

SQS deployment

The SQS application deployment provisions the following AWS infrastructure components for each dataset and region combination:

An SQS queue (StreamIngressQueue) that receives initial messages from Stream.
A lambda (StreamFanoutLambda) that identifies whether a message contains subscription details or data.
A second SQS queue (SubscriptionConfirmationQueue) that forwards subscription confirmation messages to a second lambda (SubscriptionConfirmationLambda) that confirms the subscription.
An SNS topic (StreamFanoutDataTopic that) forwards data through a KinesisDateFirehouse (StreamStorageFirehose) to an S3 bucket (StreamStorageBucket) where the data is stored.

Note: The provisioning of each SQS queue also includes an associated dead-letter queue.

Firehose deployment

The Firehose application deployment provisions the following AWS infrastructure components for each dataset and region combination:

A KinesisDateFirehouse (StreamStorageFirehose).
An S3 bucket (StreamStorageBucket) where the data is stored.
A [IAM Roles] (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) (FirehoseSubscriberRole) and (FirehoseSubscriptionRole) so that amazon marketing stream can send the messages to KinesisDateFirehouse.

Video tutorial For SQS Destination

Learn how to set up the reference application by watching our video demo.

Supported datasets

This package includes following templates for all available datasets for advertising regions NA, EU, and FE. All NA stacks will be deployed in AWS region us-east-1, EU stacks will be deployed in AWS region eu-west-1, and FE stacks will be deployed in AWS region us-west-2 to minimize latency of message delivery. For more information on datasets, see the Stream data guide.

AmzStream-NA-sp-traffic
AmzStream-NA-sp-conversion
AmzStream-NA-budget-usage
AmzStream-NA-sd-traffic
AmzStream-NA-sd-conversion
AmzStream-NA-sb-traffic
AmzStream-NA-sb-conversion
AmzStream-NA-sb-clickstream
AmzStream-NA-sb-rich-media
AmzStream-NA-campaigns
AmzStream-NA-adgroups
AmzStream-NA-ads
AmzStream-NA-targets
AmzStream-NA-sponsored-ads-campaign-diagnostics-recommendations
AmzStream-NA-sp-budget-recommendations
AmzStream-EU-sp-traffic
AmzStream-EU-sp-conversion
AmzStream-EU-budget-usage
AmzStream-EU-sd-traffic
AmzStream-EU-sd-conversion
AmzStream-EU-sb-traffic
AmzStream-EU-sb-conversion
AmzStream-EU-sb-clickstream
AmzStream-EU-sb-rich-media
AmzStream-EU-campaigns
AmzStream-EU-adgroups
AmzStream-EU-ads
AmzStream-EU-targets
AmzStream-EU-sponsored-ads-campaign-diagnostics-recommendations
AmzStream-EU-sp-budget-recommendations
AmzStream-FE-sp-traffic
AmzStream-FE-sp-conversion
AmzStream-FE-budget-usage
AmzStream-FE-sd-traffic
AmzStream-FE-sd-conversion
AmzStream-FE-sb-traffic
AmzStream-FE-sb-conversion
AmzStream-FE-sb-clickstream
AmzStream-FE-sb-rich-media
AmzStream-FE-campaigns
AmzStream-FE-adgroups
AmzStream-FE-ads
AmzStream-FE-targets
AmzStream-FE-sponsored-ads-campaign-diagnostics-recommendations
AmzStream-FE-sp-budget-recommendations

Development prerequisites

AWS account
AWS Cloud Development Kit (CDK)
Python 3.7 or later including pip and virtualenv

We recommend exploring the contents of this project and familiarizing yourself with the AWS infrastructure before deploying.

Deployment steps

Initialize your project and activate a virtualenv. The cdk.json file tells the CDK Toolkit how to execute your app. This project is set up like a standard Python project. The initialization process creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv, it assumes that there is a python3 executable in your path with access to the venv package. If the automatic creation of the virtualenv fails, you can always create the virtualenv manually once the init process completes.

Manually create a virtualenv on MacOS and Linux

Creating a virtual environment
```
python3 -m venv .venv
```
After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.
```
source .venv/bin/activate
```
Manually create a virtualenv on Windows
```
% .venv\Scripts\activate.bat
```
Install the required dependencies.
```
pip install -r requirements.txt
```
Synthesize the CloudFormation templates for this code.
```
cdk synth
```
To view the CloudFormation templates created by the synthesize step.
```
cdk ls
```

Deploy CloudFormation templates.

Depending on your requirements, you can choose to deploy all CloudFormation templates

SQS -

cdk deploy --all

Firehose -

cdk deploy --all --context delivery_type=firehose

or individual templates

SQS -

cdk deploy AmzStream-NA-sp-traffic

Firehose -

cdk deploy AmzStream-NA-sp-traffic --context delivery_type=firehose

At the end of deployment, your output should resemble:

SQS Outputs:
AmzStream-NA-sp-traffic.IngressIngressQueue91B67342 = arn:aws:sqs:us-east-1:2xxxxxxxxxxx:AmzStream-NA-sp-traffic-IngressQueue26236266-Jvxxxxxxxxxx
AmzStream-NA-sp-traffic.StorageLandingZoneBucketFE2101CB = arn:aws:s3:::amzstream-na-sp-traffic-storagelz10f6c360-1hxxxxxxxxxxx
Stack ARN:
arn:aws:cloudformation:us-east-1:2xxxxxxxxxxx:stack/AmzStream-NA-sp-traffic/57151cc0-b625-11ed-a641-12730e200e31

Firehose Outputs:
AmzStream-NA-sb-traffic.FirehoseSubscriberRoleInfraArn3358A9BC = arn:aws:iam::88xxxxxxxx:role/sb-traff-NA-subscriber
AmzStream-NA-sb-traffic.FirehoseSubscriptionRoleInfraArn22E3DAAD = arn:aws:iam::88xxxxxxxx:role/sb-traff-NA-subscription
AmzStream-NA-sb-traffic.StorageDeliveryStreamArnBB2306AC = arn:aws:firehose:us-east-1:88xxxxxxxx:deliverystream/AmzStream-NA-sb-traffic-StorageFirehoseEEA939E5-rDi6fvblOARo
AmzStream-NA-sb-traffic.StorageLandingZoneBucketFE2101CB = arn:aws:s3:::amzstream-na-sb-traffic-storagelz10f6c360-4xbrfa1i3lxh
Stack ARN:
arn:aws:cloudformation:us-east-1:88xxxxxxxx:stack/AmzStream-NA-sb-traffic/e63f06f0-0790-11ef-b509-127247d88f9b

Note:

This example uses AmzStream-NA-sp-traffic as an example.
AmzStream-NA-sp-traffic.IngressIngressQueue91B67342 is the name of the example queue that will receive messages for dataset sp-traffic from NA region.
arn:aws:sqs:us-east-1:2xxxxxxxxxxx:AmzStream-NA-sp-traffic-IngressQueue26236266-Jvxxxxxxxxxx is the ARN of the example queue and should be used for field destinationArn while calling the subscription API as listed in the subscription step of the onboarding guide.
AmzStream-NA-sp-traffic.StorageLandingZoneBucketFE2101CB is the name of the example S3 bucket that will store all the received messages for this dataset.

Useful CDK commands

cdk ls Lists all stacks in the app
cdk synth Emits the synthesized CloudFormation template
cdk bootstrap Deploys the CDK toolkit stack into an AWS environment
cdk deploy Deploys this stack to your default AWS account/region
cdk diff Compares deployed stack with current state
cdk docs Opens CDK documentation
cdk destroy Destroys this stack to your default AWS account/region

Using the Stream subscription management command line tool

We provide a Stream subscription management command line tool that supports following commands:

Create - Creates an Amazon Marketing Stream subscription.
Get - Gets information on a Amazon Marketing Stream subscription by ID.
List - Lists all Amazon Marketing Stream subscriptions associated with your Amazon Advertising API profile.
Update - Updates an Amazon Marketing Stream subscription by ID.

In order to use the CLI, you must create a credentials.yml file with your Amazon Ads API credentials. If you don't have credentials for the Ads API, review the Onboarding process.

Search path for credentials.yml

macOS and Other Unix: ~/.config/python-ad-api
Windows: %APPDATA%\python-ad-api where the APPDATA environment variable falls back to %HOME%\AppData\Roaming if undefined

For more information, see Python Confuse module help.

Example: ~/.config/python-ad-api/credentials.yml

version: '1.0'

default:
  refresh_token: 'your-refresh-token'
  client_id: 'your-client-id'
  client_secret: 'your-client-secret'
  profile_id: 'your-profile-id'

Querying and aggregating Stream data

Once you start receiving Stream data in AWS, you can learn more about aggregating and querying Stream data in our documentation.

Querying Stream data
Aggregating and joining traffic and conversion Stream data

Using the CLI

You can view instructions for using the CLI using python -m amz_stream_cli --help.

Example:

% python -m amz_stream_cli --help

                                                                                                                                       
 Usage: amz_stream_cli [OPTIONS] COMMAND [ARGS]...                                                                                     
                                                                                                                                       
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --version             -v        Show the application's version and exit.                                           │
│ --install-completion            Install completion for the current shell.                                          │
│ --show-completion               Show completion for the current shell, to copy it or customize the installation.   │
│ --help                          Show this message and exit.                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ create    Creates Amazon Marketing Stream subscription.                                                            │
│ get       Gets information on specific Amazon Marketing Stream subscription by ID.                                 │
│ list      Lists all Amazon Marketing Stream subscriptions associated with your Amazon Advertising API account.     │
│ update    Updates specific Amazon Marketing Stream subscription by ID.                                             │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

For help on individual commands, use the following:

python -m amz_stream_cli create --help
python -m amz_stream_cli get --help
python -m amz_stream_cli list --help
python -m amz_stream_cli update --help

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Amazon Marketing Stream reference implementation using AWS CDK

Disclaimer

SQS Solution architecture

Firehose Solution architecture

SQS deployment

Firehose deployment

Video tutorial For SQS Destination

Supported datasets

Development prerequisites

Deployment steps

Useful CDK commands

Using the Stream subscription management command line tool

Search path for credentials.yml

Querying and aggregating Stream data

Using the CLI

Files

README.md

Latest commit

History

README.md

File metadata and controls

Amazon Marketing Stream reference implementation using AWS CDK

Disclaimer

SQS Solution architecture

Firehose Solution architecture

SQS deployment

Firehose deployment

Video tutorial For SQS Destination

Supported datasets

Development prerequisites

Deployment steps

Useful CDK commands

Using the Stream subscription management command line tool

Search path for credentials.yml

Querying and aggregating Stream data

Using the CLI