connor.fun-Custodian

This is the repo that contains the files and instructions to get started in your moody weather journey!

Requirements

+'d files are generated by the ansible script

*'d files are supplied by you

Custodian
│
├── Create.yaml
├── Prep-AWS.yaml
├── Create-Custodian.yaml
├── Create-Workers.yaml
│
├── Remove.yaml
├── Clean_AWS.yaml
├── Remove-Cluster.yaml
│
├── hosts
├── LICENSE
├── README.md
│
├── configs
│   ├── environment
│   ├── environmentmd
│   ├── librdkafka_install.sh
│   ├── sample.properties
│   ├── server.properties
│   ├── storm-env.sh
│   ├── storm-nimbus.conf
│   ├── storm-supervisors.conf
│   └── zoo.cfg
│
├─+ logs
│ + ├── AMI_generation.yaml
│ + ├── Base_AMI_search.yaml
│ + ├── Custodian_generation.yaml
│ + ├── Security_Group_generation.yaml
│ + ├── SSHKey_generation.yaml
│ + └── Worker_generation.yaml
│
├── services
│   ├── kafka-broker.service
│   ├── storm-logviewer.service
│   ├── storm-nimbus.service
│   ├── storm-supervisor.service
│   ├── storm-ui.service
│   ├── streamproducer.service
│   └── zookeeper.service
│
├── templates
│   ├── environment.j2
│   ├── environmentmd.j2
│   ├── sample.properties.j2
│   ├── server.properties.j2
│   ├── storm-nimbus.conf.j2
│   └── storm-supervisors.conf.j2
│
└── vars
  * ├── config.yaml
  * ├── keys.yaml
    ├── machines.yaml
    └── security_groups.yaml

The contents of keys.yaml should look like this:

weather_auth:
  key: "Secret"

twitter_consumer:
  key: "Extra"
  secret: "Secret"

twitter_access:
  key: "Mostest"
  secret: "Secret"

aws_auth:
  access: "Ultra"
  secret: "Secret"

The contents of config.yaml should look like this:

region: someAWSregion
ssh_key_name: connor.fun
base_ami_name: some-ami-name-in-amzn-ec2-ami-store

If you wish to, you may tweak the vars/machines.yaml and vars/security_groups.yaml to experiment with more workers or larger machines

Usage

To complete spin up a cluster use the command:

ansible-playbook -i hosts Create.yaml --extra-vars "doPrep=op doCustodian=op doWorkers=op"

where op is either true or false.

For example, if you have already prepped AWS then you can say:

ansible-playbook -i hosts Create.yaml --extra-vars "doPrep=false doCustodian=true doWorkers=true"

Other Notes

1. If you want to add more workers, you must take down all current workers and create a new set of workers with the script due to a Kafka worker_id dependency. (These actions can be done at the same time)

2. rsync is a dependency.