Back End Notes

Dynamo db

sudo yum install git
git clone https://github.com/CUBigDataClass/tweet_spread.git
sudo pip install boto3
aws configure

Kafka

Full tutorial is here.

Install zookeeper and kafka

wget http://mirrors.gigenet.com/apache/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz
tar -zxf zookeeper-3.4.11.tar.gz
wget http://www.gtlib.gatech.edu/pub/apache/kafka/1.0.1/kafka_2.11-1.0.1.tgz
tar -zxf kafka_2.11-1.0.1.tgz
sudo pip install kafka-python

Run and stop zookeeper

bin/zkServer.sh start
bin/zkCli.sh
bin/zkCli.sh stop

Run and stop kafka

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Storm Topology Deployment

Command: bin/storm jar jarName.jar [TopologyMainClass] [Args]

~/storm/bin/storm jar target/Kafka_twitter_topology-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.stormadvance.Kafka_twitter_topology.topology.StormHDFSTopology StormHDFSTopology1