-
Notifications
You must be signed in to change notification settings - Fork 4
Setup Zookeeper on cluster
We will setup Kafka on cluster of 4 nodes. Maybe you heard about that Kafka must be implemented in cluster of odd nodes however there is no problem if you implement it in even nodes because:
For the ZooKeeper service to be active, there must be a majority of non-failing machines that can communicate with each other. To create a deployment that can tolerate the failure of F machines, you should count on deploying 2xF+1 machines. Thus, a deployment that consists of three machines can handle one failure, and a deployment of five machines can handle two failures. Note that a deployment of six machines can only handle two failures since three machines is not a majority. For this reason, ZooKeeper deployments are usually made up of an odd number of machines.
- Java 8 must be installed.
JAVA_HOMEmust be set correctly. - We assume that we have 4 clusters which theirs name is:
master,slave-1,slave-2andslave-3.
First download Zookeeper v3.5.5 in all of your clusters. put extracted zip file inside a common folder in all your servers:
$ wget https://www-us.apache.org/dist/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
$ tar -xvf apache-zookeeper-3.5.5-bin.tar.gz
$ mv apache-zookeeper-3.5.5-bin /opt/zookeeper
$ rm apache-zookeeper-3.5.5-bin.tar.gz
Before you configure your zookeeper, you must choose a folder as dataDir for keep information of Zookeeper in each node. We choose /opt/zookeeper/data as dataDir folder. Each node must have a unique id which is used to reference them inside zookeeper. This unique id must be choose for each node and written inside a myid file inside zookeeper data folder. We choose ids of 0, 1, 2 and 3 for our nodes:
root@master echo "0" > /opt/zookeeper/data/myid
root@slave-1 echo "1" > /opt/zookeeper/data/myid
root@slave-2 echo "2" > /opt/zookeeper/data/myid
root@slave-3 echo "3" > /opt/zookeeper/data/myid
After that, we must configure zookeeper. There is a sample zoo_sample.cfg inside /opt/zookeeper/conf folder. create a copy from that and do this configuration based on id you set before:
cp /opt/zookeeper/conf/zoo_sample.cfg /opt/zookeeper/conf/zoo.cfg
nano /opt/zookeeper/conf/zoo.cfg
Configure zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=200
server.0=master:2888:3888 # or your host IP (use corresponding id)
server.1=slave-1:2888:3888 # or your host IP (use corresponding id)
server.2=slave-2:2888:3888 # or your host IP (use corresponding id)
server.3=slave-3:2888:3888 # or your host IP (use corresponding id)Note: Do not change :2888:3888 ports
To start Zookeeper, use this command:
/opt/zookeeper/bin/zkServer.sh start
Use jps command to know when it is started. It's process will be seen as QuorumPeerMain process.
If it is failed to start, look at logs to understand what is problem. it's probably because of incorrect configuration.
To restart Zookeeper, you must stop it and after stopped successfully, start it again.
/opt/zookeeper/bin/zkServer.sh stop
Use jps command to know when it is stopped.
Use command below to see status Zookeeper cluster.
/opt/zookeeper/bin/zkServer.sh status
In a cluster, One node will be leader and other nodes will be follower. You must see something like this when you run this command:
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower