Hadoop学习之路(9)ZooKeeper安装
程序员文章站
2022-05-07 09:33:25
本文详细的介绍了Linux环境下的ZooKeeper安装步骤 ......
1、环境准备
1.1下载zookeeper
查阅hadoop2.7.3的文档我们可以看到hadoop2.7.3在搭建高可用的时候使用的是zookeeper-3.4.2版本，所以我们也按照hadoop官网的提示，接下来我们安装zookeeper-3.4.2版本.进入官网下载zookeeper3.4.2版本
官网地址：https://zookeeper.apache.org/
点击download
1.3安装zookeeper
#1.把zookeeper的压缩安装包解压到/opt/bigdata/目录下 [root@node1 ~]# tar -xzvf zookeeper-3.4.2.tar.gz -c /opt/bigdata/ #输入完命令后回车 #2.切换到bigdata目录下 [root@node1 ~]# cd /opt/bigdata/ #3.按照安装hadoop的方式，将zookeeper的安装目录的所属组修改为hadoop:hadoop #修改zookeeper安装目录的所属用户和组为hadoop:hadoop [root@node1 bigdata]# chown -r hadoop:hadoop zookeeper-3.4.2/ #4.修改zookeeper安装目录的读写权限 [root@node1 bigdata]# chmod -r 755 zookeeper-3.4.2/
1.4配置zookeeper环境变量
#1.切换到hadoop用户目录下 [root@node1 bigdata]# su - hadoop last login: thu jul 18 16:07:39 cst 2019 on pts/0 [hadoop@node1 ~]$ cd /opt/bigdata/zookeeper-3.4.2/ [hadoop@node1 zookeeper-3.4.2]$ cd .. [hadoop@node1 bigdata]$ cd ~ #2.修改hadoop用户下的环境变量配置文件 [hadoop@node1 ~]$ vi .bash_profile # get the aliases and functions # get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # user specific environment and startup programs java_home=/usr/java/jdk1.8.0_211-amd64 hadoop_home=/opt/bigdata/hadoop-2.7.3 spark_home=/opt/spark-2.4.3-bin-hadoop2.7 m2_home=/opt/apache-maven-3.0.5 #3.新增zookeeper的环境变量zookeeper_home zookeeper_home=/opt/bigdata/zookeeper-3.4.2/ path=$path:$home/bin:$java_home/bin:$hadoop_home/bin:$hadoop_home/sbin:$m2_home/bin #4.将zookeeper的环境变量zookeeper_home加入到path中 path=$path:$spark_home/bin:$spark_home/sbin:$zookeeper_home/bin export java_home export hadoop_home export m2_home export spark_home export hadoop_conf_dir=$hadoop_home/etc/hadoop export hdfs_conf_dir=$hadoop_home/etc/hadoop #5.导出zookeeper环境变量 export zookeeper_home #6.保存修改内容 :wq! #记得回车 #7.使得环境变量生效 [hadoop@node1 ~]$ source .bash_profile #8.输入zk然后按键盘左侧的tab键 [hadoop@node1 ~]$ zk #有如下的提示,表名zookeeper的配置完成 zkcleanup.sh zkcli.cmd zkcli.sh zkenv.cmd zkenv.sh zkserver.cmd zkserver.sh [hadoop@node1 ~]$ zk
1.5 修改zookeeper集群配置文件
将目录切换到zookeeper的安装目录下的conf目录下复制zoo_sample.cfg文件为zoo.cfg
[hadoop@node1 ~]$ cd /opt/bigdata/zookeeper-3.4.2/conf/ [hadoop@node1 conf]$ ll total 12 -rwxr-xr-x 1 hadoop hadoop 535 dec 22 2011 configuration.xsl -rwxr-xr-x 1 hadoop hadoop 2161 dec 22 2011 log4j.properties -rwxr-xr-x 1 hadoop hadoop 808 dec 22 2011 zoo_sample.cfg #1.复制zoo_sample.cfg模板配置文件为正式的配置文件zoo.cfg [hadoop@node1 conf]$ cp zoo_sample.cfg zoo.cfg [hadoop@node1 conf]$ ll total 16 -rwxr-xr-x 1 hadoop hadoop 535 dec 22 2011 configuration.xsl -rwxr-xr-x 1 hadoop hadoop 2161 dec 22 2011 log4j.properties -rwxr-xr-x 1 hadoop hadoop 808 jul 19 11:20 zoo.cfg -rwxr-xr-x 1 hadoop hadoop 808 dec 22 2011 zoo_sample.cfg [hadoop@node1 conf]$
修改datadir的值为 datadir=/var/lib/zookeeper,在文件的末尾添加如下配置:
server.1=node1:2888:3888 server.2=node2:2888:3888 server.3=node3:2888:3888
修改完配置文件记得保存
# the number of milliseconds of each tick ticktime=2000 # the number of ticks that the initial # synchronization phase can take initlimit=10 # the number of ticks that can pass between # sending a request and getting an acknowledgement synclimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. datadir=/var/lib/zookeeper # the port at which the clients will connect clientport=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxclientcnxns=60 # # be sure to read the maintenance section of the # administrator guide before turning on autopurge. 开课吧 kaikeba.com 精选领域名师，只为人才赋能 6 1.6 创建myid文件 在节点node1,node2,node3对应的/var/lib/zookeeper目录下(datadir配置的目录/var/lib/zookeeper)创建myid文 件,几个文件内容依次为1,2,3 如下图我们切换到root用户，在/var/lib目录下创建zookeeper目录，因为hadoop用户对/var/lib目录没有写权限， 所以我们在创建zookeeper目录时需要切换到root用户(拥有最大权限) # # http://zookeeper.apache.org/doc/current/zookeeperadmin.html#sc_maintenance # # the number of snapshots to retain in datadir #autopurge.snapretaincount=3 # purge task interval in hours # set to "0" to disable auto purge feature #autopurge.purgeinterval=1 server.1=node1:2888:3888 server.2=node2:2888:3888 server.3=node3:2888:3888 #修改完配置文件记得保存
1.6 创建myid文件
在节点node1,node2,node3对应的/var/lib/zookeeper目录下(datadir配置的录/var/lib/zookeeper)创建myid文件,几个文件内容依次为1,2,3。切换到root用户，在/var/lib目录下创建zookeeper目录，因为hadoop用户对/var/lib目录没有写权限，所以我们在创建zookeeper目录时需要切换到
root用户(拥有最大权限) [hadoop@node1 conf]$ vi zoo.cfg #1.切换到root用户 [hadoop@node1 conf]$ su - root password: last login: fri jul 19 10:53:59 cst 2019 from 192.168.200.1 on pts/0 #2.创建zookeeper目录 [root@node1 ~]# mkdir -p /var/lib/zookeeper #3.进入到/var/lib/zookeeper/目录 [root@node1 ~]# cd /var/lib/zookeeper/ you have new mail in /var/spool/mail/root #4.创建myid配置文件 [root@node1 zookeeper]# touch myid #5.编辑myid文件，输入1，我们目前编辑的是node1的节点的myid文件，node2的myid内容为2，node3的myid内容为3 [root@node1 zookeeper]# vi myid you have new mail in /var/spool/mail/root #6.查看一下myid文件内容为1 [root@node1 zookeeper]# cat myid 1 you have new mail in /var/spool/mail/root
1.7 修改myid目录权限
#1.配置完成后记得修改zookeeper目录的所属组和读写权限 [root@node1 zookeeper]# cd .. you have new mail in /var/spool/mail/root #2.修改zookeeper目录所属组 [root@node1 lib]# chown -r hadoop:hadoop zookeeper/ #3.修改zookeeper目录的读写权限为755 [root@node1 lib]# chmod -r 755 zookeeper/ [root@node1 lib]#
2、复制zookeeper
#1.复制/var/lib目录下的zookeeper目录到node2和node3的/var/lib目录下 [root@node1 lib]# scp -r zookeeper node2:$pwd [root@node1 lib]# scp -r zookeeper node3:$pwd #2.复制zookeeper安装目录到node2和node3的安装目录下/opt/bigdata目录下 [root@node1 lib]# scp -r /opt/bigdata/zookeeper-3.4.2/ node2:/opt/bigdata/ [root@node1 lib]# scp -r /opt/bigdata/zookeeper-3.4.2/ node3:/opt/bigdata/
3、修改node2和node3节点zookeeper的相关目录权限
修改node2节点zookeeper 相关目录权限
#1.修改zookeeper的myid配置目录所属组和读写权限 [root@node2 lib]# cd ~ [root@node2 ~]# chown -r hadoop:hadoop /var/lib/zookeeper [root@node2 ~]# chmod -r 755 /var/lib/zookeeper #2.修改zookeeper安装目录所属组和读写权限 [root@node2 ~]# chown -r hadoop:hadoop /opt/bigdata/zookeeper-3.4.2/ you have new mail in /var/spool/mail/root [root@node2 ~]# chmod -r 755 /opt/bigdata/zookeeper-3.4.2/ [root@node2 ~]#
修改node3节点zookeeper 相关目录权限
#1.修改zookeeper的myid配置目录所属组和读写权限 [root@node3 bigdata]# cd ~ you have new mail in /var/spool/mail/root [root@node3 ~]# chown -r hadoop:hadoop /var/lib/zookeeper [root@node3 ~]# chmod -r 755 /var/lib/zookeeper #2.修改zookeeper安装目录所属组和读写权限 [root@node3 ~]# chown -r hadoop:hadoop /opt/bigdata/zookeeper-3.4.2/ you have new mail in /var/spool/mail/root [root@node3 ~]# chmod -r 755 /opt/bigdata/zookeeper-3.4.2/ [root@node3 ~]#
4、修改node2和node3的myid文件内容
修改node2节点zookeeper 的myid内容为2：
[root@node2 ~]# vi /var/lib/zookeeper/myid you have new mail in /var/spool/mail/root [root@node2 ~]# cat /var/lib/zookeeper/myid 2 [root@node2 ~]#
修改node3节点zookeeper 的myid内容为3
[root@node3 ~]# vi /var/lib/zookeeper/myid you have new mail in /var/spool/mail/root [root@node3 ~]# cat /var/lib/zookeeper/myid 3 [root@node3 ~]#
5、配置node2和node3的zookeeper环境变量
我们在node1节点上直接将hadoop用户的环境变量配置文件远程复制到node2和node3的hadoop用户家目录下
#1.如果当前登录用户是root用户，需要切换到hadoop用户下，如果当前用户是hadoop用户，请将目录切换到hadoop用 户的家目录下,在进行环境变量文件的远程复制. [root@node1 lib]# su - hadoop last login: fri jul 19 11:08:44 cst 2019 on pts/0 [hadoop@node1 ~]$ scp .bash_profile node2:$pwd .bash_profile 100% 681 64.8kb/s 00:00 [hadoop@node1 ~]$ scp .bash_profile node3:$pwd .bash_profile 100% 681 156.8kb/s 00:00 [hadoop@node1 ~]$
5.1 使得node2和node3的环境变量生效
使得node2的hadoop的环境变量生效
#注意：切换到hadoop用户下 #1.使得环境变量生效 [hadoop@node2 ~]$ source .bash_profile #2.输入zk然后按键盘左侧的tab键 [hadoop@node2 ~]$ zk #3.有如下命令和shell脚本的提示，说明zookeeper的环境变量配置成功. zkcleanup.sh zkcli.sh zkenv.sh zkserver.sh zkcli.cmd zkenv.cmd zkserver.cmd [hadoop@node2 ~]$ zk
使得node3的hadoop的环境变量生效
#注意：切换到hadoop用户下 [root@node3 bigdata]# su - hadoop last login: thu jul 18 15:37:50 cst 2019 on :0 #1.使得环境变量生效 [hadoop@node3 ~]$ source .bash_profile #2.输入zk然后按键盘左侧的tab键 [hadoop@node3 ~]$ zk #3.有如下命令和shell脚本的提示，说明zookeeper的环境变量配置成功. zkcleanup.sh zkcli.sh zkenv.sh zkserver.sh zkcli.cmd zkenv.cmd zkserver.cmd [hadoop@node3 ~]$ zk
6.启动zookeeper集群
6.1 启动zookeeper集群
启动zookeeper集群需要手动分别依次在三台机器上启动,启动前需要在三台机器上都将用户切换为hadoop用户.
node1上启动zookeeper
[hadoop@node1 ~]$ zkserver.sh start jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg starting zookeeper ... started [hadoop@node1 ~]$
node2上启动zookeeper
[hadoop@node2 ~]$ zkserver.sh start jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg starting zookeeper ... started [hadoop@node2 ~]$
node3上启动zookeeper
[hadoop@node3 ~]$ zkserver.sh start jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg starting zookeeper ... started [hadoop@node3 ~]$
6.2 查看zookeeper集群状态
使用zkserver.sh status命令在三个节点分别执行查看状态
在node1上查看
[hadoop@node1 bin]$ zkserver.sh status jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg mode: follower [hadoop@node1 bin]$
在node2上查看
[hadoop@node2 bin]$ zkserver.sh status jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg mode: follower [hadoop@node2 bin]$
在node3上查看
[hadoop@node3 bin]$ zkserver.sh status jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg mode: leader [hadoop@node3 bin]$
至此我们zookeeper集群安装完成.
6.3 zookeeper安装遇到问题
由于按照hadoop2.7.3版本官方文档中使用zookeeper-3.4.2版本，但是zookeeper-3.4.2版本比较低，我们在启动zookeeper后，可以使用jps命令或者ps -ef|grep zookeeper命令查看zookeeper主进程的状态，但是我们发现是正常的，如果我们使用zkserver.sh status命令查看zookeeper的状态却显示是异常的,不管启动多少次都会得到同样的结果。
[hadoop@node1 bin]$ zkserver.sh status jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg error contacting service. it is probably not running. [hadoop@node2 bin]$ zkserver.sh status jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg error contacting service. it is probably not running. [hadoop@node3 bin]$ zkserver.sh status jmx enabled by default using config: /opt/bigdata/zookeeper-3.4.2/bin/../conf/zoo.cfg error contacting service. it is probably not running.
分析主要有以下两个原因造成:
1.centos7上没有安装nc工具.
2.zookeeper启动脚本中的nc命令在不同的linux版本中使用了无效的参数导致获取状态异常或者获取的状态为
空状态导致的。
解决方法:
1.使用yum 在三个节点上分别安装nc工具
yum install nc -y
2.修改zookeeper安装目录下的bin目录下的zkserver.sh脚本文件内容
修改完成后我们在使用zkserver.sh status就能看到zookeeper的状态了