Hadoop2.7.1+Hbase1.2.1集群环境搭建(4)hbase安装准备

(1)hadoop2.7.1源码编译	http://aperise.iteye.com/blog/2246856
(2)hadoop2.7.1安装准备	http://aperise.iteye.com/blog/2253544
(3)1.x和2.x都支持的集群安装	http://aperise.iteye.com/blog/2245547
(4)hbase安装准备	http://aperise.iteye.com/blog/2254451
(5)hbase安装	http://aperise.iteye.com/blog/2254460
(6)snappy安装	http://aperise.iteye.com/blog/2254487
(7)hbase性能优化	http://aperise.iteye.com/blog/2282670
(8)雅虎YCSBC测试hbase性能测试	http://aperise.iteye.com/blog/2248863
(9)spring-hadoop实战	http://aperise.iteye.com/blog/2254491
(10)基于ZK的Hadoop HA集群安装	http://aperise.iteye.com/blog/2305809

 

1.hadoop环境安装

       hbase是建立在Hadoop之上的一个NOSQL列存储数据库，所以你得首先安装Hadoop，安装Hadoop参见http://aperise.iteye.com/blog/2245547

2.zookeeper3.4.6集群安装

       1，hbase regionserver 向zookeeper注册，提供hbase regionserver状态信息（是否在线）

       2，hmaster启动时候会将hbase 系统表-ROOT- 加载到 zookeeper cluster，通过zookeeper cluster可以获取当前系统表.META.的存储所对应的regionserver信息。

        3.zookeeper集群节点最少3台，这样才能满足选举规则，少数服从多数

1) 下载解压

下载zookeeper-3.4.6.tar.gz到/opt

解压tar -zxvf zookeeper-3.4.6.tar.gz

2) 配置/etc/hosts

 

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.186.234 inddb1-vip
192.168.186.235 inddb2-vip
192.168.88.22 nmsc2
192.168.88.21 nmsc1
192.168.181.66 nmsc0

 

3) 创建zookeeper数据文件

#zookeeper每台机器上都要创建

sudo rm -r /home/hadoop/zookeeper

cd  /home/hadoop

mkdir zookeeper

4) 配置zoo.cfg

将zookeeper-3.3.4/conf目录下面的 zoo_sample.cfg复制一份为zoo.cfg，配置文件内容如下所示

 

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/hadoop/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#数据文件保存最近的3个快照，默认是都保存，时间长的话会占用很大磁盘空间

autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#单位为小时，每小时清理一次快照数据

autopurge.purgeInterval=1
server.1=nmsc0:2888:3888
server.2=nmsc1:2888:3888
server.3=nmsc2:2888:3888

 

5) 远程复制分发安装文件到其它zookeeper集群节点

scp -r /opt/zookeeper-3.3.4 root@nmsc1:/opt/

scp -r /opt/zookeeper-3.3.4 root@nmsc2:/opt/

6) 设置myid必须为整数

#zookeeper集群节点之一

ssh nmsc0

echo "1" > /home/hadoop/zookeeper/myid  

##zookeeper集群节点之二

ssh nmsc1

echo "2" > /home/hadoop/zookeeper/myid  

##zookeeper集群节点之三

ssh nmsc2

echo "3" > /home/hadoop/zookeeper/myid  

7) 启动ZooKeeper集群

cd /opt/zookeeper-3.4.6

bin/zkServer.sh start  

8) 查看单机ZooKeeper是leader还是follower

cd /opt/zookeeper-3.4.6

bin/zkServer.sh status 

9) 停止ZooKeeper集群

cd /opt/zookeeper-3.4.6

bin/zkServer.sh stop 

10) 用客户端查看zookeeper上数据

客户端链接zookeeper服务端

cd /opt/zookeeper-3.4.6/
bin/zkCli.sh -server nmsc0:2181,nmsc2:2181,nmsc1:2181

 登录后提示信息如下：

[root@nmsc1 bin]# ./zkCli.sh -server nmsc0:2181,nmsc2:2181,nmsc1:2181
Connecting to nmsc0:2181,nmsc2:2181,nmsc1:2181
2016-04-19 10:36:20,475 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-04-19 10:36:20,479 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=nmsc1
2016-04-19 10:36:20,479 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_65
2016-04-19 10:36:20,481 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2016-04-19 10:36:20,481 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/opt/java/jdk1.7.0_65/jre
2016-04-19 10:36:20,481 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper-3.4.6/bin/../build/classes:/opt/zookeeper-3.4.6/bin/../build/lib/*.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/opt/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.6/bin/../conf:.:/opt/java/jdk1.7.0_65/jre/lib/rt.jar:/opt/java/jdk1.7.0_65/lib/dt.jar:/opt/java/jdk1.7.0_65/lib/tools.jar
2016-04-19 10:36:20,481 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=:/opt/jprofiler7/bin/linux-x64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-04-19 10:36:20,481 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2016-04-19 10:36:20,482 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2016-04-19 10:36:20,482 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2016-04-19 10:36:20,482 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2016-04-19 10:36:20,482 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.18-308.el5
2016-04-19 10:36:20,482 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2016-04-19 10:36:20,482 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2016-04-19 10:36:20,482 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/zookeeper-3.4.6/bin
2016-04-19 10:36:20,484 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=nmsc0:2181,nmsc2:2181,nmsc1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@8a61d64
Welcome to ZooKeeper!
2016-04-19 10:36:20,511 [myid:] - INFO  [main-SendThread(nmsc2:2181):ClientCnxn$SendThread@975] - Opening socket connection to server nmsc2/192.168.88.22:2181. Will not attempt to authenticate using SASL (unknown error)
2016-04-19 10:36:20,515 [myid:] - INFO  [main-SendThread(nmsc2:2181):ClientCnxn$SendThread@852] - Socket connection established to nmsc2/192.168.88.22:2181, initiating session
JLine support is enabled
2016-04-19 10:36:20,528 [myid:] - INFO  [main-SendThread(nmsc2:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server nmsc2/192.168.88.22:2181, sessionid = 0x353a8f6e9547a07, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

 查看根目录下数据节点

[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 0] ls /
[isr_change_notification, hbase, zookeeper, admin, consumers, config, controller, kafka-manager, brokers, controller_epoch]
[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 1] ls /controller
[]
[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 2] ls /hbase
[meta-region-server, backup-masters, table, draining, region-in-transition, table-lock, running, master, namespace, hbaseid, online-snapshot, replication, splitWAL, recovering-regions, rs, flush-table-proc]
[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 3] ls /zookeeper
[quota]
[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 4] ls /brokers
[seqid, topics, ids]
[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 5] ls /brokers/topics
[signal2015, __consumer_offsets]
[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 6] ls /brokers/topics/signal2015
[partitions]
[zk: nmsc0:2181,nmsc2:2181,nmsc1:2181(CONNECTED) 7] 

 

3.时间同步

      注意：集群中机器时间同步很关键，时间最好相差在1秒以下，不然hbase启动会失败

方法一

安装NTP服务，参见http://wenku.baidu.com/link?url=0outiNUpCi-RzCtmWCKgqyVE0S_hyEcc3cjfbes_XHtxDN4wCh3-TXAe8YrtW1iOudzloNbMRMFe6niTS_uODhWEdPjD7MEwimib1a02FVm

 

方法二

我这是公司内部有时间同步的机器，直接从该机器同步数据，执行命令如下：

#从机器192.168.162.23同步系统时间,可以采用linux定时任务每天凌晨执行一次

ntpdate -u 192.168.162.23