Hadoop集群(CDH4)实践之 (3) Hive搭建
目录结构 Hadoop集群(CDH4)实践之 (0) 前言 Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建 Hadoop集群(CDH4)实践之 (2) HBaseZookeeper搭建 Hadoop集群(CDH4)实践之 (3) Hive搭建 Hadoop集群(CHD4)实践之 (4) Oozie搭建 Hadoop集群(CHD4)实践之 (5) Sqoop安
目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装
本文内容
Hadoop集群(CDH4)实践之 (3) Hive搭建
参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html
环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
- namenode
- hbase-master
hadoop-secondary: 172.17.20.234 内存10G
- secondarybackupnamenode,jobtracker
- hive-server,hive-metastore
hadoop-node-1: 172.17.20.231 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server
hadoop-node-2: 172.17.20.232 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server
hadoop-node-3: 172.17.20.233 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server
对以上角色做一些简单的介绍:
namenode - 整个HDFS的命名空间管理服务
secondarynamenode - 可以看做是namenode的冗余服务
jobtracker - 并行计算的job管理服务
datanode - HDFS的节点服务
tasktracker - 并行计算的job执行服务
hbase-master - Hbase的管理服务
hbase-regionServer - 对Client端插入,删除,查询数据等提供服务
zookeeper-server - Zookeeper协作与配置管理服务
hive-server - Hive的管理服务
hive-metastore - Hive的元存储,用于对元数据进行类型检查与语法分析
本文定义的规范,避免在配置多台服务器上产生理解上的混乱:
以下操作都只需要在 Hive 所在主机,即 hadoop-secondary 上执行。
1. 安装前的准备
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
2. 安装Hive
$ sudo yum install hive hive-metastore hive-server
$ sudo yum install hive-jdbc hive-hbase
3. 安装MySQL JDBC Connector
$ sudo yum install mysql-connector-java
$ sudo ln -s /usr/share/java/mysql-connector-java.jar /usr/lib/hive/lib/mysql-connector-java.jar
4. 安装MySQL
$ sudo yum install mysql-server
$ sudo /etc/init.d/mysqld start
$ sudo /usr/bin/mysql_secure_installation
[...] Enter current password for root (enter for none): OK, successfully used password, moving on... [...] Set root password? [Y/n] y New password: hiveserver Re-enter new password: hiverserver Remove anonymous users? [Y/n] Y [...] Disallow root login remotely? [Y/n] N [...] Remove test database and access to it [Y/n] Y [...] Reload privilege tables now? [Y/n] Y All done!
5. 创建数据库并授权
$ mysql -u root -phiveserver
mysql> CREATE DATABASE metastore; mysql> USE metastore; mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql; mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hiveserver'; mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'%'; mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'%'; mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hiveserver'; mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'localhost'; mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'localhost'; mysql> CREATE USER 'hive'@'127.0.0.1' IDENTIFIED BY 'hiveserver'; mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'127.0.0.1'; mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'127.0.0.1';
6. 配置hive-site.xml
$ sudo vim /etc/hive/conf/hive-site.xml
javax.jdo.option.ConnectionURL jdbc:mysql://hadoop-secondary/metastore the URL of the MySQL database javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver javax.jdo.option.ConnectionUserName hive javax.jdo.option.ConnectionPassword hiveserver datanucleus.autoCreateSchema false datanucleus.fixedDatastore true datanucleus.autoStartMechanism SchemaTable hive.metastore.uris thrift://hadoop-secondary:9083 IP address (or fully-qualified domain name) and port of the metastore host hive.aux.jars.path file:////usr/lib/hive/lib/hbase.jar,file:///usr/lib/hive/lib/zookeeper.jar,file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar,file:///usr/lib/hive/lib/guava-11.0.2.jar hbase.zookeeper.quorum hadoop-node-1,hadoop-node-2,hadoop-node-3
7. 启动Hive
$ /etc/init.d/hive-metastore start
$ /etc/init.d/hive-server start
8. 创建Hive所需的HDFS目录
$ sudo -u hdfs hadoop fs -mkdir /user/hive
$ sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse
$ sudo -u hdfs hadoop fs -ls -R /user
$ sudo -u hdfs hadoop fs -chown -R hive /user/hive
$ sudo -u hdfs hadoop fs -chmod -R 1777 /user/hive/warehouse
$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hadoop-mapred
$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hive-hive
$ sudo chown -R hive:hive /var/lib/hive/.hivehistory
9. 至此,Hive的搭建就已经完成。
下一篇: [转]PHP中fopen,file