CDH6.3.2安装部署

CDH安装过程（6.3.2）环境篇

一、系统级别配置

1.服务器磁盘做raid

略

2.安装操作系统

操作系统选择为CentOS7.6

3.磁盘分区

由于数据盘都是大于2T的磁盘，因此采用如下方式，如果磁盘大小小于2T，可采用fdisk的方式

cd /dev
ls
parted /dev/sdb
print
unit TB
mklabel gpt
mkpart primary 0 2.40TB
quit

4.关闭防火墙

systemctl stop firewalld
chkconfig firewalld off

5.关闭selinux

vim /etc/sysconfig/selinux
SELINUX=disabled

setenforce 0

6.更改文件句柄数

vim /etc/security/limits.conf
添加
*		soft		nofile		1769445
*		hard		nofile		1029345
*		soft		nproc		unlimited

7.挂载盘符

这里以有7块盘为例
++++++++++++++++++++++++++++++++++
mkdir /data1
mkdir /data2
mkdir /data3
mkdir /data4
mkdir /data5
mkdir /data6
mkdir /data7
++++++++++++++++++++++++++++++++++
mkfs.xfs /dev/sdb1
mkfs.xfs /dev/sdc1
mkfs.xfs /dev/sdd1
mkfs.xfs /dev/sde1
mkfs.xfs /dev/sdf1
mkfs.xfs /dev/sdg1
mkfs.xfs /dev/sdh1
++++++++++++++++++++++++++++++++++
mount /dev/sdb1 /data1
mount /dev/sdc1 /data2
mount /dev/sdd1 /data3
mount /dev/sde1 /data4
mount /dev/sdf1 /data5
mount /dev/sdg1 /data6
mount /dev/sdh1 /data7
++++++++++++++++++++++++++++++++++
vim /etc/fstab
/dev/sdb1		/data1		xfs		defaults		0		0
/dev/sdc1		/data2		xfs		defaults		0		0
/dev/sdd1		/data3		xfs		defaults		0		0
/dev/sde1		/data4		xfs		defaults		0		0
/dev/sdf1		/data5		xfs		defaults		0		0
/dev/sdg1		/data6		xfs		defaults		0		0
/dev/sdh1		/data7		xfs		defaults		0		0

8.关闭透明大页面

所有节点执行

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

vim /etc/rc.d/rc.local
添加如下内容
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
	echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
	echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

9.设置swappiness

所有节点执行

sysctl vm.swappiness=1
echo 1 > /proc/sys/vm/swappiness

vim /etc/sysctl.conf
vm.swappiness=1

10.修改主机名

vim /etc/hostname

vim /etc/hosts

11.配置网络

nmtui

band和ip等关于网络的配置结合自己的生产环境进行配置

12.制作本地yum源

上传本地yum源文件到服务器主节点


vi /etc/yum.repos.d/centos7.3.repo
[cehtos7]
name=centos7
baseurl=file:///home/cdrom
gpgkey=file:///home/cdrom/RPM-GPG-KEY-CentOS-7
enabled=1
gpgcheck=0

安装http服务
yum -y install httpd
systemctl start httpd
chkconfig httpd on

将本地yum源发布到http服务
mv /home/cdrom/* /var/www/html/centos7

将之前的本地yum源修改为网络版
vi /etc/yum.repos.d/centos7.3.repo
[cehtos7]
name=centos7
baseurl=http://yum源ip/centos7
gpgkey=http://yum源ip/centos7/RPM-GPG-KEY-CentOS-7
enabled=1
gpgcheck=0

将/etc/yum.repos.d/centos7.3.repo文件分发到其余的节点
scp /etc/yum.repos.d/centos7.3.repo 节点ip：/etc/yum.repos.d/

所有节点运行
yum clean all
yum makecache

所有节点安装ntp服务
yum -y remove chrony （删除系统自带的时间同步工具）
yum -y install ntp
设置同步的服务器
vi /etc/ntp.conf
主节点
server 127.127.1.0 iburst
其他节点
server 主节点ip iburst
server 127.127.1.0
fudge 127.127.1.0 stratum 8
restrict 主节点ip nomodify notrap noquery

所有节点执行
systemctl restart ntpd
chkconfig ntpd on
ntpstat

13.制作安装cm和cdh的本地yum源

上传cm和cdh到主节点/var/www/html目录下
createrepo .
到cm目录下执行
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

vim /etc/yum.repo.d/cm.repo
[cm]
name = cm
baseurl = http://主节点ip/cm/
enabled = 1
gpgcheck = 0

vim /etc/yum.repo.d/cdh.repo
[cdh]
name = cdh
baseurl = http://主节点ip/cdh/
enabled = 1
gpgcheck = 0


将/etc/yum.repo.d/cdh.repo和/etc/yum.repo.d/cm.repo分发至其他节点
scp /etc/yum.repos.d/cm.repo 节点ip：/etc/yum.repos.d/
scp /etc/yum.repos.d/cdh.repo 节点ip：/etc/yum.repos.d/

所有节点执行
yum clean all
yum makecache

13.安装mysql

安装方式为离线安装：
上传mysql的tar包到mysql安装节点
tar -xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar 
安装之前先删除miniserver对应的rpm包，否则会影响安装
rm -rf mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm
安装
yum install mysql-community-{server,client,common,libs}-*
启动mysql
systemctl start mysqld.service
systemctl enable mysqld.service
使用临时密码登录
grep 'temporary password' /var/log/mysqld.log
登录
mysql -u root -p
输入查询到的临时密码
修改mysql root用户的密码
ALTER USER 'root'@'localhost' IDENTIFIED BY '新密码';
修改mysql root用户的登录权限
update mysql.user set Host='%' where HOST='localhost' and User='root';
flush privileges;

创建各组件需要的元数据库
create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;
create database cm default character set utf8;
CREATE USER 'cm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH PRIVILEGES;
create database rm default character set utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH PRIVILEGES;
create database hue default character set utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
FLUSH PRIVILEGES;
create database oozie default character set utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';
FLUSH PRIVILEGES;
create database ranger default character set utf8;
CREATE USER 'rangeradmin'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON ranger. * TO 'rangeradmin'@'%';
FLUSH PRIVILEGES;
（根据需要适当添加）


MySQL驱动(cm安装在哪一个节点，哪一个节点就做如下操作)
mkdir -p /usr/share/java/
mv mysql-connector-java-5.1.34.jar /usr/share/java/
cd /usr/share/java
chmod 777 mysql-connector-java-5.1.34.jar
ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar

14.安装cm

yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

15.注入

注入CM-server库 与 CM-agent端的server（MySQL安装完之后执行）

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmf root 'password（mysql中cmf库的密码）'

16.启动cm

service cloudera-scm-server start 
cd /var/log/cloudera-scm-server/
tail -f cloudera-scm-server.log

二、组件安装

根据自己场景需求，安装相应的组件即可