Centos7上用Ambari+HDP2.6.4安装大数据环境

 

HDP2.6.4离线安装

1      概述

Apache Ambari是一个基于Web的支持Apache Hadoop集群的供应、管理和监控的开源工具，Ambari已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI进行可视化的集群管理，简化了大数据平台的安装、使用难度。

大数据平台：

2008 年成立的 Cloudera 是最早将 Hadoop 商用的公司，为合作伙伴提供 Hadoop 的商用解决方案，主要是包括支持，咨询服务，培训。Cloudera产品主要为CDH (Cloudera’s Distribution Including Apache Hadoop)，Cloudera Manager，Cloudera Navigator，Cloudera Support。CDH是Cloudera的hadoop发行版，完全开源，比Apache hadoop在兼容性，安全性，稳定性上有增强。

2011年成立的Hortonworks是雅虎与硅谷风投公司Benchmark Capital合资组建的公司。Hortonworks 的主打产品是Hortonworks Data Platform (HDP)，也同样是100%开源的产品，HDP除去常用服务还包含Ambari，开源的安装和管理系统，Hcatalog，元数据管理系统。

本文的大数据安装说明是基于HDP的。

 

2      系统环境配置

2.1      Hdp所需软件下载

wget -c http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz
wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget -c http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz
#jdk下载
wget -c http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz

2.2      主机规划

	Ip	用户名/密码	宿主机ip	主机名	Os	Cpu	内存	存储
1	*.*.*.*	*.*.*.*	*.*.*.*	ambari-server	Centos7.2	2*2	8G	80G
2	*.*.*.*	*.*.*.*	*.*.*.*	hadoop-1	Centos7.2	2*2	8G	80G
3	*.*.*.*	*.*.*.*	*.*.*.*	hadoop-2	Centos7.2	2*2	8G	80G
4	*.*.*.*	*.*.*.*	*.*.*.*	hadoop-3	Centos7.2	2*2	8G	80G

2.3      部署

2.3.1        修改主机名与关闭防火墙、SElinux

systemctl stop firewalld
systemctl disable firewalld  #避免防火墙的开机自启动
hostnamectl set-hostname  ambari-server   #更改个主机名
sed -i 's/SELINUX=enforcing/SELINUX=disable/g' /etc/selinux/config
setenforce 0

cat >>/etc/hosts<<EOF
*.*.*.* ambari-server
*.*.*.* hadoop-1
*.*.*.* hadoop-2
*.*.*.* hadoop-3
EOF

2.3.2        配置ambari-server（ams）免密登录其他节点

ssh-****** -t rsa     #在ambari-server输入该命令一路回车生成**对
for num in `seq 1 3`;do ssh-copy-id -i /root/.ssh/id_rsa.pub [e-mail]$num;done

2.3.3        配置阿里yum源（所有节点）

cd  /etc/yum.repos.d/
wget  http://mirrors.aliyun.com/repo/Centos-7.repo
mv  CentOs-Base.repo CentOs-Base.repo.bak
mv Centos-7.repo CentOs-Base.repo

yum clean all
yum makecache
yum list #查看yum源是否配置成功。

 

2.3.4        配置Java（所有节点）

tar -xvzf jdk*tar.gz -C /usr/local/
vim /etc/profile

在文件末尾加上

export JAVA_HOME=/usr/local/jdk1.8.0_112/
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

执行使之立即生效

source /etc/profile

查看Java环境配置成功

java –version

2.3.5        在ams安装mariadb(mysql)数据库

yum -y install mariadb-server mysql-connector-java
systemctl start mariadb
mysql_secure_installation #配置对应的用户名和密码
mysql -uroot password "mysqladmin"

2.3.6        配置本地的ambari yum源和hdp yum源

yum install httpd -y && systemctl start httpd

#创建目录

mkdir -p /var/www/html/ambari
mkdir /var/www/html/hdp/{HDP,HDP-UTILS-1.1.0.22}

#解压软件

tar -xvzf ambari-2.6.1.5-centos7.tar.gz -C /var/www/html/ambari/
tar -xvzf HDP-2.6.4.0-centos7-rpm.tar.gz -C /var/www/html/hdp/HDP/
tar -xvzf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.22/

配置ambari YUM源

cat >/etc/yum.repos.d/ambari.repo<<EOF
#VERSION_NUMBER=2.6.1.5
[Updates-ambari-2.6.1.5]
name=ambari-2.6.1.5 - Updates
baseurl=http://192.169.0.41/ambari/ambari/centos7/2.6.1.5-3
gpgcheck=1
gpgkey=http://192.169.0.41/ambari/ambari/centos7/2.6.1.5-3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF

#配置hdp  YUM源

cat >/etc/yum.repos.d/hdp.repo<<EOF
#VERSION_NUMBER=2.6.4.0-91
[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://192.169.0.41/hdp/HDP/HDP/centos7/2.6.4.0-91
gpgcheck=1
gpgkey=http://192.169.0.41/hdp/HDP/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.169.0.41/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=1
gpgkey=http://192.169.0.41/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF

2.3.7        安装配置ambari-server

安装ambari-server

yum clean all
yum makecache
yum -y install ambari-server

#进入数据库创建ambari库和导入数据

mysql>create database ambari DEFAULT CHARSET utf8;
mysql>use ambari
mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

#创建ambari用户与赋权

MariaDB [(none)]> CREATE USER 'ambari'@'%'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';

#以防万一可以加上这几条语句

CREATE USER 'ambari'@'ambari-server'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'ambari-server';
CREATE USER 'ambari'@'localhost'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';

#刷新

flush privileges;
SELECT host,user,password,Grant_priv,Super_priv FROM mysql.user;
UPDATE mysql.user SET Grant_priv='Y', Super_priv='Y'  WHERE User='ambari';
flush privileges;

配置ambari

[[e-mail] hdp]# ambari-server setup

Using python  /usr/bin/python

Setup ambari-server

Checking SELinux...

SELinux status is 'enabled'

SELinux mode is 'permissive'

WARNING: SELinux is set to 'permissive' mode and temporarily disabled.

OK to continue [y/n] (y)? y

Customize user account for ambari-server daemon [y/n] (n)? y

Enter user account for ambari-server daemon (root):root

Adjusting ambari-server permissions and ownership...

Checking firewall status...

Checking JDK...

Do you want to change Oracle JDK [y/n] (n)? y

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

[3] Custom JDK

==============================================================================

Enter choice (1): 3

……

Enter advanced database configuration [y/n] (n)? y

Configuring database...

==============================================================================

Choose one of the following options:

[1] - PostgreSQL (Embedded)

[2] - Oracle

[3] - MySQL / MariaDB

[4] - PostgreSQL

[5] - Microsoft SQL Server (Tech Preview)

[6] - SQL Anywhere

[7] - BDB

==============================================================================

Enter choice (3): 3

Hostname (ambari-server): ambari-server

Port (3306):

Database name (ambari):

Username (root): ambari

Enter Database Password (mysqladmin):

Configuring ambari database...

Copying JDBC drivers to server resources...

Configuring remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/r

esources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? y

Extracting system views...

............

Adjusting ambari-server permissions and ownership...

Ambari Server 'setup' completed successfully.

启动ambari-server

ambari-server start

3      安装服务

3.1      Web安装ambari-agent

浏览器访问：http://ambari-server:8080 默认用户名密码admin/admin

选择本地hdp源

添加写需要安装的各个目标节点的主机名，填写ambari-server免**登录其他节点的私钥，即id-rsa，确认注册

3.2      安装大数据服务

等待安装完成

3.3      Wordcount检测

在Hadoop1上起一个简单的wc任务来进行检测

su hdfs
hdfs dfs -mkdir /input
hdfs dfs –ls /
exit

$ cat wc.input
boding hadoop
yarn hadoop
hadoop spark
spark1 spark

hdfs dfs -put wc.input /input
hadoop jar /usr/hdp/2.6.4.0-91/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount  /input/ /output

查看结果