利用shell脚本快速启动hadoop集群-程序员的福音

今天来记录一下编写的shell脚本快速启动hadoop集群

各位同学可以参考参考,如有不妥请多多指教

1-准备工作，集群机器之间需相互ssh免密登录

点击此处跳转免密登录教程

2-集群脚本注意脚本的权限和文件格式为(xxx.sh)

3-给文件夹或脚本添加可执行权限: chmod +5 start_hadoop.sh

#!/bin/bash
#我们将脚本存入在hadoop01，因为它有到hadoop02、hadoop03的免登录
clear
echo "开始启动集群..."
echo "->->->->->->->->->->->->start zkServer-----------------------------------"
for ip in hadoop01 hadoop02 hadoop03
do
    echo "ssh to $ip"
    ssh [email protected]$ip "source /etc/profile;/export/install/zookeeper-3.4.9/bin/zkServer.sh start"
    ssh [email protected]$ip "source /etc/profile;jps"
done
echo "->->->->->->->->start hadoop01 hadoop02 hadoop03 hdfs----------------"
ip=hadoop01
ssh [email protected]$ip "source /etc/profile;/export/install/hadoop-2.7.5/sbin/start-dfs.sh"
ip=hadoop01
ssh [email protected]$ip "source /etc/profile;jps"
ip=hadoop02
ssh [email protected]$ip "source /etc/profile;jps"
ip=hadoop03
ssh [email protected]$ip "source /etc/profile;jps"
echo "->->->->->->->->->->->->start hadoop01 yarn------------------------------"
ip=hadoop01
ssh [email protected]$ip "source /etc/profile;/export/install/hadoop-2.7.5/sbin/start-yarn.sh"
ssh [email protected]$ip "source /etc/profile;jps"
echo "->->->->->->->->->->->->start hadoop01 jobhistory------------------------------"
ip=hadoop01
ssh [email protected]$ip "source /etc/profile;/export/install/hadoop-2.7.5/sbin/mr-jobhistory-daemon.sh start historyserver"
ssh [email protected]$ip "source /etc/profile;jps"
echo "集群启动成功..."
echo http://hadoop01:50070/explorer.html#/     查看hdfs接口:50070
echo http://hadoop01:8088/cluster     查看yarn集群接口:8088
echo http://hadoop01:19888/jobhistory     查看历史完成的任务接口:19888

启动命令:./start_hadoop.sh

启动集群的窗口打印目录:

[[email protected] shell]# ./start_hadoop.sh 
开始启动集群...
->->->->->->->->->->->->start zkServer-----------------------------
ssh to hadoop01
ZooKeeper JMX enabled by default
Using config: /export/install/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
9265 QuorumPeerMain
9305 Jps
ssh to hadoop02
ZooKeeper JMX enabled by default
Using config: /export/install/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
6055 Jps
6027 QuorumPeerMain
ssh to hadoop03
ZooKeeper JMX enabled by default
Using config: /export/install/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
4196 QuorumPeerMain
4237 Jps
->->->->->->->->start hadoop01 hadoop02 hadoop03 hdfs----------------
Starting namenodes on [hadoop01]
hadoop01: starting namenode, logging to /export/install/hadoop-2.7.5/logs/hadoop-root-namenode-hadoop01.out
hadoop01: starting datanode, logging to /export/install/hadoop-2.7.5/logs/hadoop-root-datanode-hadoop01.out
hadoop03: starting datanode, logging to /export/install/hadoop-2.7.5/logs/hadoop-root-datanode-hadoop03.out
hadoop02: starting datanode, logging to /export/install/hadoop-2.7.5/logs/hadoop-root-datanode-hadoop02.out
Starting secondary namenodes [hadoop01]
hadoop01: starting secondarynamenode, logging to /export/install/hadoop-2.7.5/logs/hadoop-root-secondarynamenode-hadoop01.out
9265 QuorumPeerMain
9765 SecondaryNameNode
9943 Jps
9577 DataNode
9465 NameNode
6216 Jps
6105 DataNode
6027 QuorumPeerMain
4387 Jps
4196 QuorumPeerMain
4276 DataNode
->->->->->->->->->->->->start hadoop01 yarn------------------------------
starting yarn daemons
starting resourcemanager, logging to /export/install/hadoop-2.7.5/logs/yarn-root-resourcemanager-hadoop01.out
hadoop03: starting nodemanager, logging to /export/install/hadoop-2.7.5/logs/yarn-root-nodemanager-hadoop03.out
hadoop01: starting nodemanager, logging to /export/install/hadoop-2.7.5/logs/yarn-root-nodemanager-hadoop01.out
hadoop02: starting nodemanager, logging to /export/install/hadoop-2.7.5/logs/yarn-root-nodemanager-hadoop02.out
10177 Jps
9265 QuorumPeerMain
10018 ResourceManager
9765 SecondaryNameNode
9577 DataNode
9465 NameNode
10127 NodeManager
->->->->->->->->->->->->start hadoop01 jobhistory------------------------------
starting historyserver, logging to /export/install/hadoop-2.7.5/logs/mapred-root-historyserver-hadoop01.out
9265 QuorumPeerMain
10018 ResourceManager
9765 SecondaryNameNode
10375 Jps
9577 DataNode
9465 NameNode
10253 JobHistoryServer
10127 NodeManager
集群启动成功...
http://hadoop01:50070/explorer.html#/ 查看hdfs接口:50070
http://hadoop01:8088/cluster 查看yarn集群接口:8088
http://hadoop01:19888/jobhistory 查看历史完成的任务接口:19888

这样集群便启动成功了,希望同学们学习愉快+3