Hadoop伪分布式部署之ssh免密钥登陆

Hadoop伪分布式部署之ssh免密钥登陆

在之前的章节中，我们有介绍到伪分布式的hdfs、yarn和mapreduce、历史服务与日志聚集、SecondaryNameNode的部署。接下来我们一起探讨下hadoop的ssh免密钥登陆。

我们的hadoop环境如下
操作系统：CentOS6.4
Java版本：Oracle jdk1.7
Hadoop版本：Hadoop2.5.0
主机hostname：hadoop01.datacenter.com
hadoop目录：/opt/modules/hadoop-2.5.0

启动停止hadoop服务

在hadoop的sbin目录下，我们可以看到一些start-*.sh和stop-*.sh的脚本：

[hadoop@hadoop01 ~]$ cd /opt/modules/hadoop-2.5.0/
[hadoop@hadoop01 hadoop-2.5.0]$ ll sbin/
total 88
-rwxr-xr-x 1 hadoop hadoop 2752 Aug  7  2014 distribute-exclude.sh
-rwxr-xr-x 1 hadoop hadoop 6435 Aug  7  2014 hadoop-daemon.sh
-rwxr-xr-x 1 hadoop hadoop 1360 Aug  7  2014 hadoop-daemons.sh
-rwxr-xr-x 1 hadoop hadoop 1427 Aug  7  2014 hdfs-config.sh
-rwxr-xr-x 1 hadoop hadoop 2291 Aug  7  2014 httpfs.sh
-rwxr-xr-x 1 hadoop hadoop 4063 Aug  7  2014 mr-jobhistory-daemon.sh
-rwxr-xr-x 1 hadoop hadoop 1648 Aug  7  2014 refresh-namenodes.sh
-rwxr-xr-x 1 hadoop hadoop 2145 Aug  7  2014 slaves.sh
-rwxr-xr-x 1 hadoop hadoop 1471 Aug  7  2014 start-all.sh
-rwxr-xr-x 1 hadoop hadoop 1128 Aug  7  2014 start-balancer.sh
-rwxr-xr-x 1 hadoop hadoop 3705 Aug  7  2014 start-dfs.sh
-rwxr-xr-x 1 hadoop hadoop 1357 Aug  7  2014 start-secure-dns.sh
-rwxr-xr-x 1 hadoop hadoop 1347 Aug  7  2014 start-yarn.sh
-rwxr-xr-x 1 hadoop hadoop 1462 Aug  7  2014 stop-all.sh
-rwxr-xr-x 1 hadoop hadoop 1179 Aug  7  2014 stop-balancer.sh
-rwxr-xr-x 1 hadoop hadoop 3206 Aug  7  2014 stop-dfs.sh
-rwxr-xr-x 1 hadoop hadoop 1340 Aug  7  2014 stop-secure-dns.sh
-rwxr-xr-x 1 hadoop hadoop 1340 Aug  7  2014 stop-yarn.sh
-rwxr-xr-x 1 hadoop hadoop 4278 Aug  7  2014 yarn-daemon.sh
-rwxr-xr-x 1 hadoop hadoop 1353 Aug  7  2014 yarn-daemons.sh
[hadoop@hadoop01 hadoop-2.5.0]$ 

现在我们使用start-yarn.sh和stop-yarn.sh启动停止yarn服务试试：

[hadoop@hadoop01 hadoop-2.5.0]$ sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-resourcemanager-hadoop01.datacenter.com.out
The authenticity of host 'hadoop01.datacenter.com (192.168.190.151)' can't be established.
RSA key fingerprint is e2:ca:19:e5:04:0e:3c:11:d3:1a:cb:1f:b0:03:e6:87.
Are you sure you want to continue connecting (yes/no)? yes
hadoop01.datacenter.com: Warning: Permanently added 'hadoop01.datacenter.com,192.168.190.151' (RSA) to the list of known hosts.
[e-mail]'s password: 
hadoop01.datacenter.com: starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-nodemanager-hadoop01.datacenter.com.out
[hadoop@hadoop01 hadoop-2.5.0]$ 
[hadoop@hadoop01 hadoop-2.5.0]$ sbin/stop-yarn.sh 
stopping yarn daemons
stopping resourcemanager
[e-mail]'s password: 
hadoop01.datacenter.com: stopping nodemanager
no proxyserver to stop
[hadoop@hadoop01 hadoop-2.5.0]$ 

上面的启动停止过程中，有提到RSA，并且让我们输入了hadoop用户的密码，熟悉linux的同学都知道，这是用了ssh登陆hadoop01.datacenter.com这台机器。
我们可以查看这些shell脚本的代码，可以看到调用关系是start-yarn.sh->yarn-daemons.sh->slaves.sh。
在slaves.sh中，有下面这样一段：

...
# start the daemons
for slave in $SLAVE_NAMES ; do
 ssh $HADOOP_SSH_OPTS $slave $"${@// /\\ }" \
   2>&1 | sed "s/^/$slave: /" &
 if [ "$HADOOP_SLAVE_SLEEP" != "" ]; then
   sleep $HADOOP_SLAVE_SLEEP
 fi
...

可以看出来，这里有使用ssh协议访问每个从节点。如果整个集群有成百上千个节点，那么我们在主节点使用ssh协议开启从节点的服务的时候，输入密码这个工作量非常大，不利于集群维护的便利性，所以我们可以采用免密钥登陆的方式。

ssh免密钥登陆配置

首先删除现有的ssh信息：

[hadoop@hadoop01 hadoop-2.5.0]$ cd ~/.ssh
[hadoop@hadoop01 .ssh]$ ll
total 4
-rw-r--r-- 1 hadoop hadoop 421 Apr 15 20:40 known_hosts
[hadoop@hadoop01 .ssh]$ rm known_hosts 
[hadoop@hadoop01 .ssh]$ ll
total 0
[hadoop@hadoop01 .ssh]$ 

然后不输入密码（直接按三次回车）生成私钥和公钥：

[hadoop@hadoop01 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b6:35:ea:59:32:ed:3a:24:65:5b:8b:67:63:88:84:a9 [e-mail]
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|     o           |
|    o . o .      |
|   . . +S=o.     |
|  E   o.==*.     |
|       o=+o.     |
|       ..*       |
|        +o.      |
+-----------------+
[hadoop@hadoop01 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Apr 15 21:15 id_rsa
-rw-r--r-- 1 hadoop hadoop  412 Apr 15 21:15 id_rsa.pub
[hadoop@hadoop01 .ssh]$ 

其中id_rsa为私钥文件，id_rsa.pub为公钥文件。
接下来我们将公钥发送给从节点hadoop01.datacenter.com：

[hadoop@hadoop01 .ssh]$ ssh-copy-id hadoop01.datacenter.com
The authenticity of host 'hadoop01.datacenter.com (192.168.190.151)' can't be established.
RSA key fingerprint is e2:ca:19:e5:04:0e:3c:11:d3:1a:cb:1f:b0:03:e6:87.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop01.datacenter.com,192.168.190.151' (RSA) to the list of known hosts.
[e-mail]'s password: 
Now try logging into the machine, with "ssh 'hadoop01.datacenter.com'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[hadoop@hadoop01 .ssh]$ ll
total 16
-rw------- 1 hadoop hadoop  412 Apr 15 21:19 authorized_keys
-rw------- 1 hadoop hadoop 1675 Apr 15 21:15 id_rsa
-rw-r--r-- 1 hadoop hadoop  412 Apr 15 21:15 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  421 Apr 15 21:18 known_hosts
[hadoop@hadoop01 .ssh]$ 

现在我们就可以通过无密码通过ssh登陆到hadoop01.datacenter.com节点了：

[hadoop@hadoop01 .ssh]$ ssh [e-mail]       
Last login: Sun Apr 15 21:12:33 2018 from 192.168.190.1
[hadoop@hadoop01 ~]$ exit
logout
Connection to hadoop01.datacenter.com closed.
[hadoop@hadoop01 .ssh]$ 

接下来我们试一下hadoop中的start-yarn.sh和stop-yarn.sh：

[hadoop@hadoop01 .ssh]$ cd /opt/modules/hadoop-2.5.0/
[hadoop@hadoop01 hadoop-2.5.0]$ sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-resourcemanager-hadoop01.datacenter.com.out
hadoop01.datacenter.com: starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-nodemanager-hadoop01.datacenter.com.out
[hadoop@hadoop01 hadoop-2.5.0]$ jps
4281 ResourceManager
4708 Jps
4461 NodeManager
[hadoop@hadoop01 hadoop-2.5.0]$ sbin/stop-yarn.sh 
stopping yarn daemons
stopping resourcemanager
hadoop01.datacenter.com: stopping nodemanager
no proxyserver to stop
[hadoop@hadoop01 hadoop-2.5.0]$ jps
4843 Jps
[hadoop@hadoop01 hadoop-2.5.0]$ 

无需输入ssh登陆用户的密码，便成功启动和停止了yarn的相关服务。

总结

1、hadoop可以通过ssh协议启动和停止集群中的节点的相关服务。
2、可以通过配置无密钥登陆，来访问hadoop集群中的节点。
3、使用“ssh-keygen -t rsa”命令可以生成公私密钥对。
4、使用“ssh-copy-id 节点IP或者主机名”可以将公钥发送给相应节点。