hadoop集群搭建部署
环境:
- 配好IP,关闭iptables, 关闭selinux,配置hosts
配置hosts:
192.168.1.18 zhusw07-1
192.168.1.20 zhusw07-64
192.168.1.11 zhusw07-62
- 三台服务器上都要创建普通用户,hadoop,配置密码: 123456
[aaa@qq.com-64 ~]# useradd -u 8000 hadoop ; echo 123456 | passwd --stdin hadoop
更改用户 hadoop 的密码 。
passwd: 所有的身份验证令牌已经成功更新。
- 设置namenode能够无**登录另外两台服务器
[aaa@qq.com-1 ~]# ssh-******
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
14:35:ad:7d:68:de:9a:4d:e0:83:06:e4:5a:d4:c5:42 aaa@qq.com-1
The key's randomart image is:
+--[ RSA 2048]----+
| .+E+. |
| o.o.o |
| +. + . |
| .+ . = . |
| oS. = + |
| . o + o |
| . * |
| o . |
| |
+-----------------+
[aaa@qq.com ~]# ssh-copy-id aaa@qq.com
The authenticity of host '192.168.1.11 (192.168.1.11)' can't be established.
RSA key fingerprint is 3f:70:d9:6f:c8:54:70:6f:14:0f:73:10:44:fc:51:7d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.11' (RSA) to the list of known hosts.
aaa@qq.com192.168.1.11's password:
[aaa@qq.com-1 ~]# ssh-copy-id aaa@qq.com192.168.1.20
测试(是否能登录成功)
[aaa@qq.com-1 ~]# ssh 192.168.1.20
Last login: Tue Mar 3 13:36:50 2020 from 192.168.1.4
- 安装jdk
[aaa@qq.com-1 ~]# rpm -ivh jdk-8u131-linux-x64_.rpm
//查看java的安装目录
195 rpm -pql /root/jdk-8u131-linux-x64_.rpm
修改/etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_131/
export JAVA_BIN=/usr/java/jdk1.8.0_131/bin/
export PATH=${JAVA_HOME}/bin:$PATH
export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
//j加载:
[aaa@qq.com-1 ~]# source /etc/profile
[aaa@qq.com-1 ~]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
[aaa@qq.com-1 ~]#
- 查看java版本
[aaa@qq.com-1 native]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
[aaa@qq.com-1 native]#
问题: source 在数据库里还可以用来做什么?
答: 导入
两个datanode节点也需要安装java,可以直接从主节点拷贝过去
203 scp jdk-8u131-linux-x64_.rpm aaa@qq.com192.168.1.11:/root
204 scp jdk-8u131-linux-x64_.rpm aaa@qq.com192.168.1.20:/root
205 scp /etc/profile aaa@qq.com192.168.1.20:/etc/profile
206 scp /etc/profile aaa@qq.com192.168.1.11:/etc/profile
- 在另外两个节点安装java(一台192.168.1.11.另一台192.168.1.20)
[aaa@qq.com-64 ~]# rpm -ivh jdk-8u131-linux-x64_.rpm
```bash
[aaa@qq.com-62 ~]# source /etc/profile
[aaa@qq.com-62 ~]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
6. 创建hadoop用户(三台机器都需要创建hadoop)
```bash
[aaa@qq.com-64 ~]# useradd -u 8000 hadoop ; echo 123456 | passwd --stdin hadoop
更改用户 hadoop 的密码 。
passwd: 所有的身份验证令牌已经成功更新。
- 安装namenode
Hadoop 安装目录:/home/hadoop/hadoop-3.13 使用 root 帐号将 hadoop-3.1.3.tar.gz 上传到服务器,并且放到/home/hadoop下!
1)创建dfs和tmp
rz
[aaa@qq.com-1 hadoop]# su - hadoop
[aaa@qq.com-1 ~]$ whoami
hadoop
$ mkdir -p /home/hadoop/dfs/name /home/hadoop/dfs/data /home/hadoop/tmp
2) 解压
[aaa@qq.com-1 ~]$ tar xvf hadoop-3.1.3.tar.gz
[aaa@qq.com-1 ~]$ cd hadoop-3.1.3
[aaa@qq.com-1 hadoop-3.1.3]$ ll
总用量 200
drwxr-xr-x. 2 hadoop hadoop 4096 9月 12 12:46 bin
drwxr-xr-x. 3 hadoop hadoop 4096 9月 12 10:51 etc
drwxr-xr-x. 2 hadoop hadoop 4096 9月 12 12:46 include
drwxr-xr-x. 3 hadoop hadoop 4096 9月 12 12:46 lib
drwxr-xr-x. 4 hadoop hadoop 4096 9月 12 12:46 libexec
-rw-rw-r--. 1 hadoop hadoop 147145 9月 4 17:31 LICENSE.txt
-rw-rw-r--. 1 hadoop hadoop 21867 9月 4 17:31 NOTICE.txt
-rw-rw-r--. 1 hadoop hadoop 1366 9月 4 17:31 README.txt
drwxr-xr-x. 3 hadoop hadoop 4096 9月 12 10:51 sbin
drwxr-xr-x. 4 hadoop hadoop 4096 9月 12 13:08 share
[aaa@qq.com-1 hadoop]$ pwd
/home/hadoop/hadoop-3.1.3/etc/hadoop
[aaa@qq.com-1 hadoop]$ ls
hadoop-env.sh // java的环境变量
yarn-env.sh //制定yarn框架的Java运行环境
slaves//指定datanode数据存储服务器
core-site.xml // hadoop-web界面路径
hdfs-site.xml //文件系统的配置文件
mapred-site.xml// mapreducer 任务配置文件
yarn-site.xml // yarn框架配置,主要一些任务的启动位置
[aaa@qq.com-1 hadoop]$
3) 修改配置文件
[aaa@qq.com-1 hadoop]$ vim hadoop-env.sh
38 exprot JAVA_HOME=/usr/java/jdk1.8.0_13
[aaa@qq.com-1 hadoop]$ vim yarn-env.sh
27 JAVA_HOME=/usr/java/jdk1.8.0_131
[aaa@qq.com-01 hadoop]$ vim slaves
xinsz08-64
xinsz08-62
[aaa@qq.com-1 hadoop]$ vim core-site.xml
备注:这个是hadoop的核心配置,这里需要配置两属性, fs.default.name 配置hadoop的HDFS系统命令,位置为主机的9000端口, hadoop.tmp.dir 配置haddop的tmp目录的根位置。
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://xinsz08-1:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hadoop/tmp</value>
<description>Abase for other tmporary directries.</description>
</property>
[aaa@qq.com hadoop]$ vim hdfs-site.xml
备注: HDFS主要的配置文件, dfs.http.address配置了hdfs的http的访问位置;
dfs.replication 配置文件的副本,一般不大于从机个数
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>xinsz08-1:9000</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
[aaa@qq.com-1 hadoop]$ vim mapred-site.xml
备注: 这个是mapreduce任务配置文件,mapreduce.framework.name 属性下配置yarn,
mapred.map.tasks和mapred.reduce.tasks 分别为map和reduce 的任务数。同时指定hadoop历史服务器hsitoryserver
我们可以通过historyserver查看mapreduce的作业记录,比如用了多少个map,用了多少个reduce,作业启动时间,作业完成时间。默认清空下,hadoop历史服务器是没有启动的,我们需要通过命令来启动
[aaa@qq.com-1 hadoop]$ /home/hadoop/hadoop-3.1.3/sbin/mr-jobhistory-daemon.sh start historyserver
/home/hadoop/hadoop-3.1.3/etc/hadoop/hadoop-env.sh: line 38: exprot: command not found
WARNING: Use of this script to start the MR JobHistory daemon is deprecated.
WARNING: Attempting to execute replacement "mapred --daemon start" instead.
WARNING: /home/hadoop/hadoop-3.1.3/logs does not exist. Creating.
[aaa@qq.com-1 hadoop]$
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>xinsz08-1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>xinsz08-1:19888</value>
</property>
</configuration>
[aaa@qq.com hadoop]$ vim yarn-site.xml
备注: yarn框架的配置,主要是一些任务的启动位置
<configuration>
<!-- Site specific YARN configuration properties -->
<proetry>
<name>yarn.nodemanager.aux-service</name>
<value>mapreduce_shuffle</value>
</proetry>
<proetry>
<name>yarn.nodemanager.uax-service.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapreduced.ShuffleHandle</value>
</proetry>
<proetry>
<name>yarn.resoucemanager.address</name>
<value>xinsz08-1:8032</value>
</proetry>
<proetry>
<name>yarn.resourcemanager.shceduler.address</name>
<value>xinsz08-1:8030</value>
</proetry>
<proetry>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>xinsz08-1:8031</value>
</proetry>
<proetry>
<name>yarn.resourcemanager.admin.address</name>
<value>xinsz08-1:8033</value>
</proetry>
<proetry>
<name>yarn.resourcemanager.webapp.address</name>
<value>xinsz08-1:8088</value>
</proetry>
</configuration>
- datanode配置文件生成
[aaa@qq.com hadoop]$ scp -r /home/hadoop/hadoop-3.13 aaa@qq.com:~/
[aaa@qq.com hadoop]$ scp -r /home/hadoop/hadoop-3.13 aaa@qq.com:~/
5) namenode格式化数据:
一般第一次的时候需要初始化,之后就不需要了
2020-03-04 16:05:17,247 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 393 bytes saved in 0 seconds .
2020-03-04 16:05:17,268 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2020-03-04 16:05:17,277 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid = 0 when meet shutdown.
2020-03-04 16:05:17,278 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at xinsz08-1/192.168.1.18
************************************************************/
6) 查看是否生成相应的内容
[aaa@qq.com-1 ~]$ cd /home/hadoop/dfs/
[aaa@qq.com-1 dfs]$ ls
data name
[aaa@qq.com-1 dfs]$ tree
.
├── data
└── name
└── current
├── fsimage_0000000000000000000
├── fsimage_0000000000000000000.md5
├── seen_txid
└── VERSION
3 directories, 4 files
7) 配置免密要登录
[aaa@qq.com-1 dfs]$ ssh-******
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
cb:ff:ad:50:92:fb:38:cb:1d:93:ed:7e:3f:54:d7:cd aaa@qq.com-1
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| .o|
| . E|
| S o . o|
| . . + o . |
| o o + o |
| o.= = ..|
| =+=o+.+|
+-----------------+
[aaa@qq.com-1 dfs]$
[aaa@qq.com hadoop]$ ssh-******
[aaa@qq.com hadoop]$ ssh-copy-id zhusw07-64
[aaa@qq.com hadoop]$ ssh-copy-id zhusw07-1 //对自己也做一次
[aaa@qq.com hadoop]$ ssh-copy-id wg61.cn
备注: 方便后期复制文件或者启动服务。 因为namenode启动时候,会链接到datanode上启动对应的服务,
- 启动hdfs
[aaa@qq.com hadoop]$ pwd
/home/hadoop/hadoop-3.1.3/etc/hadoop
报错:2020-03-04 16:16:45,394 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
解答: http://dl.bintray.com/sequenceiq/sequenceiq-bin/ 下载对应版本
解压,覆盖hadoop下/lib/native/
上传之后解压:
[aaa@qq.com-1 native]$ tar xf hadoop-native-64.tar
[aaa@qq.com-1 native]$ ls
examples libhadoop.so.1.0.0 libnativetask.a
hadoop-native-64.tar libhadooputils.a libnativetask.so
libhadoop.a libhdfs.a libnativetask.so.1.0.0
libhadooppipes.a libhdfs.so
libhadoop.so libhdfs.so.0.0.0
覆盖完之后重启
[aaa@qq.com hadoop]$ …/…/sbin/stop-dfs.sh
关闭之后在启动
[aaa@qq.com-1 hadoop]$ ../../sbin/start-dfs.sh
Starting namenodes on [zhusw07-1]
Starting datanodes
Starting secondary namenodes [zhusw07-1]
- 启动yarn
也就是说我们要启动 分布式计算
[aaa@qq.com hadoop]$ pwd
/home/hadoop/hadoop-3.1.3/etc/hadoop
[aaa@qq.com-1 hadoop]$ ../../sbin/start-yarn.sh
除了使用上面的脚本之外,我们还可以使用:
[aaa@qq.com-1 hadoop]$ ../../sbin/start-all.sh
-
启动jobhistory
…/…/sbin/mr-jobhistory-daemon.sh start historyserver -
启动存储服务和资源管理主服务
[aaa@qq.com-1 hadoop-3.1.3]$ sbin/hadoop-daemon.sh start datanode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
[aaa@qq.com-1 hadoop-3.1.3]$ sbin/yarn-daemon.sh start nodemanager
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.
[aaa@qq.com-1 hadoop-3.1.3]$
- Web查看集群状态
浏览器输入http://192.168.1.18:8088