centos6.5 安装hadoop1.2.1的教程详解【亲测版】
本篇只简单介绍安装步骤
1. 角色分配
10.11.84.4 web-crawler--1.novalocal master/slave 10.11.84.5 web-crawler--2.novalocal slave 10.11.84.6 web-crawler--3.novalocal slave
2.安装目录 /usr/local/hadoop (hadoop_home)
3,创建group hadoop
groupadd hadoop
4.创建用户hadoop
useradd -d /home/hadoop -g hadoop hadoop chown -r hadoop:hadoop /usr/local/hadoop
5. 每台机器安装jdk(安装在相同的目录)
1)查看java 版本,可以看到jdk为openjdk,最好更换为sun jdk
[hadoop@hadoop2 ~]$ java -version openjdk version "1.8.0_131" openjdk runtime environment (build 1.8.0_131-b12) openjdk 64-bit server vm (build 25.131-b12, mixed mode) [hadoop@hadoop2 ~]$ rpm -qa | grep java java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 javapackages-tools-3.4.1-11.el7.noarch tzdata-java-2017b-1.el7.noarch java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 python-javapackages-3.4.1-11.el7.noarch
2)卸载openjdk
rpm -qa | grep java | xargs yum -y remove
3)安装sunjdk
/usr/local 目录下解压jdk-8u181-linux-i586.tar.gz 并命名为jdk
配置环境变量如下
vim /etc/profile 末尾添加如下内容
export java_home=/usr/local/jdk export jre_home=${java_home}/jre export path=${java_home}/bin:${path} export classpath=.:${java_home}/lib:${jre_home}/lib:${classpath}
4)查看jdk是否安装成功
[root@hadoop2 local]# java -version java version "1.8.0_151" java(tm) se runtime environment (build 1.8.0_151-b12) java hotspot(tm) 64-bit server vm (build 25.151-b12, mixed mode)
6. master 到 slave ssh 免密码登录
在每台机器hadoop用户下执行如下命令
[hadoop@hadoop1 ~]$ ssh-keygen -t rsa -p '' #一路直接回车 发送master 机器~/.ssh/id_rsa.pub 到所有slave节点 scp id_rsa.pub hadoop@10.11.84.5:~ scp id_rsa.pub hadoop@10.11.84.6:~ 让后在每台slave节点hadoop 用户下执行如下命令 cd ~/.ssh cat ../id_rsa.pub >> authorized_keys chmod 644 authorized_keys
7.关闭所有机器防火墙、selinux
1)centos 7 以下
防火墙
查看防火墙状态
service iptables status
临时关闭防火墙
service iptables stop
永久关闭防火墙
chkconfig iptables off
selinux
setenforce 0 #临时关闭
永久关闭需要修改如下文件
vim /etc/selinux/config
将selinux=enforcing改为selinux=disabled
2)centos 7
firewall-cmd --reload #重启firewall
systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动
firewall-cmd --state #查看默认防火墙状态(关闭后显示notrunning,开启后显示running)
selinux 同上
8.修改hadoop配置文件
core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" rel="external nofollow" rel="external nofollow" ?> <!-- put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value> <description>abase for other temporary directories.</description> </property> <property> <name>fs.default.name</name> <value>hdfs://web-crawler--1.novalocal:9000</value> </property> </configuration> hadoop-env.sh 添加java_home export java_home=/usr/local/jdk hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" rel="external nofollow" rel="external nofollow" ?> <!-- put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.name.dir</name> <value>/home/hadoop/name</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/data</value> </property> </configuration> mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" rel="external nofollow" rel="external nofollow" ?> <!-- put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>http://web-crawler--1.novalocal:9001</value>;;; </property> <property> <name>mapred.system.dir</name> <value>/home/hadoop/mapred/system</value> </property> <property> <name>mapred.local.dir</name> <value>/home/hadoop/mapred/local</value> </property> </configuration>
5)masters
web-crawler--1.novalocal
6)slaves
web-crawler--1.novalocal web-crawler--2.novalocal web-crawler--3.novalocal
9. 修改环境变量,添加hadoop_home 和java_home
vim /etc/profile (root) export ant_home=/usr/local/ant export path=${path}:${ant_home}/bin export hadoop_home=/usr/local/hadoop-1.2.1 export java_home=/usr/local/jdk export path=$java_home/bin:$java_home/jre/bin:$path export classpath=$java_home/lib:$java_home/jre/lib:$classpath export path=/usr/local/lib/phantomjs/bin:$path export ant_home=/usr/local/ant export path=${path}:${ant_home}/bin export path=$path:${hadoop_home}/bin:${hadoop_home}/sbin
10.启动hadoop
1)在master上format namenode
hadoop namenode -format
2)启动hadoop
sh /user/local/hadoop/bin/start-all.sh
11.进行简单测试
hadoop fs -mkdir -p /user/hadoop/private/in hadoop fs -put /user/local/hadoop/bin/start-all.sh /user/hadoop/private/in cd /usr/local/hadoop hadoop jar hadoop-examples-1.2.1.jar wordcount /user/hadoop/private/in /user/hadoop/private/out
总结
以上所述是小编给大家介绍的centos6.5 安装hadoop1.2.1的教程详解,希望对大家有所帮助
上一篇: CentOS7设置定时任务
下一篇: 服务器安全设置之 IIS用户设置方法
推荐阅读
-
mysql 8.0.12 解压版安装教程 个人亲测!
-
centos6.5 安装hadoop1.2.1的教程详解【亲测版】
-
win10安装zip版MySQL8.0.19的教程详解
-
最新IntelliJ IDEA 2020版本的安装教程详解
-
Navicat Premium15安装及破解教程详解亲测有效(附破解失败解决方案)
-
win10 安装 mysql 5.7 msi版的教程图文详解
-
RedHat6.5/CentOS6.5安装Mysql5.7.20的教程详解
-
MySQL 5.6.36 Windows x64位版本的安装教程详解
-
Ubuntu 18.04.4安装mysql的过程详解 亲测可用
-
Goland激活码破解永久版及安装详细教程(亲测可以)