欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  数据库

hadoop2 单机搭建

程序员文章站 2022-03-19 19:38:19
...

Hadoop2.2.0 环境配置说明(多节点配置在最后) 1.关闭防火墙 #chkconfigiptablesoff 2.检查状态 #chkconfig –list|grepiptables 全 off 即可 3.将 hadoop-2.2.0.tar.gz 文件复制到 /usr/local 目录下 4.解压 #tar –zxvfhadoop-2.2.0.tar.gz 5.改名 #mv Ha

Hadoop2.2.0环境配置说明(多节点配置在最后)

1. 关闭防火墙# chkconfig iptables off

2. 检查状态#chkconfig –list|grep iptables off即可

3. 将hadoop-2.2.0.tar.gz文件复制到/usr/local目录下

4. 解压# tar –zxvf hadoop-2.2.0.tar.gz

5. 改名 # mv Hadoop-2.2.0 hadoop2.2

6. 修改环境变量 # vim /etc/profile

7. 添加 export HADOOP_HOME=/usr/local/hadoop2.2

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

PATH下添加:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

8. 重载# source /etc/profile

9. 目录切换到 # cd /usr/local/hadoop2.2/etc/Hadoop

10. 在如下文件中添加对应内容

11. hadoop-env.sh 27行修改为

export JAVA_HOME=/usr/local/jdk1.6

12. yarn-env.sh 23行修改为

export JAVA_HOME=/usr/local/jdk1.6

13. 将mapred-site.xml.template 复制为mapred-site.xml

# cp mapred-site.xml.template mapred-site.xml

14. mapred-site.xml中第20(configuration)添加

mapreduce.framework.name

yarn

15. yarn-site.xml18(configuration)添加

yarn.resourcemanager.hostname

localhost

hostanem of RM

yarn.resourcemanager.resource-tracker.address

localhost:5274

host is the hostname of the resource manager and

port is the port on which the NodeManagers contact the Resource Manager.

yarn.resourcemanager.scheduler.address

localhost:5273

host is the hostname of the resourcemanager and port is the port

on which the Applications in the cluster talk to the Resource Manager.

yarn.resourcemanager.scheduler.class

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

In case you do not want to use the default scheduler

yarn.resourcemanager.address

localhost:5271

the host is the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager.

yarn.nodemanager.local-dirs

the local directories used by the nodemanager

yarn.nodemanager.address

localhost:5272

the nodemanagers bind to this port

yarn.nodemanager.resource.memory-mb

10240

the amount of memory on the NodeManager in GB

yarn.nodemanager.remote-app-log-dir

/app-logs

directory on hdfs where the application logs are moved to

yarn.nodemanager.log-dirs

/usr/log

the directories used by Nodemanagers as log directories

yarn.nodemanager.aux-services

mapreduce_shuffle

shuffle service that needs to be set for Map Reduce to run

16. core-site.xml20(configuration)添加

hadoop.tmp.dir

/usr/local/hadoop2.2/tmp

fs.defaultFS

hdfs://localhost:9000

true

17. hdfs-site.xml20(configuration)添加

dfs.namenode.name.dir

file:///dfs/name

true

dfs.datanode.data.dir

file:///dfs/data

true

dfs.replication

1

dfs.permissions.enabled

false

hadoop-env.sh里面# export JAVA_HOME=/usr/local/jdk

18. 设置ssh

# ssh-keygen –t rsa 一路回车默认值

进行查看 (应有id_rsaid_rsa.pub一对密钥文件)

# cd ~/.ssh

# ls

复制出公钥

# cp id_rsa.pub authorized_keys

查看# ls 应有三个文件了

确认过程

# ssh localhost (输入yes)

# exit

# ssh localhost

19. 格式化

# hadoop namenode –format

20. 启动

#start-dfs.sh

#start-yarn.sh

21. 查看 # jps 应有6

22. 用自带浏览器,能打开http://localhost:50070/ 和http://localhost:8088/cluster即可

(多机环境配置)

23. DNS配置(建议修改之后重启虚拟机)

24. 将主机名换为hadoop2

# vim /etc/sysconfig/network

2行,localhost.localdomain改为hadoop2

25. 修改地址映射

# vim /etc/hosts

第三行添加192.168.100.11 hadoop2 (地址不固定)