欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

Hadoop 安装及报错解决方法论 hadoophadoop编译安装hadoop安装报错hadoop编译报错

程序员文章站 2024-03-24 15:05:10
...

Da数据时代 ,不掌握点hadoop生态体系可还行。今儿记录下64位centos6.9安装hadoop2.7.6过程。

 

第一次安装图方便,直接下载了安装包,一路顺利安装完成,执行第一个mapredurce时报错。

意识到64位系统,hadoop编译安装的重要性,重新来过。

一、编译hadoop

1.hadoop-2.7.6-src.tar.gz hadoop官网下载资源包;

2.解压后目录下有个说明文档BUILDING.txt。

按照Requirements的要求装好所有编译需要软件,此处不做细致介绍。清单中每一个都不能少,否则编译过程会让你很痛苦。

 

Requirements:

* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

 3. 开始编译,进入解压目录后 执行:

 

 mvn package -Pdist,native,docs -DskipTests -Dtar

 

二、遇到问题:

1.编译到hadoop tools时报错退出:

Apache Hadoop tools FAILURE
检查了一般Requirements发现有软件包没准备好
yum install zlib zlib-devel
yum install openssl devel
yum install openssl openssl--devel

 重新编译,问题解决。
2.编译到Apache Hadoop Pipes FAILURE
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part …<exec dir=”/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native” executable=”cmake” failonerror=”true”>… @ 5:125 in /export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
此时继续检查 Requirements,每个软件都已安装。反复尝试了半天没有解决。
只好仔细查看日志,报错信息大概含义是:根据/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml这个文件,编译其中某处时报错。
打开文件,找到exec标签所执行的内容(标红):
root@node-1 bin]# cd /export/
[root@node-1 export]# cd /export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun
[root@node-1 antrun]# pwd
/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun
[root@node-1 antrun]# ls
build-main.xml
[root@node-1 antrun]# cat build-main.xml
<?xml version="1.0" encoding="UTF-8" ?>
<project name="maven-antrun-" default="main"  >
<target name="main" if="tar">
  <echo file="/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/dist-maketar.sh">cd &quot;/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target&quot;
                      tar cf - hadoop-common-2.7.6 | gzip &gt; hadoop-common-2.7.6.tar.gz</echo>
  <exec failonerror="true" dir="/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target" executable="sh">
    <arg line="./dist-maketar.sh"/>
  </exec>
</target>
 通过sh 执行./dist-maketar.sh脚本,那么我们就在命令行 执行下该脚本,即可看到具体错误信息,根据错误信息配置相关软件包即可。
 
 
ps:本次安装过程中还遇到一个cmake 的报错。
后来发现是 openssl报错,我只安装了 openssl 并没有按装openssl-devel。
二者不同之处是,后者为开发包,编译过程需要后者。
 
INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 43:57 min
[INFO] Finished at: 2018-05-15T08:56:26-07:00
[INFO] Final Memory: 129M/393M
[INFO] ------------------------------------------------------------------------
 编译成功,晒一下。