Hadoop 安装及报错解决方法论 hadoophadoop编译安装hadoop安装报错hadoop编译报错
程序员文章站
2024-03-24 15:05:10
...
Da数据时代 ,不掌握点hadoop生态体系可还行。今儿记录下64位centos6.9安装hadoop2.7.6过程。
第一次安装图方便,直接下载了安装包,一路顺利安装完成,执行第一个mapredurce时报错。
意识到64位系统,hadoop编译安装的重要性,重新来过。
一、编译hadoop
1.hadoop-2.7.6-src.tar.gz hadoop官网下载资源包;
2.解压后目录下有个说明文档BUILDING.txt。
按照Requirements的要求装好所有编译需要软件,此处不做细致介绍。清单中每一个都不能少,否则编译过程会让你很痛苦。
Requirements: * Unix System * JDK 1.7+ * Maven 3.0 or later * Findbugs 1.3.9 (if running findbugs) * ProtocolBuffer 2.5.0 * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac * Zlib devel (if compiling native code) * openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance ) * Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs ) * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
3. 开始编译,进入解压目录后 执行:
mvn package -Pdist,native,docs -DskipTests -Dtar
二、遇到问题:
1.编译到hadoop tools时报错退出:
Apache Hadoop tools FAILURE
检查了一般Requirements发现有软件包没准备好
yum install zlib zlib-devel
yum install openssl devel
yum install openssl openssl--devel
重新编译,问题解决。2.编译到Apache Hadoop Pipes FAILURE
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 [ERROR] around Ant part …<exec dir=”/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native” executable=”cmake” failonerror=”true”>… @ 5:125 in /export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml此时继续检查 Requirements,每个软件都已安装。反复尝试了半天没有解决。
只好仔细查看日志,报错信息大概含义是:根据/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml这个文件,编译其中某处时报错。
打开文件,找到exec标签所执行的内容(标红):
root@node-1 bin]# cd /export/ [root@node-1 export]# cd /export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun [root@node-1 antrun]# pwd /export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun [root@node-1 antrun]# ls build-main.xml [root@node-1 antrun]# cat build-main.xml <?xml version="1.0" encoding="UTF-8" ?> <project name="maven-antrun-" default="main" > <target name="main" if="tar"> <echo file="/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/dist-maketar.sh">cd "/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target" tar cf - hadoop-common-2.7.6 | gzip > hadoop-common-2.7.6.tar.gz</echo> <exec failonerror="true" dir="/export/server/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target" executable="sh"> <arg line="./dist-maketar.sh"/> </exec> </target>通过sh 执行./dist-maketar.sh脚本,那么我们就在命令行 执行下该脚本,即可看到具体错误信息,根据错误信息配置相关软件包即可。
ps:本次安装过程中还遇到一个cmake 的报错。
后来发现是 openssl报错,我只安装了 openssl 并没有按装openssl-devel。
二者不同之处是,后者为开发包,编译过程需要后者。
INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 43:57 min [INFO] Finished at: 2018-05-15T08:56:26-07:00 [INFO] Final Memory: 129M/393M [INFO] ------------------------------------------------------------------------编译成功,晒一下。
推荐阅读
-
Hadoop 安装及报错解决方法论 hadoophadoop编译安装hadoop安装报错hadoop编译报错
-
centos下编译安装mysql报错解决方案
-
解决Android Studio编译后安装apk报错:Error while Installing APK
-
centos下编译安装mysql报错解决方案
-
解决安装vs2019后原vs2017项目无法编译各种报错的问题
-
安装Hadoop时Unable to load native-hadoop library for your platform报错原因
-
php编译安装报错:make: *** [sapi/cli/php] Error 1 问题的解决办法
-
hadoop1.2.1下编译及安装eclipse4.2插件
-
编译安装PHP使用session_start()时报错&解决办法
-
更新yum源解决:centos7 安装 Redis ,make编译报错,安装gcc 提示:没有可用软件包 gcc