spark3.0基于hadoop2.6.0编译问题
程序员文章站
2024-01-19 10:18:40
...
spark3.0出来一段时间了,内部做了很多的优化,所以想尝尝新。
下载下来spark3.0的源码,查看pom.xml文件,发现profile中的hadoop版本是2.7,所以把这个属性改成2.6, 当然我们是cdh5.14.2,hadoop版本是2.6.0。开始编译,发现编译报错,这是因为在2.6.0到2.6.3hadoop中有个class在之后的版本变了,而spark里使用的是之后版本的新API。
找到resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala,把下面源码注释掉,并修改如下:
//sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern => // try { // val logAggregationContext = Records.newRecord(classOf[LogAggregationContext]) // logAggregationContext.setRolledLogsIncludePattern(includePattern) // sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern => // logAggregationContext.setRolledLogsExcludePattern(excludePattern) // } // appContext.setLogAggregationContext(logAggregationContext) // } catch { // case NonFatal(e) => // logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " + // "does not support it", e) // } //} //appContext.setUnmanagedAM(isClientUnmanagedAMEnabled) //sparkConf.get(APPLICATION_PRIORITY).foreach { appPriority => // appContext.setPriority(Priority.newInstance(appPriority)) //}
修改后:
sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern => try { val logAggregationContext = Records.newRecord(classOf[LogAggregationContext]) val setRolledLogsIncludePatternMethod = logAggregationContext.getClass.getMethod("setRolledLogsIncludePattern", classOf[String]) setRolledLogsIncludePatternMethod.invoke(logAggregationContext, includePattern) sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern => val setRolledLogsExcludePatternMethod = logAggregationContext.getClass.getMethod("setRolledLogsExcludePattern", classOf[String]) setRolledLogsExcludePatternMethod.invoke(logAggregationContext, excludePattern) } appContext.setLogAggregationContext(logAggregationContext) } catch { case NonFatal(e) => logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " + "does not support it", e) } }
为什么要修改? 我们可以在hadoop源码中找到LogAggregationContext.java,这个类在2.6.3之后的版本是修改了的。可以对比下。