spark3.0基于hadoop2.6.0编译问题

        spark3.0出来一段时间了，内部做了很多的优化，所以想尝尝新。

 

        下载下来spark3.0的源码，查看pom.xml文件，发现profile中的hadoop版本是2.7，所以把这个属性改成2.6, 当然我们是cdh5.14.2，hadoop版本是2.6.0。开始编译，发现编译报错，这是因为在2.6.0到2.6.3hadoop中有个class在之后的版本变了，而spark里使用的是之后版本的新API。

 

         找到resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala，把下面源码注释掉，并修改如下：

         

 

    //sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern =>

    //  try {

    //    val logAggregationContext = Records.newRecord(classOf[LogAggregationContext])

    //    logAggregationContext.setRolledLogsIncludePattern(includePattern)

    //    sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern =>

    //      logAggregationContext.setRolledLogsExcludePattern(excludePattern)

    //    }

    //    appContext.setLogAggregationContext(logAggregationContext)

    //  } catch {

    //    case NonFatal(e) =>

    //      logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " +

    //        "does not support it", e)

    //  }

    //}

    //appContext.setUnmanagedAM(isClientUnmanagedAMEnabled)

 

    //sparkConf.get(APPLICATION_PRIORITY).foreach { appPriority =>

    //  appContext.setPriority(Priority.newInstance(appPriority))

    //}

 

 

修改后：

 sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern =>

      try {

        val logAggregationContext = Records.newRecord(classOf[LogAggregationContext])

        val setRolledLogsIncludePatternMethod =

          logAggregationContext.getClass.getMethod("setRolledLogsIncludePattern", classOf[String])

        setRolledLogsIncludePatternMethod.invoke(logAggregationContext, includePattern)

 

        sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern =>

          val setRolledLogsExcludePatternMethod =

            logAggregationContext.getClass.getMethod("setRolledLogsExcludePattern", classOf[String])

          setRolledLogsExcludePatternMethod.invoke(logAggregationContext, excludePattern)

        }

 

        appContext.setLogAggregationContext(logAggregationContext)

      } catch {

        case NonFatal(e) =>

          logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " +

            "does not support it", e)

      }

    }

 

 

为什么要修改？ 我们可以在hadoop源码中找到LogAggregationContext.java，这个类在2.6.3之后的版本是修改了的。可以对比下。