欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

spark3.0基于hadoop2.6.0编译问题

程序员文章站 2024-01-19 10:18:40
...

        spark3.0出来一段时间了,内部做了很多的优化,所以想尝尝新。

 

        下载下来spark3.0的源码,查看pom.xml文件,发现profile中的hadoop版本是2.7,所以把这个属性改成2.6, 当然我们是cdh5.14.2,hadoop版本是2.6.0。开始编译,发现编译报错,这是因为在2.6.0到2.6.3hadoop中有个class在之后的版本变了,而spark里使用的是之后版本的新API。

 

         找到resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala,把下面源码注释掉,并修改如下:

         

 

    //sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern =>

    //  try {

    //    val logAggregationContext = Records.newRecord(classOf[LogAggregationContext])

    //    logAggregationContext.setRolledLogsIncludePattern(includePattern)

    //    sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern =>

    //      logAggregationContext.setRolledLogsExcludePattern(excludePattern)

    //    }

    //    appContext.setLogAggregationContext(logAggregationContext)

    //  } catch {

    //    case NonFatal(e) =>

    //      logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " +

    //        "does not support it", e)

    //  }

    //}

    //appContext.setUnmanagedAM(isClientUnmanagedAMEnabled)

 

    //sparkConf.get(APPLICATION_PRIORITY).foreach { appPriority =>

    //  appContext.setPriority(Priority.newInstance(appPriority))

    //}

 

 

修改后:

 sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern =>

      try {

        val logAggregationContext = Records.newRecord(classOf[LogAggregationContext])

        val setRolledLogsIncludePatternMethod =

          logAggregationContext.getClass.getMethod("setRolledLogsIncludePattern", classOf[String])

        setRolledLogsIncludePatternMethod.invoke(logAggregationContext, includePattern)

 

        sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern =>

          val setRolledLogsExcludePatternMethod =

            logAggregationContext.getClass.getMethod("setRolledLogsExcludePattern", classOf[String])

          setRolledLogsExcludePatternMethod.invoke(logAggregationContext, excludePattern)

        }

 

        appContext.setLogAggregationContext(logAggregationContext)

      } catch {

        case NonFatal(e) =>

          logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " +

            "does not support it", e)

      }

    }

 

 

为什么要修改? 我们可以在hadoop源码中找到LogAggregationContext.java,这个类在2.6.3之后的版本是修改了的。可以对比下。

 

相关标签: spark hadoop