欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

spark eclipse写wordcount

程序员文章站 2022-05-22 17:06:50
...

安装spark,见上文

http://blackproof.iteye.com/blog/2182393

 

配置window开发环境

window安装scala

下载scala http://www.scala-lang.org/files/archive/scala-2.10.4.msi

安装即可

 

window配置eclipse

下载eclipse  

http://downloads.typesafe.com/scalaide-pack/4.0.0-vfinal-luna-211-20150119/scala-SDK-4.0.0-vfinal-2.11-win32.win32.x86_64.zip

解压即可

 

写wordcount scala代码

创建scala 项目

新建scala object,命名WordCount

代码如下

package com.dirk.test

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext.rddToPairRDDFunctions
import scala.collection.mutable.ListBuffer
import org.apache.spark.SparkConf


object WordCount {  
    def main(args: Array[String]){  
      if(args.length != 3){  
        println("usage: com.qiurc.test.WordCount <master> <input> <output>")  
        return  
      }  
      val jars = ListBuffer[String]()
      jars.+=("/home/hadoop-cdh/app/test/sparktest/aa.jar") //aa.jar发布位置
      val conf = new SparkConf()
      
      conf.setMaster(args(0))//设置spark master url
      
      .setAppName("word count")
      
      .setJars(jars)//解决找不到jar包的问题
      
      .set("spark.executor.memory","200m")
      
      val sc = new SparkContext(conf)
      

      val textFile  = sc.textFile(args(1))  
      val result = textFile.flatMap(_.split(" "))  
              .map(word => (word, 1)).reduceByKey(_ + _)  
      result.saveAsTextFile(args(2))  
        
    }  
}  

 打包scala项目,和java项目打jar相同,名为aa.jar,

  jar包发布位置为

/home/hadoop-cdh/app/test/sparktest/aa.jar

 

  发布到spark服务器上

  写运行脚本

#!/usr/bin/env bash  

$SPARK_HOME/bin/spark-submit --name SparkWordCount --class com.dirk.test.WordCount --master spark://host143:7077 --executor-memory 512M --total-executor-cor
es 1 aa.jar spark://host143:7077 hdfs://XXX/user/dirk.zhang/data/word.txt hdfs://XXX/user/dirk.zhang/output

遇到的问题

1.参数解释 参数1为spark master url,参数2为hdfs输入,参数3位hdfs输出,xxx为HA的defaultFS

2.找不到class文件的原因是缺少setJars(jars)

 

参考

http://bit1129.iteye.com/blog/2172164 

http://www.tuicool.com/articles/qq2mQj