Mac Spark 运行 wordcount 程序
程序员文章站
2022-06-14 14:59:24
...
1. mac 安装 spark
略
2. 安装sbt
brew install sbt
3. 写wordcount scala程序
import org.apache.spark.{SparkConf, SparkContext}
object SparkWordCount {
def FILE_NAME:String = "word_count_results_";
def main(args:Array[String]): Unit ={
if(args.length < 1){
println("Usage:SparkWordCount FileName");
System.exit(1);
}
val conf = new SparkConf().setAppName("Spark Exercise: Spark Version Word Count Program");
val sc = new SparkContext(conf);
val textFile = sc.textFile(args(0));
val wordCounts = textFile.flatMap(line => line.split(" ")).map(
word => (word, 1)
).reduceByKey((a, b) => a + b)
wordCounts.saveAsTextFile(FILE_NAME + System.currentTimeMillis());
println("Word Count program running results are successfully saved.");
}
}
3. sbt 文件
name := "SparkWordCount" version := "1.0" scalaVersion := "2.11.8" libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
这边的scala的版本和spark的版本要根据你自己本地的版本进行修改
4. 目录结构
除了箭头的,其他忽略,是编译生成的。
5. 编译
sbt package
6. 提交到spark 运行
spark-submit --class SparkWordCount ./target/scala-2.11/sparkwordcount_2.11-1.0.jar /usr/local/Cellar/spark-2.3.0/README.md
这边传入的文件要是你本地有的。
http://www.codeblogbt.com/archives/144037
推荐阅读
-
Spark程序运行报错:JVM申请的memory不足解决办法
-
三方模拟软件上线:M1处理器Mac已可运行Windows程序及游戏
-
运行Spark程序时出现 Exception in thread “main“ org.apache.spark.SparkException: A master URL must be set
-
Flink(三) —— Mac 上搭建 Flink 1.6.0 环境并构建运行简单程序入门
-
hadoop平台运行WordCount程序
-
spark stream读取kafka wordcount程序
-
将java开发的wordcount程序部署到spark集群上运行
-
MapReduce运行wordcount程序
-
Mac Spark 运行 wordcount 程序
-
CDH 开发运行Spark wordcount程序