Exception in thread “main“ org.apache.spark.SparkException: Task not serializable
程序员文章站
2022-07-15 12:55:06
...
Exception in thread “main” org.apache.spark.SparkException: Task not serializable
原因:
当需要在Driver之间传递变量时,这个变量是需要可以序列化的
错误代码
错误分析:
获取和关闭jedis连接应该在分区内进行,此处相当于在driver处获取了连接,所以需要序列化jedis变量.
解决方法:
把jedis连接获取放入分区内即可解决此问题.
代码
val filteredStream: DStream[JSONObject] = jsonObjDStream.mapPartitions {
//获取和关闭jedis连接应该在分区内进行,此处相当于在driver处获取了连接,所以需要序列化jedis变量
val jedis: Jedis = RedisUtil.getJedisClient
datas => {
println("该分区过滤前数据量-----" + datas.size)
val filterd: Iterator[JSONObject] = datas.filter(
jsonObj => {
val dt: String = jsonObj.getString("dt")
val mid: String = jsonObj.getJSONObject("common").getString("mid")
val dau: String = "dau:" + dt
val isNew: lang.Long = jedis.sadd(dau, mid)
if (isNew == 1L) true else false
}
)
jedis.close()
println("该分区过滤后数据量*******" + filterd.size)
filterd
}
}
修正代码
val filteredStream: DStream[JSONObject] = jsonObjDStream.mapPartitions {
datas => {
//只用修改此处即可------>
val jedis: Jedis = RedisUtil.getJedisClient
println("该分区过滤前数据量-----" + datas.size)
val filterd: Iterator[JSONObject] = datas.filter(
jsonObj => {
val dt: String = jsonObj.getString("dt")
val mid: String = jsonObj.getJSONObject("common").getString("mid")
val dau: String = "dau:" + dt
val isNew: lang.Long = jedis.sadd(dau, mid)
if (isNew == 1L) true else false
}
)
jedis.close()
println("该分区过滤后数据量*******" + filterd.size)
filterd
}
}
上一篇: org.apache.spark.SparkException: Task not serializable
下一篇: Exception in thread "main" org.apache.spark.SparkException: Task not serializable
推荐阅读
-
storm报错:Exception in thread "main" java.lang.RuntimeException: InvalidTopologyException(msg:Component: [mybolt] subscribes from non-existent
-
JDBC:Exception in thread “main“ java.sql.SQLException: The server time zone value ‘�й���ʱ�报错
-
归并排序栈溢出异常Exception in thread "main" java.lang.*Error
-
Exception in thread “main“ java.lang.*Error ——Spark栈溢出解决方案
-
org.apache.spark.SparkException: Task not serializable
-
org.apache.spark.SparkException: Task not serializable
-
spark 程序 org.apache.spark.SparkException: Task not serializable
-
spark学习-52-Spark的org.apache.spark.SparkException: Task not serializable
-
org.apache.spark.SparkException: Task not serializable
-
Exception in thread “main“ org.apache.spark.SparkException: Task not serializable