spark 连接 kafka 遇到的问题
程序员文章站
2022-06-14 09:08:00
...
java.lang.NoClassDefFoundError: kafka/api/TopicMetadataRequest
at org.apache.spark.streaming.kafka.KafkaCluster.getPartitionMetadata(KafkaCluster.scala:130)
at org.apache.spark.streaming.kafka.KafkaCluster.getPartitions(KafkaCluster.scala:120)
at org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:212)
at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:485)
at com.company.CF_from_kudu$.main(CF_from_kudu.scala:42)
at com.company.CF_from_kudu.main(CF_from_kudu.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)
Caused by: java.lang.ClassNotFoundException: kafka.api.TopicMetadataRequest
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
将kafka_2.12-2.1.0.jar 替换成kafka_2.11-0.8.2.0.jar
./spark-submit --class com.company.CF_from_kudu --master yarn --deploy-mode cluster --executor-memory 6g --executor-cores 2 --jars /data1/testdata/dependencesjar/kafka_2.11-0.8.2.0.jar,/data1/testdata/dependencesjar/metrics-core-2.2.0.jar,/data1/testdata/dependencesjar/spark-streaming-kafka-0-8_2.11-2.3.0.jar /data1/testdata/executorjar/CF-1.0-SNAPSHOT.jar datanode4:9092,datanode5:9092,datanode6:9092 mytopic
上一篇: 导出exec,合并单元格
下一篇: POI合并单元格