欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

spark 连接 kafka 遇到的问题

程序员文章站 2022-06-14 09:08:00
...
java.lang.NoClassDefFoundError: kafka/api/TopicMetadataRequest
        at org.apache.spark.streaming.kafka.KafkaCluster.getPartitionMetadata(KafkaCluster.scala:130)
        at org.apache.spark.streaming.kafka.KafkaCluster.getPartitions(KafkaCluster.scala:120)
        at org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:212)
        at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:485)
        at com.company.CF_from_kudu$.main(CF_from_kudu.scala:42)
        at com.company.CF_from_kudu.main(CF_from_kudu.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)
Caused by: java.lang.ClassNotFoundException: kafka.api.TopicMetadataRequest
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

将kafka_2.12-2.1.0.jar 替换成kafka_2.11-0.8.2.0.jar

./spark-submit --class com.company.CF_from_kudu --master yarn --deploy-mode cluster --executor-memory 6g --executor-cores 2 --jars /data1/testdata/dependencesjar/kafka_2.11-0.8.2.0.jar,/data1/testdata/dependencesjar/metrics-core-2.2.0.jar,/data1/testdata/dependencesjar/spark-streaming-kafka-0-8_2.11-2.3.0.jar /data1/testdata/executorjar/CF-1.0-SNAPSHOT.jar datanode4:9092,datanode5:9092,datanode6:9092 mytopic