Flume Sink到Kafka遇到的问题
程序员文章站
2022-06-14 09:08:18
...
1、版本问题
2018-12-02 15:23:06,334 (conf-file-poller-0) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:427)] Sink k1 has been removed due to an error during configuration
org.apache.flume.conf.ConfigurationException: brokerList must contain at least one Kafka broker
at org.apache.flume.sink.kafka.KafkaSinkUtil.addDocumentedKafkaProps(KafkaSinkUtil.java:55)
at org.apache.flume.sink.kafka.KafkaSinkUtil.getKafkaProperties(KafkaSinkUtil.java:37)
at org.apache.flume.sink.kafka.KafkaSink.configure(KafkaSink.java:217)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
查看官网使用了最新的Flume版本(1.8.0)的配置信息,导致了上面的一个错误,数据无法写入Kafka。
使用相对应的版本的配置即可。
所以配置里的a1.sinks.k1.kafka.bootstrap.servers = master:9092,slave1:9092,slave2:9092
换成a1.sinks.k1.brokerList= master:9092,slave1:9092,slave2:9092
Flume1.8.0有以下过期配置:
Deprecated Properties
Property Name | Default | Description |
---|---|---|
brokerList | – | Use kafka.bootstrap.servers |
topic | default-flume-topic | Use kafka.topic |
batchSize | 100 | Use kafka.flumeBatchSize |
requiredAcks | 1 | Use kafka.producer.acks |
其他可能因为版本出现的警告:
WARN kafka.KafkaSink: The Property 'topic' is not set. Using the default topic name: default-flume-topic
上一篇: PHPExcel导出纵向合并单元格
下一篇: 最近遇到的kafka对接问题
推荐阅读
-
从.NET CORE2.2升级到3.0过程及遇到的一些问题
-
从零到一详聊创建Vue工程及遇到的常见问题
-
springboot项目打成war包部署到tomcat遇到的一些问题
-
从零到一详聊创建Vue工程及遇到的常见问题
-
如何升级到Windows10系统?升级win10系统遇到的问题解决汇总(含视频体验)
-
vue项目部署到Apache服务器中遇到的问题解决
-
从.NET CORE2.2升级到3.0过程及遇到的一些问题
-
.Net Core 发布到 Linux 上时遇到的问题记录
-
Mysql升级到5.7后遇到的group by查询问题解决
-
Python中LOADDATAINFILE语句导入数据到MySQL遇到问题的解决方案分享