Spark Streaming实时流处理项目实战笔记——Kafka实战之整合Flume和Kafka完成实时数据采集

整体架构拓扑图 

Agent1 错误（exec source 不是avro source）

一号机配置文件

监听/opt/flume/flume/data/data.log作为flume线生产者

a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /opt/flume/flume/data/data.log
a1.sources.r1.shell = /bin/sh -c

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop2
a1.sinks.k1.port = 44444

a1.channels.c1.type = memory


a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

二号机配置文件

消费一号机sink信息，并输出给kafka

[[e-mail] conf]# more avro-memory-kafka.conf 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop2
a1.sources.r1.port = 44444

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList = hadoop:9092
a1.sinks.k1.topic = zz
a1.sinks.k1.batchSize = 5
a1.sinks.k1.requiredAcks = 1

a1.channels.c1.type = memory


a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

二号机启用Flume
 

flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/avro-memory-kafka.conf -Dflume.root.logger=INFO,console

一号机启用Flume

flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-avro.conf -Dflume.root.logger=INFO,console

一号机启用Kafka消费消息

kafka-console-consumer.sh --bootstrap-server hadoop:9092 --from-beginning  --topic zz

一号机向/opt/flume/flume/data/data.log中追加信息，并查看一号机Kafka消费端打印信息