Linux安装Kafka和简单使用

1、下载https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz

2、解压
tar -xvf kafka_2.11-2.0.0.tgz

3、设置环境变量

• vim /etc/profile 或者 vim ~/.bashrc

#Kafka Environment
export KAFKA_HOME=$USER_HOME/software/kafka_2.11-2.0.0
source /etc/profile 或者 source ~/.bashrc

4、修改配置文件

• vim $KAFKA_HOME/config/server.properties

#listeners=PLAINTEXT://:9092

改成

listeners=PLAINTEXT://:9092

log.dirs=/tmp/kafka-logs

改成

log.dirs=/home/hadoop/tmp/kafka-logs

使用非自带的Zookeeper，不用改动下面的配置

• vim $KAFKA_HOME/config/zookeeper.properties

dataDir=/tmp/zookeeper

改成

dataDir=/home/hadoop/tmp/zookeeper
dataLogDir=/home/hadoop/logs/zookeeper

5、原始启动

• 启动Zookeeper

nohup ~/software/kafka_2.11-2.0.0/bin/zookeeper-server-start.sh   $KAFKA_HOME/config/zookeeper.properties &

• 启动Kafka

nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh   $KAFKA_HOME/config/server.properties &

6、起停脚本(需提前安装zookeeper)

~/software/kafka_2.11-2.0.0/bin/start-kafka.sh
~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh

• vim ~/software/kafka_2.11-2.0.0/bin/start-kafka.sh

#Start Zookeeper
nohup ~/software/kafka_2.11-2.0.0/bin/zookeeper-server-start.sh  $KAFKA_HOME/config/zookeeper.properties &
#Start Kafka
nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

• vim ~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh

#Stop Kafka
nohup  ~/software/kafka_2.11-2.0.0/bin/kafka-server-stop.sh $KAFKA_HOME/config/server.properties &
#Stop Zookeeper
nohup ~/software/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh  $KAFKA_HOME/config/zookeeper.properties &

使用非自带的Zookeeper，不用改动下面的配置

• vim ~/software/kafka_2.11-2.0.0/bin/start-kafka.sh

#Start Kafka
#nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log &
nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log 2>&1 &

• vim ~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh

#Stop Kafka
#nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-stop.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log &
nohup ~/software/kafka_2.11-2.0.0/bin/kafka-server-stop.sh $KAFKA_HOME/config/server.properties >$USER_HOME/logs/kafka/kafka_start.log 2>&1 &

7、添加执行权限

chmod +x ~/software/kafka_2.11-2.0.0/bin/start-kafka.sh ~/software/kafka_2.11-2.0.0/bin/stop-kafka.sh

8、Kafka常用命令
创建Topic

~/software/kafka_2.11-2.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看创建的Topic

~/software/kafka_2.11-2.0.0/bin/kafka-topics.sh -list -zookeeper localhost:2181

启动生产者

~/software/kafka_2.11-2.0.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

在另外一个终端启动消费者

~/software/kafka_2.11-2.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

9、FAQ
[2019-04-23 16:07:30,396] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
rm -rf /home/hadoop/tmp/kafka-logs
清理日志信息