kafka架构讲解

kafka架构

producer:生产者

comsumer:消费者

broker:中间人–经纪人

topic:主题、类别

kafka是基于zookeeper进行管理的，所以要先安装zookeeper

zookeeper下载安装

下载版本：cdh5.7.0

上传：rz

解压：tar -zvxf zookeeper-3.4.5-cdh5.7.0.tar.gz -C ~/soft_install/

配置环境变量：vi ~/.bash_profile

export ZK_HOME=/root/soft_install/zookeeper-3.4.5-cdh5.7.0

export PATH=$ZK_HOME/bin:$PATH

修改配置文件

cd conf/

cp zoo_sample.cfg zoo.cfg

mkdir /root/soft_workspace/zookeeper/tmp

vi zoo.cfg --> dataDir = /root/soft_workspace/zookeeper/tmp

启动zookeeper：

默认端口2181

./zkServer.sh start

jps -->QuorumPeerMain

启动client: ./zkCli.sh

kafka下载安装

上传：rz

解压：tar -zvxf kafka_2.11-0.9.0.0.tgz -C ~/soft_install/

配置环境变量：vi ~/.bash_profile

export KAFKA_HOME=/root/soft_install/kafka_2.11-0.9.0.0

export PATH=$KAFKA_HOME/bin:$PATH

修改配置文件：

cd config/

mkdir ~/soft_workspace/kafka/kafka-logs

vi server.properties

>### broker.id=0

>### host.name = 192.168.145.128

>### log.dirs = /root/soft_workspace/kafka/kafka-logs

>### zookeeper.connection= 192.168.145.128:2181

默认端口是:9092

启动server:

kafka-server-start.sh -daemon $KAFKA_HOME/config/zookeeper.properties

jps -->Kafka

基本操作(单节点单broker)

创建topic：

kafka-topics.sh --create --zookeeper 192.168.145.128:2181 --replication-factor 1 --partitions 1 --topic firstTopic

查看topic:

kafka-topics.sh --list --zookeeper 192.168.145.128:2181

生成消息(producer)

kafka-console-producer.sh --broker-list 192.168.145.128:9092 --topic firstTopic

消费消息(comsumer)

kafka-console-consumer.sh --zookeeper 192.168.145.128:2181 --topic firstTopic --from-beginning

单节点多broker

复制配置文件

cp server.properties server-1.properties

cp server.properties server-2.properties

cp server.properties server-3.properties

broker.id=1

listeners=PLAINTEXT://:9093

log.dirs=/root/soft_workspace/kafka/kafka-logs-1

启动kafka:

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &

创建topic

kafka-topics.sh --create --zookeeper 192.168.145.128:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

查看topic:

列表：kafka-topics.sh --list --zookeeper 192.168.145.128:2181

详细信息：kafka-topics.sh --describe --zookeeper 192.168.145.128:2181 --topic my-replicated-topic

生成消息(producer)

kafka-console-producer.sh --broker-list 192.168.145.128:9093,192.168.145.128:9094,192.168.145.128:9095 --topic my-replicated-topic

消费消息(comsumer)

kafka-console-consumer.sh --zookeeper 192.168.145.128:2181 --from-beginning --topic my-replicated-topic

容错性

从单节点多broker(多篮子)测试

slaver进程可以删除，consumer会出现warn级别的日志

当leader进程被删除时，会自动从slaver中选择新的leader;

当所有的slaver及leader都被删除时，producer和comsumer会报错

进程

查看进程：jps -m

杀死进程：kill -9 进程号