kafka架构讲解
kafka架构
producer:生产者
comsumer:消费者
broker:中间人–经纪人
topic:主题、类别
kafka是基于zookeeper进行管理的，所以要先安装zookeeper
zookeeper下载安装
下载版本：cdh5.7.0
上传：rz
解压：tar -zvxf zookeeper-3.4.5-cdh5.7.0.tar.gz -C ~/soft_install/
配置环境变量：vi ~/.bash_profile
export ZK_HOME=/root/soft_install/zookeeper-3.4.5-cdh5.7.0
export PATH=$ZK_HOME/bin:$PATH
修改配置文件
cd conf/
cp zoo_sample.cfg zoo.cfg
mkdir /root/soft_workspace/zookeeper/tmp
vi zoo.cfg --> dataDir = /root/soft_workspace/zookeeper/tmp
启动zookeeper：
默认端口2181
./zkServer.sh start
jps -->QuorumPeerMain
启动client: ./zkCli.sh
kafka下载安装
上传：rz
解压：tar -zvxf kafka_2.11-0.9.0.0.tgz -C ~/soft_install/
配置环境变量：vi ~/.bash_profile
export KAFKA_HOME=/root/soft_install/kafka_2.11-0.9.0.0
export PATH=$KAFKA_HOME/bin:$PATH
修改配置文件：
cd config/
mkdir ~/soft_workspace/kafka/kafka-logs
vi server.properties
>### broker.id=0
>### host.name = 192.168.145.128
>### log.dirs = /root/soft_workspace/kafka/kafka-logs
>### zookeeper.connection= 192.168.145.128:2181
默认端口是:9092
启动server:
kafka-server-start.sh -daemon $KAFKA_HOME/config/zookeeper.properties
jps -->Kafka
基本操作(单节点单broker)
创建topic：
kafka-topics.sh --create --zookeeper 192.168.145.128:2181 --replication-factor 1 --partitions 1 --topic firstTopic
查看topic:
kafka-topics.sh --list --zookeeper 192.168.145.128:2181
生成消息(producer)
kafka-console-producer.sh --broker-list 192.168.145.128:9092 --topic firstTopic
消费消息(comsumer)
kafka-console-consumer.sh --zookeeper 192.168.145.128:2181 --topic firstTopic --from-beginning
单节点多broker
复制配置文件
cp server.properties server-1.properties
cp server.properties server-2.properties
cp server.properties server-3.properties
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/root/soft_workspace/kafka/kafka-logs-1
启动kafka:
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &
创建topic
kafka-topics.sh --create --zookeeper 192.168.145.128:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
查看topic:
列表：kafka-topics.sh --list --zookeeper 192.168.145.128:2181
详细信息：kafka-topics.sh --describe --zookeeper 192.168.145.128:2181 --topic my-replicated-topic
生成消息(producer)
kafka-console-producer.sh --broker-list 192.168.145.128:9093,192.168.145.128:9094,192.168.145.128:9095 --topic my-replicated-topic
消费消息(comsumer)
kafka-console-consumer.sh --zookeeper 192.168.145.128:2181 --from-beginning --topic my-replicated-topic
容错性
从单节点多broker(多篮子)测试
slaver进程可以删除，consumer会出现warn级别的日志
当leader进程被删除时，会自动从slaver中选择新的leader;
当所有的slaver及leader都被删除时，producer和comsumer会报错
进程
查看进程：jps -m
杀死进程：kill -9 进程号