欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

kafka Windows集群配置实践

程序员文章站 2022-07-13 08:35:32
...

Windows下集群安装

1.安装jdk。设置java_home,然后把%java_home%\bin放到path环境变量里。

2.安装配置zookeeper

#缺少这个有时启动zookeeper会报错,建议加上
admin.serverPort=7070
#两台windows机器的ip和端口
#2888为服务器与集群中的leader服务器交换信息的端口
#3888为选举时服务器相互通信的端口

#机器192.168.1.101的zoo.cfg最后加上
server.0=192.168.1.101:2888:3888

#机器192.168.1.102的zoo.cfg最后加上
server.1=192.168.1.102:2888:3888

启动:cmd下 bin\zkServer.cmd。

建议先启动zookeeper。

3.安装配置kafka

ip为192.168.1.101下的配置:

broker.id=0

listeners=PLAINTEXT://192.168.1.101:9092

log.dirs=/work/project/gj/logs/kafka

zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181

ip为192.168.1.102下的server.properties配置:

broker.id=1

listeners=PLAINTEXT://192.168.1.102:9092

log.dirs=/work/project/gj/logs/kafka

zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181

4.执行命令结果

192.168.1.101机器的文件夹内容:

kafka Windows集群配置实践

192.168.1.102机器的文件夹内容:

kafka Windows集群配置实践

5.执行命令过程及分析

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 127.0.0.1:9092  --create  --topic topic-sjy  --partitions 4  --replication-factor  1
[2020-06-17 10:36:58,551] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:00,656] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:02,761] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:05,068] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:07,573] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:10,480] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:13,690] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:16,597] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:19,808] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
终止批处理操作吗(Y/N)? y

#分析:执行失败,broker连接不上。
#下面改为固定ip连接

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy  --partitions 4  --replication-factor  1

#创建主题-topic-sjy ,分区数为4,副本为1。结果4个分区分布在了2台机器上,各2个。结果如上图。

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --describe  --topic topic-sjy
Topic:topic-sjy PartitionCount:4        ReplicationFactor:1     Configs:segment.bytes=1073741824
        Topic: topic-sjy        Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: topic-sjy        Partition: 1    Leader: 1       Replicas: 1     Isr: 1
        Topic: topic-sjy        Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: topic-sjy        Partition: 3    Leader: 1       Replicas: 1     Isr: 1

#查看分区描述
#分区0对应的文件夹为topic-sjy-0,leader在broker=0的机器上
#分区1对应的文件夹为topic-sjy-1,leader在broker=1的机器上
#分区2对应的文件夹为topic-sjy-2,leader在broker=0的机器上
#分区3对应的文件夹为topic-sjy-3,leader在broker=1的机器上
#心得:分区可以跨机器,每个分区都有一个对应的leader

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy-a  --partitions 2  --replication-factor  2

#创建主题-topic-sjy-a ,分区数为2,副本为2。结果2台机器上各有2个相同的文件。结果如上图。

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --describe  --topic topic-sjy-a
Topic:topic-sjy-a       PartitionCount:2        ReplicationFactor:2     Configs:segment.bytes=1073741824
        Topic: topic-sjy-a      Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0,1
        Topic: topic-sjy-a      Partition: 1    Leader: 1       Replicas: 1,0   Isr: 1,0

#查看分区描述
#分区0对应的文件夹为topic-sjy-a-0,leader在broker=0的机器上,broker=1上的同名文件夹为follower
#分区0对应的文件夹为topic-sjy-a-1,leader在broker=1的机器上,broker=0上的同名文件夹为follower
#心得:分区可以跨机器,每个分区都有一个对应的leader,副本没有跨机器(可以跨).两台机器压力一样大

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy-b  --partitions 2  --replication-factor  3
Error while executing topic command : org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.
[2020-06-17 10:43:54,373] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
        at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:190)
        at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:149)
        at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:144)
        at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:172)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:60)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.
 (kafka.admin.TopicCommand$)

#执行失败,副本数不能超过合法的brokers数。

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy-b  --partitions 3  --replication-factor  2

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --describe  --topic topic-sjy-b
Topic:topic-sjy-b       PartitionCount:3        ReplicationFactor:2     Configs:segment.bytes=1073741824
        Topic: topic-sjy-b      Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0,1
        Topic: topic-sjy-b      Partition: 1    Leader: 1       Replicas: 1,0   Isr: 1,0
        Topic: topic-sjy-b      Partition: 2    Leader: 0       Replicas: 0,1   Isr: 0,1

#查看分区描述
#分区0对应的文件夹为topic-sjy-b-0,leader在broker=0的机器上,broker=1上的同名文件夹为follower
#分区1对应的文件夹为topic-sjy-b-1,leader在broker=1的机器上,broker=0上的同名文件夹为follower
#分区2对应的文件夹为topic-sjy-b-2,leader在broker=0的机器上,broker=1上的同名文件夹为follower
#broker=0的机器通信压力较大

#Replicas: 0,1 表示该分区保存在 broker 0 和 broker 1下。

#Isr: 0,1 表示当前我们可以在 broker 0 和 broker 1 下访问该分区,如果 broker 0挂了,那就像这样子了 Isr: 1

最后:Windows下建议使用的版本:kafka_2.12-2.3.0-SNAPSHOT【解决不能在window下使用的BUG】  密码:mfhu

并把kafka配置参数log.retention.hours设置小一点。

具体查看https://blog.csdn.net/chengtanyong4777/article/details/102542326

 

 

相关标签: kafka zookeeper