kafka Windows集群配置实践

Windows下集群安装

1.安装jdk。设置java_home，然后把%java_home%\bin放到path环境变量里。

2.安装配置zookeeper

#缺少这个有时启动zookeeper会报错，建议加上
admin.serverPort=7070
#两台windows机器的ip和端口
#2888为服务器与集群中的leader服务器交换信息的端口
#3888为选举时服务器相互通信的端口

#机器192.168.1.101的zoo.cfg最后加上
server.0=192.168.1.101:2888:3888

#机器192.168.1.102的zoo.cfg最后加上
server.1=192.168.1.102:2888:3888

启动：cmd下 bin\zkServer.cmd。

建议先启动zookeeper。

3.安装配置kafka

ip为192.168.1.101下的配置：

broker.id=0

listeners=PLAINTEXT://192.168.1.101:9092

log.dirs=/work/project/gj/logs/kafka

zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181

ip为192.168.1.102下的server.properties配置：

broker.id=1

listeners=PLAINTEXT://192.168.1.102:9092

log.dirs=/work/project/gj/logs/kafka

zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181

4.执行命令结果

192.168.1.101机器的文件夹内容：

192.168.1.102机器的文件夹内容：

5.执行命令过程及分析

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 127.0.0.1:9092  --create  --topic topic-sjy  --partitions 4  --replication-factor  1
[2020-06-17 10:36:58,551] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:00,656] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:02,761] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:05,068] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:07,573] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:10,480] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:13,690] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:16,597] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-17 10:37:19,808] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
终止批处理操作吗(Y/N)? y

#分析：执行失败，broker连接不上。
#下面改为固定ip连接

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy  --partitions 4  --replication-factor  1

#创建主题-topic-sjy ，分区数为4，副本为1。结果4个分区分布在了2台机器上，各2个。结果如上图。

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --describe  --topic topic-sjy
Topic:topic-sjy PartitionCount:4        ReplicationFactor:1     Configs:segment.bytes=1073741824
        Topic: topic-sjy        Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: topic-sjy        Partition: 1    Leader: 1       Replicas: 1     Isr: 1
        Topic: topic-sjy        Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: topic-sjy        Partition: 3    Leader: 1       Replicas: 1     Isr: 1

#查看分区描述
#分区0对应的文件夹为topic-sjy-0,leader在broker=0的机器上
#分区1对应的文件夹为topic-sjy-1,leader在broker=1的机器上
#分区2对应的文件夹为topic-sjy-2,leader在broker=0的机器上
#分区3对应的文件夹为topic-sjy-3,leader在broker=1的机器上
#心得：分区可以跨机器，每个分区都有一个对应的leader

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy-a  --partitions 2  --replication-factor  2

#创建主题-topic-sjy-a ，分区数为2，副本为2。结果2台机器上各有2个相同的文件。结果如上图。

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --describe  --topic topic-sjy-a
Topic:topic-sjy-a       PartitionCount:2        ReplicationFactor:2     Configs:segment.bytes=1073741824
        Topic: topic-sjy-a      Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0,1
        Topic: topic-sjy-a      Partition: 1    Leader: 1       Replicas: 1,0   Isr: 1,0

#查看分区描述
#分区0对应的文件夹为topic-sjy-a-0,leader在broker=0的机器上,broker=1上的同名文件夹为follower
#分区0对应的文件夹为topic-sjy-a-1,leader在broker=1的机器上,broker=0上的同名文件夹为follower
#心得：分区可以跨机器，每个分区都有一个对应的leader，副本没有跨机器（可以跨）.两台机器压力一样大

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy-b  --partitions 2  --replication-factor  3
Error while executing topic command : org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.
[2020-06-17 10:43:54,373] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
        at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:190)
        at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:149)
        at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:144)
        at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:172)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:60)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.
 (kafka.admin.TopicCommand$)

#执行失败，副本数不能超过合法的brokers数。

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --create  --topic topic-sjy-b  --partitions 3  --replication-factor  2

E:\work\project\gj\kafka_2.12-2.3.0>bin\windows\kafka-topics.bat --bootstrap-server 192.168.1.102:9092  --describe  --topic topic-sjy-b
Topic:topic-sjy-b       PartitionCount:3        ReplicationFactor:2     Configs:segment.bytes=1073741824
        Topic: topic-sjy-b      Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0,1
        Topic: topic-sjy-b      Partition: 1    Leader: 1       Replicas: 1,0   Isr: 1,0
        Topic: topic-sjy-b      Partition: 2    Leader: 0       Replicas: 0,1   Isr: 0,1

#查看分区描述
#分区0对应的文件夹为topic-sjy-b-0,leader在broker=0的机器上,broker=1上的同名文件夹为follower
#分区1对应的文件夹为topic-sjy-b-1,leader在broker=1的机器上,broker=0上的同名文件夹为follower
#分区2对应的文件夹为topic-sjy-b-2,leader在broker=0的机器上,broker=1上的同名文件夹为follower
#broker=0的机器通信压力较大

#Replicas: 0,1 表示该分区保存在 broker 0 和 broker 1下。

#Isr: 0,1 表示当前我们可以在 broker 0 和 broker 1 下访问该分区，如果 broker 0挂了，那就像这样子了 Isr: 1

最后：Windows下建议使用的版本：kafka_2.12-2.3.0-SNAPSHOT【解决不能在window下使用的BUG】  密码：mfhu

并把kafka配置参数log.retention.hours设置小一点。

具体查看https://blog.csdn.net/chengtanyong4777/article/details/102542326。