kafka修改topic副本数

 工作案例:

    大数据开发用系统脚本自动在kafka建topic,检查后才发现副本数只有1个,存在数据丢失的风险。需要立刻把副本数改为3个。

 

  开始干活，首先想到的是下面的命令：    

${bin_path}/kafka-topics.sh --zookeeper ${zookeeper_addr} --topic $1 --partitions $2 --alter
${bin_path}/kafka-topics.sh --zookeeper ${zookeeper_addr} --topic $1 --replication-factor $2 --alter

   运行时修改partition成功，修改副本数失败，错误如下:

:2181 --topic test --replication-factor 2 --alter
option "[replication-factor]" can't be used with option "[alter]"

  马上度娘，有很多方法都是用json文件的方式修改，试验了下都不成功，直接去官网查权威

https://kafka.apache.org/21/documentation.html#basic_ops_increase_replication_factor

  思路没错，原来是网上的json文件写不对。解决方法：

1/ 建一个测试topic

data1/kafka/bin/kafka-topics.sh --zookeeper ${zookeeper_addr} --create --partitions 5 --replication-factor 3 --topic test

2/ 察看test的详细信息

topic:test partitioncount:5 replicationfactor:3 configs:
topic: test partition: 0 leader: 21 replicas: 21,22,31 isr: 21,22,31
topic: test partition: 1 leader: 22 replicas: 21,22,31 isr: 22,31,21
topic: test partition: 2 leader: 31 replicas: 21,22,31 isr: 31,21,22
topic: test partition: 3 leader: 21 replicas: 21,22,31 isr: 21,31,22
topic: test partition: 4 leader: 21 replicas: 21,22,31 isr: 31,22,21

3/ 准备修改副本的json文件，格式可以找个json生成器查看。[]中是集群的broker-list

[root@zk-kafka-1 ~]# cat increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"test","partition":0,"replicas":[21,22,31,11]},
{"topic":"test","partition":1,"replicas":[21,22,31,11]},
{"topic":"test","partition":2,"replicas":[21,22,31,11]},
{"topic":"test","partition":3,"replicas":[21,22,31,11]},
{"topic":"test","partition":4,"replicas":[21,22,31,11]}
]

}

4/ 执行修改命令，把副本数修改为4

[root@zk-kafka-1 ~]# /data1/kafka/bin/kafka-reassign-partitions.sh --zookeeper ${zookeeper_addr}  --reassignment-json-file increase-replication-factor.json --execute
current partition replica assignment

{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":1,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":4,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":3,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":2,"replicas":[21,22,31],"log_dirs":["any","any","any"]}]}

save this to use as the --reassignment-json-file option during rollback
successfully started reassignment of partitions.

5/ 看到successfully,查看现在test的副本数已经修改为4

topic:test partitioncount:5 replicationfactor:4 configs:
topic: test partition: 0 leader: 21 replicas: 21,22,31,11 isr: 21,22,31,11
topic: test partition: 1 leader: 22 replicas: 21,22,31,11 isr: 22,31,21,11
topic: test partition: 2 leader: 31 replicas: 21,22,31,11 isr: 31,21,22,11
topic: test partition: 3 leader: 21 replicas: 21,22,31,11 isr: 21,31,22,11
topic: test partition: 4 leader: 21 replicas: 21,22,31,11 isr: 31,22,21,11