0
点赞
收藏
分享

微信扫一扫

apache kafka系列之迁移与扩容工具用法


kafka迁移与扩容工具使用

参考官网site:https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool

说明:

当我们对kafka集群扩容时,需要满足2点要求:

 

  1. 将指定topic迁移到集群内新增的node上。
  2. 将topic的指定partition迁移到新增的node上。

1. 迁移topic到新增的node上



假如现在一个kafka集群运行三个broker,broker.id依次为101,102,103,后来由于业务数据突然暴增,需要新增三个broker,broker.id依次为104,105,106.目的是要把push-token-topic迁移到新增node上。脚本(json格式)如下所示:



lizhitao@localhost:$  ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka
--topics-to-move-json-file  migration-push-token-topic.json  --broker-list  "104,105,106"  --generate

 脚本migration-push-token-topic.json文件内容如下:

{
"topics":
[
{
"topic": "push-token-topic"
}
],
"version":1
}

 

生成分配partitions的json脚本:

Current partition replica assignment
{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[8]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":3,"replicas":[5]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":9,"replicas":[5]},{"topic":"cluster-switch-topic","partition":1,"replicas":[5]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[5]},{"topic":"cluster-switch-topic","partition":2,"replicas":[4]},{"topic":"cluster-switch-topic","partition":0,"replicas":[4]},{"topic":"cluster-switch-topic","partition":6,"replicas":[4]},{"topic":"cluster-switch-topic","partition":8,"replicas":[4]}]}




重新分配parttions的json脚本如下:

migration-topic-cluster-switch-topic.json 

{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[5]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":3,"replicas":[4]},{"topic":"cluster-switch-topic","partition":9,"replicas":[4]},{"topic":"cluster-switch-topic","partition":1,"replicas":[4]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[4]},{"topic":"cluster-switch-topic","partition":2,"replicas":[5]},{"topic":"cluster-switch-topic","partition":0,"replicas":[5]},{"topic":"cluster-switch-topic","partition":6,"replicas":[5]},{"topic":"cluster-switch-topic","partition":8,"replicas":[5]}]}

 

lizhitao@localhost:$   bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka01 --reassignment-json-file migration-topic-cluster-switch-topic.json --execute

2.topic修改(replicats-factor)副本个数

lizhitao@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper   192.168.2.225:2183/config/mobile/mq/mafka

--reassignment-json-file  replicas-update-push-token-topic.json  --execute

假如初始时push-token-topic为一个副本,为了提高可用性,需要改为2副本模式。

脚本replicas-push-token-topic.json文件内容如下:

 

{
"partitions":
[
{
"topic": "log.mobile_nginx",
"partition": 0,
"replicas": [101,102,104]
},
{
"topic": "log.mobile_nginx",
"partition": 1,
"replicas": [102,103,106]
},
{
"topic": "xxxx",
"partition": 数字,
"replicas": [数组]
}
],
"version":1
}

3.topic的分区扩容用法

 

a.先扩容分区数量,脚本如下:

例如:push-token-topic初始分区数量为12,目前到增加到15个

lizhitao@localhost:$ ./bin/kafka-topics.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka  --alter   --partitions 15   --topic   push-token-topic

b.设置topic分区副本

 

lizhitao@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper  192.168.2.225:2183/config/mobile/mq/mafka

--reassignment-json-file partitions-extension-push-token-topic.json  --execute



 

脚本partitions-extension-push-token-topic.json文件内容如下:

 



[plain]  ​​view plain​​  ​​copy​​

1. {  
2. "partitions":
3. [
4. {
5. "topic": "push-token-topic",
6. "partition": 12,
7. "replicas": [101,102]
8. },
9. {
10. "topic": "push-token-topic",
11. "partition": 13,
12. "replicas": [103,104]
13. },
14. {
15. "topic": "push-token-topic",
16. "partition": 14,
17. "replicas": [105,106]
18. }
19. ],
20. "version":1
21. }

举报

相关推荐

0 条评论