Kafka broker migration¶
In Strimzi, the broker storage is configured in the kafkaNodePool
resource.
From time to time, you may need to expand broker storage capacity, migrate the brokers to a new storage system or even migrate the brokers to different physical nodes, if they use local storage.
These operations involve creating a new KafkaNodePool
with the updated storage configuration and then transferring data from the old brokers to the new ones using Cruise Control.
With Cruise Control, you can rebalance the data across the brokers without downtime, ensuring a smooth migration.
Note
This broker migration procedure is based on the Kafka Node Pools Storage & Scheduling Strimzi blog post.
Before you begin, ensure that Cruise Control is enabled in your Sasquatch Phalanx environment.
Check your sasquatch/values-<environment>.yaml
file for the following:
strimzi-kafka:
cruiseControl:
enabled: true
Here we illustrate the procedure to migrate the Kafka brokers to a different storage system,
assuming the new storage class is called localdrive
and is available in your Kubernetes cluster.
To create a new KafkaNodePool
with this storage class, enable the brokerMigration
section in your sasquatch/values-<environment>.yaml
file and specify the new pool name, the node IDs of the new brokers (usually use sequential numbers from the existing brokers), the size of the new storage, and the new storage class name, localdrive
.
For now, set rebalance.enabled: false
to avoid rebalancing the data immediately after creating the new KafkaNodePool
resource.
brokerMigration:
enabled: true
name: kafka-local-storage
nodeIDs: "[6,7,8]"
size: 1.5Ti
storageClassName: "localdrive"
rebalance:
enabled: false
Note
You can also specify other configuration like affinity, tolerations and Kubernetes resources for the brokers, omitted here for simplicity.
To apply this configuration sync the new KafkaNodePool
resource in Argo CD.
At this point, your data still resides on the old brokers, and the new ones will be empty.
To move the data, use Cruise Control by setting rebalance.enabled: true
and the IDs of the brokers to avoid — the ones you plan to remove after the migration.
brokerMigration:
enabled: true
name: kafka-local-storage
nodeIDs: "[6,7,8]"
size: 1.5Ti
storageClassName: "localdrive"
rebalance:
enabled: true
avoidBrokers:
- 3
- 4
- 5
This configuration will create a new KafkaRebalance
resource that will instruct Cruise Control to migrate the data from the old brokers (3, 4, 5) to the new ones (6, 7, 8).
Next, wait for Cruise Control to execute the cluster rebalance.
You can check the state of the rebalance by inspecting the KafkaRebalance
resource in the sasquatch
namespace and monitoring the status of the rebalance.
$ kubectl get kafkarebalances.kafka.strimzi.io -n sasquatch
NAME CLUSTER PENDINGPROPOSAL PROPOSALREADY REBALANCING READY NOTREADY STOPPED
broker-migration sasquatch True
During rebalance you can also verify the data being moved to the new brokers by checking the topic partitions in Kafdrop.
Once the rebalance is complete, you can swap the new KafkaNodePool
configuration with the old KafkaNodePool
configuration, by updating the broker:
section and disabling (or just removing) the brokerMigration:
section.
broker:
enabled: true
name: kafka-local-storage
nodeIDs: "[6,7,8]"
size: 1.5Ti
storageClassName: "localdrive"
brokerMigration:
enabled: false
Also, ensure that you update the Kafka external listener configuration to match the new broker IDs.
externalListener:
brokers:
- broker: 6
loadBalancerIP: "139.229.180.92"
host: sasquatch-kafka-6.lsst.codes
- broker: 7
loadBalancerIP: "139.229.180.93"
host: sasquatch-kafka-7.lsst.codes
- broker: 8
loadBalancerIP: "139.229.180.94"
host: sasquatch-kafka-8.lsst.codes
After applying this configuration, you can delete the old brokers by removing the old KafkaNodePool
resource from Argo CD.
Note that the PVCs of the old brokers need to be deleted manually, as they are orphan resources in Sasquatch.