kafka-connect-1 is started

add connector with 4 tasks via POST /connectors

checking status via GET /connectors/myConnector/status
i can see my connector and 4 tasks for it

i am starting kafka-connect-2

checking status again and can see that all tasks still on kafka-connect-1

now if i do DELETE /connectors/myConnector/ then POST /connectors
and check status again then i can see that now 4 tasks are spread between 2 connects

My questions are

is there any way to trigger rebalance automatically after new kafka-connect joining cluster?

is this del+add connector is correct way to trigger rebalance manually?

kafka-connect-1 and kafka-connect-2 in the same cluster

restapi

Hi @Sirius_Dour , you can find some details in this blog post: Incremental Cooperative Rebalancing in Apache Kafka

Can you try adding the configuration property
connect.protocol=eager

and see if that makes it work as you expect?

Keep us posted!

Hi @gianlucanatali

Looks like it is working as i wanted:
i added connect.protocol=eager in my .properties file for kafka-connect-1 and kafka-connect-2
i started kafka-connect-1 (cube) and added connector with 4 tasks on it
then i just started kafka-connect-2 (smart) and checked status
i can see that all tasks are rebalanced between these 2 nodes

Thank you!

I have read Incremental Cooperative Rebalancing in Apache Kafka
And i have additional questions: so my kafka-connect-1 and kafka-connect-2 are just “workers”. So it mean that case that i described is exact representation of 1. A new worker joins (Figure 1). and it says that during 2-d rebalance some tasks should be moved to this new worker (kafka-connect-2/smart in my case) and other tasks should not be interrupted. But it isn’t happen: without connect.protocol=eager property all tasks remains assigned to 1-st worker (cube).
(on the left status from worker-1 on the right status from worker-2)

And this property forces my worker to use Eager Rebalancing . Am i correct?
Why Incremental Cooperative Rebalancing from this Figure 1 is not working?
Will this rebalance trigger tasks revocations and interruptions as result?

I tried to find more details about this property but found only this
https://docs.confluent.io/platform/current/installation/configuration/connect/index.html#connect.protocol

Maybe my test wasn’t very accurate: i realized that there was 3 connectors but i checked status only for one of them.
Now i repeated this test with only 1 connector onboard and without this property
here is the result
look like 2 tasks moved to new worker

recheck with all 3 connectors (without property) also shows correct results if check status of each connector:
after worker-2 joined i got next picture:

worker-1:
4x connector-1 + 2x connector-3 = 6 tasks in total

worker-2:
4x connector-2 + 1x connector-3 = 5 tasks in total

it looks balanced (based on amount of on each worker)

recheck with all 3 connectors (with property) shows these results:

worker-1:
2x connector-1 + 2x connector-2 + 1x connector-3 = 5 tasks in total

worker-2:
2x connector-1 + 2x connector-2 + 2x connector-3 = 6 tasks in total

this is also balanced but in more strict way