Service verticles failing to re-register to zk on zk restart

[AnveshMora]

Questions

We have infra running on AWS ECS, zookeper is running on Docker, same as other servies.

During regular service usage in production, zookeeper intermittently restarts. During that time we are seeing vert.x service verticles are failing to re-register with new zookeeper instance

Version

Which version(s) did you encounter this bug ?
Java: 17
vertex-core: 4.4.6
vertex-zookeeper: 4.4.2

Context

I encountered an exception which looks suspicious while zookeper restarts:
[vert.x-eventloop-thread-2] WARN i.v.core.eventbus.impl.clustered.ConnectionHolder - Connecting to server a3369c02-695f-4f23-ba11-711027489f8d failed (?:?) io.vertx.core.VertxException: Not a member of the cluster at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.lambda$null$9(ZookeeperClusterManager.java:250) at java.base/java.util.Optional.orElseThrow(Unknown Source) at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.lambda$getNodeInfo$10(ZookeeperClusterManager.java:250) at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) at datadog.trace.bootstrap.instrumentation.java.concurrent.Wrapper.run(Wrapper.java:46) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source)

Do you have a reproducer?

Nope

Reproducers are very helpful for contributors and will likely help them fixing your bug faster.

Link to github project/gist

A reproducer is a simple project hosted on GitHub (or another forge supporting git clone operation) that has a build file that can be executed to reproduce the issue.

Reproducers are very helpful for contributors and will likely help them fixing your bug faster.

• Link to github project/gist

Steps to reproduce

1. Having zookeeper setup on docker
2. Having one or services registering to event-bus
3. restart the zookeeper, than we see above error message.

Extra

• Anything that can be relevant such as OS version, JVM version

[AnveshMora]

Similar issue raised by me on vert-x3 issues repo: vert-x3/issues#651

[AnveshMora]

Hi everyone, just wanted to check if any of the community had similar issue has above or any suggestions how to tackle this issue?

[AnveshMora]

Any insights/ guidance to resolve this issue?