Zookeeper ensemble won't start up - docker

Question

I've set up a Zookeeper-ensemble (version 3.4.9) with 3 instances. This works like a charm on the test-system, but doesn't come up on the live-system at all. The error message is the following:
2020-08-28 06:26:24,643 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager#400] - Cannot open channel to 2 at election address /10.3.1.173:3888
java.net.NoRouteToHostException: Host is unreachable (Host unreachable)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:745)
I've searched on here and in other places, but the only accepted solution to the problem is to set each node's server address to 0.0.0.0, which doesn't work here. My setup is fully dockerized and applied with ansible, so it might look a bit different from what people normally seem to do. But the connection string e.g. for server.1 is this:
"server.1=0.0.0.0:2888:3888 server.2=10.3.1.173:2888:3888 server.3=10.3.1.175:2888:3888"
which is also applied to the zookeepers internal configuration, as the logs show (again for server.1):
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2020-08-28 06:26:23,549 [myid:] - INFO [main:QuorumPeerConfig#124] - Reading configuration from: /conf/zoo.cfg
2020-08-28 06:26:23,559 [myid:] - INFO [main:QuorumPeer$QuorumServer#149] - Resolved hostname: 10.3.1.175 to address: /10.3.1.175
2020-08-28 06:26:23,559 [myid:] - INFO [main:QuorumPeer$QuorumServer#149] - Resolved hostname: 10.3.1.173 to address: /10.3.1.173
2020-08-28 06:26:23,560 [myid:] - INFO [main:QuorumPeer$QuorumServer#149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2020-08-28 06:26:23,560 [myid:] - INFO [main:QuorumPeerConfig#352] - Defaulting to majority quorums
(...)
2020-08-28 06:26:23,570 [myid:1] - INFO [main:QuorumPeerMain#127] - Starting quorum peer
2020-08-28 06:26:23,577 [myid:1] - INFO [main:Login#294] - successfully logged in.
2020-08-28 06:26:23,579 [myid:1] - INFO [main:NIOServerCnxnFactory#89] - binding to port 0.0.0.0/0.0.0.0:2181
This is applied to all 3 instance of zookeeper, but none of them can talk to another.
Additional information:
Apart from IP-addresses for the servers, the configuration is identical to the test-system. The Ansible Docker module is configured the same, the JAAS-Config (with DigestLoginModule) is the same, and the environment variables inside of all docker containers are the same, too.
Each server inside the live system can ping the other servers. I can also ping these servers from inside each Zookeeper container. In addition, I can curl each Zookeeper container on the JMX-port from inside any other container of the live-system. So they definitely can connect over the network.
Please help, thanks :D
Edit: #Stefano was asking how I start the docker containers, so I'll try to provide some insight. As mentioned, it's an Ansible setup in a task using the "docker_container" plugin which is used in a playbook to install the 3 instances across machines:
---
- name: Install Zookeeper
docker_container:
name: zookeeper
image: zookeeper:3.4.9
state: started
ports:
- "2181:2181" # Zookeeper Port
- "2888:2888"
- "3888:3888" # Election ports
- "9998:8080" # JMX metrics
env:
ZOO_MY_ID: "{{ ID }}" #this is 1 for server.1, etc.
ZOO_PORT: "2181"
ZOO_SERVERS: "{{ ZOO_SERVERS }}" #provided in host-vars
SERVER_JVMFLAGS: "-Djava.security.auth.login.config=/etc/kafka/zookeeper_jaas.conf -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.12.0.jar=8080:/opt/jmx-exporter/zookeeper.yml"
volumes:
- /home/ansible/volumes/zoo1/data:/data
- /home/ansible/volumes/zoo1/datalog:/datalog
- /home/ansible/jmx-exporter:/opt/jmx-exporter
- /home/ansible/zookeeper_jaas.conf:/etc/kafka/zookeeper_jaas.conf
The ZOO_SERVERS are taken from the hosts file:
all:
(...)
children:
zookeeper:
hosts:
zoo1:
ID: "1"
ZOO_SERVERS: "server.1=0.0.0.0:2888:3888 server.2=10.3.1.173:2888:3888 server.3=10.3.1.175:2888:3888"
ansible_host: 10.3.1.171
zoo2:
ID: "2"
ZOO_SERVERS: "server.1=10.3.1.171:2888:3888 server.2=0.0.0.0:2888:3888 server.3=10.3.1.175:2888:3888"
ansible_host: 10.3.1.173
zoo3:
ID: "3"
ZOO_SERVERS: "server.1=10.3.1.171:2888:3888 server.2=10.3.1.173:2888:3888 server.3=0.0.0.0:2888:3888"
ansible_host: 10.3.1.175

Answer 1

So when I read back what I commented above, I noticed that I am not actually using the "confluentinc/cp-zookeeper" docker image, but the "zookeeper" docker image.
Once I changed from "zookeeper:3.4.9" to "confluentinc/cp-zookeeper:5.4.0" and adjusted the ZOO_PORT env-var's name to ZOOKEEPER_CLIENT_PORT, it somehow worked.
This doesn't answer the "why" but maybe this workaround helps someone else. I'll mark this as the accepted answer for now, but please feel free to provide additional insight.