ERROR Invalid config, exiting abnormally in Kafka cluster - docker

Question

I have three different nodes that every one has docker with Ubuntu on it. I can run zookeeper server on docker locally (I mean without any configuration for cluster), but I want to make kafka cluster with these three nodes; In fact, I installed docker on each node with loading Ubuntu with on them. I configure "zookeeper.properties" in docker environment for "150.20.11.157" like this:
dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=0.0.0.0:2888:3888
server.2=150.20.11.134:2888:3888
server.3=150.20.11.137:2888:3888
clientPort=2186
For node 150.20.11.134, "zookeeper.properties" file in docker environment is like this:
dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=150.20.11.157:2888:3888
server.2=0.0.0.0:2888:3888
server.3=150.20.11.137:2888:3888
clientPort=2186
For node 150.20.11.137, "zookeeper.properties" file in docker environment is like this:
dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=150.20.11.157:2888:3888
server.2=150.20.11.134:2888:3888
server.3=0.0.0.0:2888:3888
clientPort=2186
Also, I setup "server.properties" like this, for node 150.20.11.157:
broker.id=0
port=9092
listeners = PLAINTEXT://150.20.11.157:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186
"server.properties" for node 150.20.11.134 is:
broker.id=1
port=9092
listeners = PLAINTEXT://150.20.11.134:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186
"server.properties" for node 150.20.11.137 is:
broker.id=2
port=9092
listeners = PLAINTEXT://150.20.11.137:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186
The problem is when I run zookeeper server on docker of each node. I get this error:
[2019-01-16 12:45:54,588] INFO Reading configuration from: ./config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-01-16 12:45:54,601] INFO Resolved hostname: 172.28.10.137 to address: /172.28.10.137 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2019-01-16 12:45:54,603] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2019-01-16 12:45:54,603] INFO Resolved hostname: 172.28.10.157 to address: /172.28.10.157 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2019-01-16 12:45:54,603] INFO Defaulting to majority quorums (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-01-16 12:45:54,604] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing ./config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Caused by: java.lang.IllegalArgumentException: /tmp/zookeeper/data/myid file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:408)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:152)
... 2 more
Invalid config, exiting abnormally
Would you please tell me how to have a Kafka cluster with three docker that each docker is on one physical node?
Thank you in advance.

Answer 1

I've run into this too.
I think the clue for this one is in the logs i.e.
Caused by: java.lang.IllegalArgumentException:
/tmp/zookeeper/data/myid file is missing at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:408)
The following lines in your zookeeper.properties file provide the details of the zookeeper ensemble:
server.1=0.0.0.0:2888:3888
server.2=150.20.11.134:2888:3888
server.3=150.20.11.137:2888:3888
When one of the zookeeper server starts, it knows which server it is by looking at the myid file in it's own data directory, which in this case would be /tmp/zookeeper/data.
So, all you need to do is just create a file with the name myid in the above mentioned directory of each server and write just x = 1, 2 or 3 (these correspond to server.x in the zookeeper.properties file).
Reference link - Apache Zookeeper.
Hope this helps!