Error in running "docker-compose up" : Invalid config, exiting abnormally - docker

Question

I have three different nodes that every one has docker with Ubuntu on it. I want to make Kafka cluster with these three nodes; In fact, I installed docker on each node with loading Ubuntu with on them. I configure "zookeeper.properties" in docker environment for "150.20.11.157" like this:
dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=0.0.0.0:2888:3888
server.2=150.20.11.134:2888:3888
server.3=150.20.11.137:2888:3888
clientPort=2186
For node 150.20.11.134, "zookeeper.properties" file in docker environment is like this:
dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=150.20.11.157:2888:3888
server.2=0.0.0.0:2888:3888
server.3=150.20.11.137:2888:3888
clientPort=2186
For node 150.20.11.137, "zookeeper.properties" file in docker environment is like this:
dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=150.20.11.157:2888:3888
server.2=150.20.11.134:2888:3888
server.3=0.0.0.0:2888:3888
clientPort=2186
Also, I setup "server.properties" like this, for node 150.20.11.157:
broker.id=0
port=9092
listeners = PLAINTEXT://150.20.11.157:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186
"server.properties" for node 150.20.11.134 is:
broker.id=1
port=9092
listeners = PLAINTEXT://150.20.11.134:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186
"server.properties" for node 150.20.11.137 is:
broker.id=2
port=9092
listeners = PLAINTEXT://150.20.11.137:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186
More over, every node has a "myid" file in "/tmp/zookeeper/data" of docker environment with its server id inside it.
To make a Kafka cluster of three node like this picture, I make a "docker-compose.yaml" file and a dockerfile for it.
This is my docker-compose file:
version: '3.7'
services:
zookeeper:
build: .
command: /root/kafka_2.11-2.0.1/bin/zookeeper-server-start.sh
/root/kafka_2.11-2.0.1/config/zookeeper.properties
ports:
- 2186:2186
kafka1:
build:
context: .
args:
brokerId: 0
command: /root/kafka_2.11-2.0.1/bin/kafka-server-start.sh
/root/kafka_2.11-2.0.1/config/server.properties
depends_on:
- zookeeper
kafka2:
build:
context: .
args:
brokerId: 1
command: /root/kafka_2.11-2.0.1/bin/kafka-server-start.sh
/root/kafka_2.11-2.0.1/config/server.properties
depends_on:
- zookeeper
kafka3:
build:
context: .
args:
brokerId: 2
command: /root/kafka_2.11-2.0.1/bin/kafka-server-start.sh
/root/kafka_2.11-2.0.1/config/server.properties
depends_on:
- zookeeper
producer:
build: .
command: bash -c "sleep 4 && /root/kafka_2.11-2.0.1/bin/kafka-
topics.sh --create --zookeeper zookeeper:2186 --replication-
factor 2 --partitions 3 --topic dates && while true; do date |
/kafka_2.11-2.0.1/bin/kafka-console-producer.sh --broker-list
kafka1:9092,kafka2:9092,kafka3:9092 --topic dates; sleep 1;
done "
depends_on:
- zookeeper
- kafka1
- kafka2
- kafka3
consumer:
build: .
command: bash -c "sleep 6 && /root/kafka_2.11-2.0.1/bin/kafka-
console-consumer.sh localhost:9092 --topic dates --bootstrap-
server kafka1:9092,kafka2:9092,kafka3:9092"
depends_on:
- zookeeper
- kafka1
- kafka2
- kafka3
The problem is after "dockerfile build ." when I do "sudo docker-compose up" on each node. It does not run completely. Some of my log is in following:
zookeeper_1 | [2019-01-17 16:09:27,197] INFO Reading configuration from: /root/kafka_2.11-2.0.1/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
kafka3_1 | [2019-01-17 16:09:29,426] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka3_1 | [2019-01-17 16:09:29,702] INFO starting (kafka.server.KafkaServer)
kafka3_1 | [2019-01-17 16:09:29,702] INFO Connecting to zookeeper on 150.20.11.157:2186,150.20.11.134:2186,150.20.11.137:2186 (kafka.server.KafkaServer)
kafka1_1 | [2019-01-17 16:09:30,012] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
zookeeper_1 | [2019-01-17 16:09:27,240] INFO Resolved hostname: 150.20.11.137 to address: /150.20.11.137 (org.apache.zookeeper.server.quorum.QuorumPeer)
kafka1_1 | [2019-01-17 16:09:30,486] INFO starting (kafka.server.KafkaServer)
kafka3_1 | [2019-01-17 16:09:29,715] INFO [ZooKeeperClient] Initializing a new session to 150.20.11.157:2186,150.20.11.134:2186,150.20.11.137:2186. (kafka.zookeeper.ZooKeeperClient)
zookeeper_1 | [2019-01-17 16:09:27,241] INFO Resolved hostname: 150.20.11.134 to address: /150.20.11.134 (org.apache.zookeeper.server.quorum.QuorumPeer)
zookeeper_1 | [2019-01-17 16:09:27,241] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)
kafka3_1 | [2019-01-17 16:09:29,720] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
zookeeper_1 | [2019-01-17 16:09:27,241] INFO Defaulting to majority quorums (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
kafka3_1 | [2019-01-17 16:09:29,721] INFO Client environment:host.name=be08b050be4c (org.apache.zookeeper.ZooKeeper)
zookeeper_1 | [2019-01-17 16:09:27,242] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1 | org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /root/kafka_2.11-2.0.1/config/zookeeper.properties
zookeeper_1 | at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
zookeeper_1 | at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
zookeeper_1 | at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
zookeeper_1 | Caused by: java.lang.IllegalArgumentException: /tmp/zookeeper/data/myid file is missing
zookeeper_1 | at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:408)
zookeeper_1 | at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:152)
zookeeper_1 | ... 2 more
kafka1_1 | [2019-01-17 16:09:30,487] INFO Connecting to zookeeper on 150.20.11.157:2186,150.20.11.134:2186,150.20.11.137:2186 (kafka.server.KafkaServer)
zookeeper_1 | Invalid config, exiting abnormall
In fact, I configured Kafka cluster without using docker on every node and I could run Zookeeper and Kafka server without any problem. Kafka cluster was like this picture:
Would you please tell me what I am doing wrong to config this cluser?
Any help would be appreciated.

Answer 1

I change the docker-compose file and problem solved. Zookeeper and Kafka server run without any problem. Topic was created. Also,Consumer and Producer worked with topic in three nodes. My docker-compose for one node is like this:
version: '3.7'
services:
zookeeper:
image: ubuntu_mesos
command: /root/kafka_2.11-2.0.1/bin/zookeeper-server-start.sh
/root/kafka_2.11-2.0.1/config/zookeeper.properties
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 2186
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 10
ZOOKEEPER_SYNC_LIMIT: 5
ZOOKEEPER_SERVERS:
0.0.0.0:2888:3888;150.20.11.134:2888:3888;150.20.11.137:2888:3888
network_mode: host
expose:
- 2186
- 2888
- 3888
ports:
- 2186:2186
- 2888:2888
- 3888:3888
kafka:
image: ubuntu_mesos
command: bash -c "sleep 20; /root/kafka_2.11-2.0.1/bin/kafka-server-
start.sh /root/kafka_2.11-2.0.1/config/server.properties"
network_mode: host
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 0
KAFKA_ZOOKEEPER_CONNECT:
150.20.11.157:2186,150.20.11.134:2186,150.20.11.137:2186
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://150.20.11.157:9092
expose:
- 9092
ports:
- 9092:9092
producer:
image: ubuntu_mesos
command: bash -c "sleep 40; /root/kafka_2.11-2.0.1/bin/kafka-topics.sh --
create --zookeeper 150.20.11.157:2186 --replication-factor 2 --partitions
3 --topic testFlink -- /root/kafka_2.11-2.0.1/bin/kafka-console-
producer.sh --broker-list 150.20.11.157:9092 --topic testFlink"
depends_on:
- zookeeper
- kafka
consumer:
image: ubuntu_mesos
command: bash -c "sleep 44; /root/kafka_2.11-2.0.1/bin/kafka-console-
consumer.sh --bootstrap-server 150.20.11.157:9092 --topic testFlink --
from-beginning"
depends_on:
- zookeeper
- kafka
Two other nodes have the docker-compose like above too.
Hope it was helpful for others.