kafka docker stuck on configuration and not working on mac - docker

Question

I am trying to use kafka on docker. But when running the docker compose, kafka just stuck on configuring mode and not indicating that already running. I do listing for kafka topics but it's not working or not giving the response. Here's my docker-compose.yml:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
hostname: zookeeper
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
networks:
- kafka_net
kafka:
image: wurstmeister/kafka
container_name: kafka
ports:
- 9092:9092
expose:
- 9092
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ADVERTISED_PORT: 9092
KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_BROKER_ID: 1
restart: always
networks:
- kafka_net
networks:
kafka_net:
driver: "bridge"
Here's the zookeeper response on docker-compose log:
2021-04-19 01:07:56,385 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=/opt/zookeeper-3.4.13
2021-04-19 01:07:56,393 [myid:] - INFO [main:ZooKeeperServer#836] - tickTime set to 2000
2021-04-19 01:07:56,394 [myid:] - INFO [main:ZooKeeperServer#845] - minSessionTimeout set to -1
2021-04-19 01:07:56,394 [myid:] - INFO [main:ZooKeeperServer#854] - maxSessionTimeout set to -1
2021-04-19 01:07:56,418 [myid:] - INFO [main:ServerCnxnFactory#117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2021-04-19 01:07:56,430 [myid:] - INFO [main:NIOServerCnxnFactory#89] - binding to port 0.0.0.0/0.0.0.0:2181
Here's the only kafka response on docker-compose log:
kafka | [Configuring] 'log.dirs' in '/opt/kafka/config/server.properties'
kafka | [Configuring] 'zookeeper.connect' in '/opt/kafka/config/server.properties'
kafka | [Configuring] 'listeners' in '/opt/kafka/config/server.properties'
kafka | Excluding KAFKA_VERSION from broker config
kafka | [Configuring] 'broker.id' in '/opt/kafka/config/server.properties'
kafka | [Configuring] 'listener.security.protocol.map' in '/opt/kafka/config/server.properties'
kafka | [Configuring] 'advertised.listeners' in '/opt/kafka/config/server.properties'
kafka | [Configuring] 'port' in '/opt/kafka/config/server.properties'
kafka | [Configuring] 'advertised.host.name' in '/opt/kafka/config/server.properties'
kafka | Excluding KAFKA_HOME from broker config
kafka | [Configuring] 'advertised.port' in '/opt/kafka/config/server.properties'
kafka | [Configuring] 'inter.broker.listener.name' in '/opt/kafka/config/server.properties'
When running this command, it's not giving a response:
docker exec -it kafka /bin/sh
/ # cd /opt/kafka_2.13-2.7.0
/opt/kafka_2.13-2.7.0 # bin/kafka-topics.sh --list --zookeeper zookeeper:2181
what should i do on this? i still trying to make it run. I already try with kafka native (without docker) and it run perfectly the i can create a topic

Answer 1

Try to use another image of kafka, to example - bitnami. I had the same problem as you on Apple M1 Big Shur, and i used bitnami and it's working!! this is my docker-compose.yml:
version: "2"
services:
zookeeper:
image: docker.io/bitnami/zookeeper:3
ports:
- "2181:2181"
volumes:
- "zookeeper_data:/bitnami"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: docker.io/bitnami/kafka:2
ports:
- "9092:9092"
volumes:
- "kafka_data:/bitnami"
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_LISTENERS=PLAINTEXT://:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
depends_on:
- zookeeper
volumes:
zookeeper_data:
driver: local
kafka_data:
driver: local

Answer 2

Seems like you missed an important part from your docker-compose log. I ran your docker-compose file on my local and would like to highlight this part from the logs.
kafka | [2021-04-19 09:20:19,209] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
kafka | [2021-04-19 09:20:19,253] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
kafka | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
kafka | at kafka.utils.CoreUtils$.validate$1(CoreUtils.scala:265)
kafka | at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:276)
kafka | at kafka.server.KafkaConfig.$anonfun$listeners$1(KafkaConfig.scala:1680)
kafka | at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:1679)
kafka | at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1779)
kafka | at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1756)
kafka | at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1312)
kafka | at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
kafka | at kafka.Kafka$.main(Kafka.scala:68)
kafka | at kafka.Kafka.main(Kafka.scala)
kafka exited with code 1
So you were using the same port for accessing Kafka from inside and outside the docker network. I changed the below setting (the INSIDE port) and it worked for me. I was able to create and list topics. I was also able to produce and consume simple messages from the kafka container.
KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092
This is some of the testing I did from the kafka container
docker exec -it kafka bash
bash-4.4# cd /opt/kafka_2.13-2.7.0/
bash-4.4# bin/kafka-topics.sh --create --topic test-topic --partitions 16 --replication-factor 1 --zookeeper zookeeper:2181
Created topic test-topic.
bash-4.4# bin/kafka-topics.sh --list --zookeeper zookeeper:2181
test-topic
This is an excellent post that I would like to refer you. Please let me know if this works for you or not.