Kafka producer says "unknown_topic_or_partition" - docker

Question

I've been trying to make kafka-docker work for a few days now and I don't know what I'm doing wrong. Right now, I can't access any topic with my ruby-kafka client because the node "doesn't exist". This is my docker-compose.yml file:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9092:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka2:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9093:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 2
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka3:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9094:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 3
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9094
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
I specify "KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'" because I want to create topics by hand, so I entered into my first broker container and typed this:
./kafka-topics.sh --create --zookeeper 172.19.0.2:2181 --topic test1 --partitions 4 --replication-factor 3
And everything seems fine:
./kafka-topics.sh --list --zookeeper 172.19.0.2:2181 -> test1
But, when I try to do this:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test1
It says:
WARN Error while fetching metadata with correlation id 24 : {test1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
If I create again the topic, it says it already exists, so I don't know what is happening anymore.

Answer 1

You need to get your networking configuration right, as Kafka works across hosts and needs to be able to access them all.
This post explains it in detail.
You might also want to reference https://github.com/confluentinc/cp-docker-images/blob/5.0.0-post/examples/cp-all-in-one/docker-compose.yml for an example of a working Docker Compose.

Answer 2

so we got this issue when we were working with kafka connect. Thre are multiple solutions to this. Either prune all the docker images or change the group id in the configuration for connect in the connect image as below:-
image: debezium/connect:1.1
ports:
- 8083:8083
links:
- schema-registry
environment:
- BOOTSTRAP_SERVERS=kafkaanalytics-mgmt.fptsinternal.com:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
- INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter