Kafka producer says "unknown_topic_or_partition" - docker

I've been trying to make kafka-docker work for a few days now and I don't know what I'm doing wrong. Right now, I can't access any topic with my ruby-kafka client because the node "doesn't exist". This is my docker-compose.yml file:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9092:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka2:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9093:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 2
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka3:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9094:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 3
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9094
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
I specify "KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'" because I want to create topics by hand, so I entered into my first broker container and typed this:
./kafka-topics.sh --create --zookeeper 172.19.0.2:2181 --topic test1 --partitions 4 --replication-factor 3
And everything seems fine:
./kafka-topics.sh --list --zookeeper 172.19.0.2:2181 -> test1
But, when I try to do this:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test1
It says:
WARN Error while fetching metadata with correlation id 24 : {test1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
If I create again the topic, it says it already exists, so I don't know what is happening anymore.

You need to get your networking configuration right, as Kafka works across hosts and needs to be able to access them all.
This post explains it in detail.
You might also want to reference https://github.com/confluentinc/cp-docker-images/blob/5.0.0-post/examples/cp-all-in-one/docker-compose.yml for an example of a working Docker Compose.

so we got this issue when we were working with kafka connect. Thre are multiple solutions to this. Either prune all the docker images or change the group id in the configuration for connect in the connect image as below:-
image: debezium/connect:1.1
ports:
- 8083:8083
links:
- schema-registry
environment:
- BOOTSTRAP_SERVERS=kafkaanalytics-mgmt.fptsinternal.com:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
- INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter

Related

Run provectus/kafka-ui for a Kafka cluster on a different Server

I'm in a bit of trouble trying to run provectus/kafka-ui docker containers.
I have a collection of servers all running on the same internal network. On three of these servers a Kafka cluster is implemented (Zookeeper and Broker). The structure of the Broker looks like the following:
192.168.0.237:9092
192.168.0.7:9092
192.168.0.54:9092
The problem is that there is no Docker instance running on any of these servers.
What I would like to achieve now is to install provectus/kafka-ui on another server (192.168.0.8) and make it accessible from within the network via a specific port (e.g. 8080).
I have found a few docker-compose.yml examples https://github.com/provectus/kafka-ui. However, the Kafka cluster is created on the same server as kafka-ui. In my case, the cluster already exists and only kafka-ui is missing on another server. Can anyone tell me how I can adapt the docker-compose.yml so that kafka-ui can be installed on 192.168.0.8 and made available from outside via port 8080?
Kind regards and thanks in advance
version: '3'
services:
kafka-ui:
container_name: kafka-ui
image: provectuslabs/kafka-ui:latest
ports:
- 8080:8080
depends_on:
- zookeeper
- kafka1
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka1:19091
KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
KAFKA_CLUSTERS_0_JMXPORT: 9997
SERVER_SERVLET_CONTEXT_PATH: /kafkaui
AUTH_TYPE: "LOGIN_FORM"
SPRING_SECURITY_USER_NAME: test_user
SPRING_SECURITY_USER_PASSWORD: test
zookeeper:
image: confluentinc/cp-zookeeper:5.2.4
hostname: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
volumes:
- ./data/zookeeper/data:/data
- ./data/zookeeper/datalog:/datalog
kafka1:
image: confluentinc/cp-kafka:5.3.1
hostname: kafka1
ports:
- "9091:9091"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9091
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "False"
volumes:
- ./data/kafka1/data:/var/lib/kafka/data
depends_on:
- zookeeper
kafka2:
image: confluentinc/cp-kafka:5.3.1
hostname: kafka2
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka2:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_BROKER_ID: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "False"
volumes:
- ./data/kafka2/data:/var/lib/kafka/data
depends_on:
- zookeeper
kafka3:
image: confluentinc/cp-kafka:5.3.1
hostname: kafka3
ports:
- "9093:9093"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka3:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 3
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "False"
volumes:
- ./data/kafka3/data:/var/lib/kafka/data
depends_on:
- zookeeper
kafdrop:
image: obsidiandynamics/kafdrop
restart: "no"
ports:
- "9000:9000"
environment:
KAFKA_BROKERCONNECT: "kafka1:19091"
depends_on:
- kafka1
- kafka2
- kafka3
You don't need Docker on the Kafka brokers... Also, IP doesn't really matter as long as advertised.listeners on Kafka is setup correctly.
Remove the three Kafka containers, the Zookeeper container, and simply change these to your actual Kafka brokers + Zookeeper servers
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
KAFKA_CLUSTERS_0_ZOOKEEPER
Also remove kafdrop? Since it's basically another UI with overlapping features.
Or, install pnpm and Java, and you can run the UI + API without Docker, just like you're doing with Kafka.
https://github.com/provectus/kafka-ui/tree/master/kafka-ui-react-app

kafka broker connection failed when create new topic with a cluster of 3 brokers

I'm trying to setup a kafka cluster with 3 brokers on Docker.
The problem is: when I do an operation (i.e create/list/delete topics), there's always 1 broker fails to be connected and restart Docker container. This problem doesn't happen on a cluster of 2 or single broker.
My steps to reproduce is:
Run docker-compose up
Open shell of 1 of kafka containers and create a topic kafka-topics --bootstrap-server ":9092" --create --topic topic-name --partitions 3 --replication-factor 3
After this, 1 random broker is disconnected and deleted from the cluster. Sometimes the reponse of the above execution is the error said that replication factor cannot be larger than 2 (since 1 broker has removed from cluster)
I'm new to Kafka. I think I'm just having some silly mistakes but I don't have any clue of what it is. I search through docs but haven't found yet.
Here is my docker-compose file:
version: "3.9"
networks:
kafka-cluster:
driver: bridge
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
# ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 2181
# ZOOKEEPER_TICK_TIME: 2000
# ZOOKEEPER_SERVERS: "zookeeper:22888:23888"
KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=*"
ports:
- 2181:2181
restart: unless-stopped
networks:
- kafka-cluster
kafka1:
image: confluentinc/cp-kafka:latest
container_name: kafka1
depends_on:
- zookeeper
ports:
- "9093:9093"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: CLIENT://:9092,EXTERNAL://:9093
KAFKA_ADVERTISED_LISTENERS: CLIENT://kafka1:9092,EXTERNAL://localhost:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: CLIENT
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
restart: unless-stopped
networks:
- kafka-cluster
kafka2:
image: confluentinc/cp-kafka:latest
container_name: kafka2
depends_on:
- zookeeper
ports:
- "9094:9094"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: CLIENT://:9092,EXTERNAL://:9094
KAFKA_ADVERTISED_LISTENERS: CLIENT://kafka2:9092,EXTERNAL://localhost:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: CLIENT
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
restart: unless-stopped
networks:
- kafka-cluster
kafka3:
image: confluentinc/cp-kafka:latest
container_name: kafka3
depends_on:
- zookeeper
ports:
- "9095:9095"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: CLIENT://:9092,EXTERNAL://:9095
KAFKA_ADVERTISED_LISTENERS: CLIENT://kafka3:9092,EXTERNAL://localhost:9095
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: CLIENT
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
restart: unless-stopped
networks:
- kafka-cluster
kafdrop:
image: obsidiandynamics/kafdrop:latest
container_name: kafdrop
ports:
- "9000:9000"
environment:
- KAFKA_BROKERCONNECT=kafka1:9092,kafka2:9092,kafka3:9092
- JVM_OPTS="-Xms32M -Xmx64M"
- SERVER_SERVLET_CONTEXTPATH="/"
depends_on:
- kafka1
networks:
- kafka-cluster
Here is the error log on the other 2 brokers:
[2022-01-17 04:32:40,078] WARN [ReplicaFetcher replicaId=1002, leaderId=1001, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=1002, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={test-topic-3-1=PartitionData(fetchOffset=0, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty), test-topic-2-1=PartitionData(fetchOffset=0, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=28449961, epoch=INITIAL), rackId=) (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to kafka1:9092 (id: 1001 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:104)
at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:218)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:321)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:137)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:136)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:136)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:119)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2022-01-17 04:32:42,088] WARN [ReplicaFetcher replicaId=1002, leaderId=1001, fetcherId=0] Connection to node 1001 (kafka1/192.168.48.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-01-17 04:32:42,088] INFO [ReplicaFetcher replicaId=1002, leaderId=1001, fetcherId=0] Error sending fetch request (sessionId=28449961, epoch=INITIAL) to node 1001: (org.apache.kafka.clients.FetchSessionHandler)
java.io.IOException: Connection to kafka1:9092 (id: 1001 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:104)
at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:218)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:321)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:137)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:136)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:136)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:119)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Assuming you don't need host connections (since you're running the Kafka CLI commands directly in the containers), you could greatly simplify your Compose file
Remove host ports
Remove non-CLIENT listeners, and stick to the defaults.
Remove the Compose network (for debugging) since one is automatically created
All in all, you'd end up with something like this
x-kafka-setup: &kafka-setup
KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
ALLOW_PLAINTEXT_LISTENER: 'yes'
version: "3.8"
services:
zookeeper:
image: docker.io/bitnami/zookeeper:3.7
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka1:
image: &broker-image docker.io/bitnami/kafka:3
environment:
KAFKA_BROKER_ID: 1
<<: *kafka-setup
depends_on:
- zookeeper
kafka2:
image: *broker-image
environment:
KAFKA_BROKER_ID: 2
<<: *kafka-setup
depends_on:
- zookeeper
kafka3:
image: *broker-image
environment:
KAFKA_BROKER_ID: 3
<<: *kafka-setup
depends_on:
- zookeeper
kafdrop:
image: obsidiandynamics/kafdrop:latest
ports:
- "9000:9000"
environment:
KAFKA_BROKERCONNECT: kafka1:9092,kafka2:9092,kafka3:9092
JVM_OPTS: "-Xms32M -Xmx64M"
SERVER_SERVLET_CONTEXTPATH: /
depends_on:
- kafka1
- kafka2
- kafka3

Creating Kafka Topic with docker-compose

I am starting my first docker proyect, trying to setup zookeeper and Kafka.
I have the following docker-compose.yml:
version: "3.8"
networks:
mynet:
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
networks:
- mynet
kafka:
image: wurstmeister/kafka:2.12-2.4.0
ports:
- "9092:9092"
expose:
- "9093"
environment:
KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
# Create a topic NAME:PARTITION:REPLICAS
KAFKA_CREATE_TOPICS: "example-topic:1:1"
networks:
- mynet
kafka-manager:
image: sheepkiller/kafka-manager:latest
environment:
ZK_HOSTS: "zookeeper:2181"
ports:
- 9000:9000
networks:
- mynet
I execute the docker-compose.yml:
sudo docker-compose up -d
To check if I have the topic created, I access the Kafka shell through:
sudo docker exec -it <<CONTAINER ID>> sh
In this shell I go to:
cd opt/kafka
And execute:
bin/kafka-topics.sh --list --zookeeper localhost:2181
And the output is a timeout error:
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:321)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Zookeeper isn't running in the Kafka container.
You should use --bootstrap-server localhost:9092 anyway since the Zookeeper argument is deprecated

Kafka producer error "1 partitions have leader brokers without a matching listener"

First time working with Kafka and Docker-compose. I'm trying to publish a message to Kafka but I get an error (look below). What is the issue?
2020-07-21 16:37:40,274 WARN [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater: [Producer clientId=producer-1] 1 partitions have leader brokers without a matching listener, including [demo-topic-0]
Here is my docker-compose.yml:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
links:
- zookeeper:zk
ports:
- "9092:9092"
expose:
- "9093"
environment:
KAFKA_ZOOKEEPER_CONNECT: zk:2181
KAFKA_MESSAGE_MAX_BYTES: 2000000
KAFKA_CREATE_TOPICS: "demo-topic:1:1"
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9093,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_LISTENERS: PLAINTEXT://kafka:9093,PLAINTEXT_HOST://localhost:9092
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- zookeeper
player-service-ci:
image: player/player-service:latest
container_name: player-service-ci
restart: unless-stopped
volumes:
- /tmp/app/logs:/logs
environment:
- "JAVA_OPTS=-Xmx256m -Xms128m"
- "spring.profiles.active=ci"
- "LOGS_FILENAME=player-service-logger-ci"
- "SPRING_KAFKA_BOOTSTRAPSERVERS=kafka:9093"
ports:
- 17500:17500
networks:
default:
external:
name: ci
My question was partially answered here Leader brokers without a matching listener error in kafka.
docker-compose rm -sfv
The above code ultimately resolved the issue of multiple consumers.

Kafka-docker: can't produce messages

I set up kafka in a docker container using this project https://github.com/wurstmeister/kafka-docker. I can successfully create and list topics on it, but once I try to produce a message with either
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
or
bin/kafka-console-producer.sh --broker-list 0.0.0.0:9092 --topic test
I get the following error:
ERROR Error when sending message to topic test with key: null, value:
4 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s)
for test-0: 1536 ms has passed since batch creation plus linger time
My docker-compose.yml lookes like this:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: docker.for.mac.localhost
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
I tried different options for the KAFKA_ADVERTISED_HOST_NAME property, including docker.for.mac.localhost which is supposed to be resolved as the actual docker host ip, and 0.0.0.0, but the result is the same.
You can try with this docker-compose, it's working for me:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
stdin_open: true
tty: true
ports:
- 2181:2181/tcp
kafka:
image: wurstmeister/kafka:0.11.0.1
environment:
KAFKA_CREATE_TOPICS: test:1:1
KAFKA_ZOOKEEPER_CONNECT: <<HOST_IP>>:2181
KAFKA_ADVERTISED_HOST_NAME: <<HOST_IP>>
KAFKA_ADVERTISED_PORT: '9092'
KAFKA_BROKER_ID: '999'
stdin_open: true
tty: true
links:
- zookeeper:zookeeper
ports:
- 9092:9092/tcp
Don't use local IPs like 127.0.0.1. Use the true IP instead.

Resources