Unable to connect to Kafka from outside application - docker

Question

I have two docker machines and I want to create a kafka cluster inside docker swarm. My docker-compose.yml looks like this:
version: '3.2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:latest
ports:
- "9092:9092"
- "29092:29092"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
I followed this question: Unable to connect to Kafka run in container from Spring Boot app run outside container and I am trying to access kafka from outside using localhost:29092.
I have already create the topic mytesttopic inside kafka. The below python code:
from kafka import KafkaConsumer, SimpleProducer, TopicPartition, KafkaClient
def consume_from_topic():
try:
consumer = KafkaConsumer('mytesttopic',
group_id= None,
bootstrap_servers=['localhost:29092'],
auto_offset_reset='earliest')
for message in consumer:
#consumer.commit()
print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
message.offset, message.key,
message.value))
except Exception as e:
print(e)
pass
if __name__ == '__main__':
consume_from_topic()
returns:
NoBrokersAvailable
Does anyone know what I am missing here?

Answer 1

Given you are running docker swarm on 2 other machines you won't be able to connect on localhost:29092 due to the fact that kafka will be exposed on port 29092 on your nodes of the docker swarm. Try connecting to kafka by using the hostname of one of your nodes + port 29092. You should be able to connect to kafka this way.
Please note that this will only work if you are running docker swarm with routing mesh, the routing mesh makes sure that each node accepts incoming requests on a published port for any service, no matter if it is running on the same hosts and makes sure the traffic reaches the actual host where your container is running.
If you have not yet setup routing mesh try connceting to the actual hostname where a kafka container is running (not recommended, but for testing purposes it works)
I hope this helps you!

Answer 2

Your listeners are the exact same.
You need to set PLAINTEXT_HOST://0.0.0.0:29092 to bind the listener to all interfaces