How to expose Kafka from Docker to the outside world? - docker

Question

Having to communicate with Kafka from a Dockerized Spring-Boot application,
the only option I was able to get working was Dockerizing Kafka too.
Here is my docker-compose-yml:
version: '3.5'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
networks:
- kafka-network
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
networks:
- kafka-network
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
kafka-network:
name: kafka-network
This way I'm able to connect to the Kafka broker from anothe container on the kafka-network using the url kafka:9092
How do I make it also available from the localhost and from other machines?
UPDATE
I updated my docker-compose as follows:
version: '3.5'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
networks:
- kafka-network
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
networks:
- kafka-network
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: ${KAFKA_LISTENERS:-PLAINTEXT://:9092}
KAFKA_ADVERTISED_LISTENERS: ${KAFKA_ADVERTISED_LISTENERS:-PLAINTEXT://127.0.0.1:9092}
depends_on:
- zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
kafka-network:
name: kafka-network
and created a .env file with the following content :
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://the.ip.of.machine:9092
I tested it on my PC (without the .env file) and I'm able to communicate with the broker using kafkakat from localhost:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6844a16fa14f wurstmeister/kafka "start-kafka.sh" 5 seconds ago Up 3 seconds 0.0.0.0:9092->9092/tcp kafka-compose_kafka_1_9573f71109c7
15d62557f3bd wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 6 seconds ago Up 4 seconds 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp kafka-compose_zookeeper_1_61a19213cde7
$ kafkacat -P -b localhost:9092 -t topic1
New test
^C
$ kafkacat -C -b localhost:9092 -t topic1
New test
% Reached end of topic topic1 [0] at offset 1
$ kafkacat -b localhost:9092 -L
Metadata for all topics (from broker -1: localhost:9092/bootstrap):
1 brokers:
broker 1001 at 127.0.0.1:9092
4 topics:
...
However I'm not able to do the same on the server, the only difference is the IP of the host machine for KAFKA_ADVERTISED_LISTENERS.
What I can see is that it keeps saying leader not available
# kafkacat -b localhost:9092 -L
Metadata for all topics (from broker -1: localhost:9092/bootstrap):
1 brokers:
broker 1002 at the.ip.of.machine:9092
topic "__consumer_offsets" with 50 partitions:
partition 0, leader -1, replicas: 1001, isrs: , Broker: Leader not available
partition 1, leader -1, replicas: 1001, isrs: , Broker: Leader not available
partition 2, leader -1, replicas: 1001, isrs: , Broker: Leader not available
partition 3, leader -1, replicas: 1001, isrs: , Broker: Leader not available
partition 4, leader -1, replicas: 1001, isrs: , Broker: Leader not available
partition 5, leader -1, replicas: 1001, isrs: , Broker: Leader not available
Shouldn't I set the IP of the server for the KAFKA_ADVERTISED_LISTENERS ?

Answer 1

You have to set the LISTENERS in environments in order to expose the Kafka brokers to external network like below :
KAFKA_LISTENERS: PLAINTEXT://:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://one.prod.com:9092
Here is the example :
https://github.com/wurstmeister/kafka-docker/blob/85821409d4d49a4edc7c5be83b68b71eceeab1bc/docker-compose-swarm.yml
You can refer here for more details:
https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Answer 2

For new kafka versions try to look here:
https://www.kaaproject.org/kafka-docker
https://github.com/bitnami/bitnami-docker-kafka/issues/29#issuecomment-435216430
For old kafka if you have only one computer might help this kafka variables:
environment:
ADVERTISED_HOST: ${COMPUTERNAME}
ADVERTISED_PORT: "9092"