accessing kafka running in docker-compose from other machines - docker

I want to run kafka in a single node, single broker, in one of computers on our network and be able to access it from other machines. for example by running docker-compose on 192.168.0.36 I want to access it from 192.168.0.19.
since we can't use any Linux distribution I have to run kafka as a docker container on windows.
I know there are already a ton of questions and documents on this topic including this question and this example and also this blog post, but unfortunately none of them worked out for me.
this is the compose file I'm using right now:
version: '3.7'
services:
zookeeper:
image: wurstmeister/zookeeper:3.4.6
ports:
- "2181:2181"
expose:
- "2181"
volumes:
- type: bind
source: "G:\\path\\to\\zookeeper"
target: /opt/zookeeper-3.4.6/data
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
expose:
- "9092"
environment:
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9093, OUTSIDE://192.168.0.36:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT, OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9092
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_BROKER_ID: 1
KAFKA_LOG_DIRS: "/kafka"
volumes:
- type: bind
source: "G:\\path\\to\\logs"
target: /kafka/
things I tried for debugging the issue:
alraedy tried all the different configurations in mentioned questions
and blog posts.
I can access Kafka from 192.168.0.36 which is machine running docker-compose but not from
192.168.0.19 (NoBrokersAvailable error in kafka-python).
just to see if it's internal networking problem or not, I tried a similar docker-compose file running a falcon API using gunicorn and I can call the API from 192.168.0.19.
I also tried the windows telnet tool to see the 9092 port is
accessible from different machines, it's accessible from 0.36 but not
from 0.19.
tried using a custom network like this one
I'm testing the connection using python's kafka-python package. we have a multi-broker kafka running on our on-premise kubernetes cluster and it's working fine, so I don't think my testing scripts have any issues.
UPDATE
as OneCricketeer suggested, I tried this solution with different configurations like 0.0.0.0:9092=>127.0.0.1:9092 and 192.168.0.36:9092=>127.0.0.1:9092. also disabled firewall. still getting NoBrokersAvailable but at least I can access 0.36:9092 from other machine's telnet now.

Related

Kafka container connecting from inside the container and outside using the same hostname

I am aware this topic has several questions and blog posts about it. I am following these two:
https://rmoff.net/2018/08/02/kafka-listeners-explained/
https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
But unfortunately, without success.
I'm trying to make it so the same code will work whether I'm running it from my IDE where the kafka client is in a container, or whether the code I'm running is in a container within the network. I am able to make each scenario work on its own, but not the two together.
My docker compose:
zoo1:
image: confluentinc/cp-zookeeper:7.2.1
hostname: zoo1
container_name: zoo1
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_SERVERS: zoo1:2888:3888
kafka1:
image: confluentinc/cp-kafka:7.2.1
hostname: kafka1
container_name: kafka1
ports:
- "9092:9092"
- "29092:29092"
environment:
KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
depends_on:
- zoo1
In this docker-compose, communication within the docker network using kafka1:29092 as bootstrap works great. but, from my laptop using the same doesn't work.
Is there anyway to ensure that both locally and inside the container network I can bootstrap to kafka1:29092? Do I even need the external listener?
Thanks
anyway to ensure that both locally and inside the container network I can bootstrap to kafka1:29092?
No.
Your host isn't aware of the DNS / service names used by Docker.
Instead, add an environment variable in your code like KAFKA_BOOTSTRAP_SERVERS, then set that as a variable in your IDE (as localhost:9092) via a run config, or as a container variable (kafka1:29092)
You can also remove - "29092:29092" from your compose file since your host will never need that port to connect with the broker

Can not test apache kafka official site commands using docker: no such file or directory

I have run the apache kafka and prometheus using docker. I will attach the docker-compose and other configurations at the bottom of this post!
Introduction: First I should explain that each metric of kafka works well on prometheus. So there is no problem in the implementation and running of the images.
Problem: The only problem is where I want to test the stream (Producer, Broker and Consumer) following the tutorial of the official site of apache kafka. But whenever I execute the commands found on the site, I faced with the command not found error, because I don't know where the files exactly are! As an example whenever I execute the bin/zookeeper-server-start.sh config/zookeeper.properties command I face with the following error:
no such file or directory: bin/zookeeper-server-start.sh
Attachments:
docker-compose.yml:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
links:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OPTS: -javaagent:/usr/app/jmx_prometheus_javaagent.jar=7071:/usr/app/prom-jmx-agent-config.yml
volumes:
- /var/run/docker.sock:/var/run/docker.sock
prometheus:
image: prom/prometheus
ports:
- 9090:9090/tcp
volumes:
- ./mount/prometheus:/etc/prometheus
links:
- kafka
Dockerfile:
FROM wurstmeister/kafka
ADD prom-jmx-agent-config.yml /usr/app/prom-jmx-agent-config.yml
ADD jmx_prometheus_javaagent-0.10.jar /usr/app/jmx_prometheus_javaagent.jar
Question: Is there any solution to find where are the original files are mapped in the created container and execute them?
The quickstart on the Apache site never references Docker. Those scripts need downloaded (as part of Kafka), or you need to docker exec into the container to run them
However, Docker already starts Kafka and Zookeeper, so you wouldn't need to run those commands. You therefore could skip to writing your own producers/consumers without using any provided scripts

Using docker service names from outside the network

My app has 2 dependencies which I specify in my docker-compose, a postgres and kafka service:
services:
postgres:
image: postgres:9.6-alpine
ports:
- "5432:5432"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
I run my code and tests outside the docker network, and use these two containers as my dependencies.
As these both expose ports, I can configure my app to hit them via: localhost:5432, localhost:9092. This works.
The problem I have is when I want to test the app image itself, I add this as a service to the docker-compose file:
app:
image: myapp
links:
- postgres
- kafka
The app is still configured to use localhost, so I allow the app container to access my network using --net=host
Whilst the app container can now access localhost:5432 and localhost:9092 (confirmed by curling from inside the container), the host names fail to resolve when the code runs and the dependencies are unreachable - possibly as a result of using localhost from inside the container and confusing the client libraries? I'm really not sure.
It feels like the use of localhost in the app configuration isn't the right approach here. Is it possible to refer to the service names 'postgres' and 'kafka' from outside the docker network?
Why do you want to continue using localhost:xxx in your app?
The best approach for you is to change connection strings in your application when it is being launched from docker-compose. You just use postgres:5432 and kafka:9092 and everything will work, because inside docker-compose network all machines are visible to each other under their service names.
If for some great reasons you insist on using localhost as a connection target, you need to turn all services into host mode. But remember - in this case ports are not exposed, so you access services with their original port values.
version: '3'
services:
postgres:
image: postgres:9.6-alpine
network_mode: "host"
kafka:
image: wurstmeister/kafka
network_mode: "host"
app:
image: myapp
network_mode: "host"
And by the way, forget about links. They are deprecated.

Exposing docker-compose services to the HOST machine as well

Here is my v3.5 docker-compose.yml definition file. It has an analytics network (using an alias of the same name), and where both included services connect to said network to communicate with one another. This works.
However, I want these services (ports) exposed to the HOST machine, as well. There's a way to do that by defining an additional network and/or specifying additional ports: entries within the services themselves, but I can't figure out exactly how because the documentation is very confusing and version-specific (moving targets).
Without destroying the below (because it works internally), what additions do I make (and where) to expose both services to the HOST machine as well?
Thank you!
version: '3.5'
networks:
analytics:
name: analytics
driver: bridge
# ===========================================
# Service: Zookeeper
# ===========================================
zookeeper:
image: 'wurstmeister/zookeeper:latest'
container_name: analytics-ZooKeeper
networks:
- analytics
ports:
- "2181:2181"
volumes:
- ./data.d/zookeeper.d:/opt/zookeeper-3.4.9/data
# ===========================================
# ===========================================
# Service: Kafka
# ===========================================
kafka:
build:
context: ./kafka.d
dockerfile: Dockerfile
image: nmvega/kafka:latest
networks:
- analytics
ports:
- 9092-9094:9092 # For one to three Kafka brokers.
environment:
#KAFKA_ADVERTISED_HOST_NAME: vps00 # Docker host Name. <--- BEFORE
KAFKA_ADVERTISED_HOST_NAME: 192.168.0.180 # Docker host IP. <--- AFTER
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./data.d/kafka.d:/kafka
depends_on:
- zookeeper
# ===========================================
EDIT:
Upon further investigation, the above configuration, as originally posted, is correct with the small modification from the name of the Docker Host to the IP of the Docker host (as prescribed by the readme for the image that I'm using). Accidentally using the name didn't matter until I attempted to access the service from the Host.
Hopefully this example will be valuable to others wanting to see one.
Thank you to the commenters below.

dynamic KAFKA_ADVERTISED_HOST_NAME on kafka continer

I use kafka in docker container.
One of the requirements is that the kafka will be available to a producer natively running on the host machine.
This is why I set the KAFKA_ADVERTISED_HOST_NAME to my host ip.
my docker-compose.yml looks like this:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.10
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERES: PLAINTEXT://localhost:9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
and it works.
The problem is, I want to be able to use this docker-compose file also on other machines, and I don't know what their IP might be.
trying to change the ip into a name like 'kafka' caused it to be unavailable to the host machine (although still available from other containers).
Is there a way to use the host IP in the docker-compose file without "hardcoding" it (so that it will be a different IP address on different machines)?
Is there another way of addressing this issue?
did you try using the property? It not recommend for production usage for whatever reason but it might work for you
host.docker.internal
https://docs.docker.com/docker-for-mac/networking/#there-is-no-docker0-bridge-on-macos#i-want-to-connect-from-a-container-to-a-service-on-the-host
https://docs.docker.com/docker-for-windows/networking/#there-is-no-docker0-bridge-on-windows#i-want-to-connect-from-a-container-to-a-service-on-the-host

Resources