I'm trying to setup Kafka in a docker container for local development. My docker-compose.yml looks as follows:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181"
hostname: zookeeper
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092"
hostname: kafka
environment:
KAFKA_CREATE_TOPICS: "UploadFile:1:1,GetFile:1:1,TrackUpload:1:1,GetEmailContent:1:1" # topic:partition:replicas
KAFKA_ADVERTISED_HOST_NAME: kafka # docker-machine ip
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
Unfortunately my node app running on my localhost (not in docker) cannot connect to it. I used the url 'kafka:9092' and even 'localhost:9092'. Nothing works. Any idea what's happening?
Expose the host port 9092 for kafka service & you should be able to connect via "localhost:9092" from the app or host machine.
....
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092:9092"
....
Related
I am trying to set up a Docker compose file environment for Kafka change data capture and I am encountering this error:
Opening socket connection to server 0.0.0.0/0.0.0.0:2181. Will not attempt to authenticate using SASL (unknown error)
Socket error occurred: 0.0.0.0/0.0.0.0:2181: Connection refused
I have been following this tutorial https://hevodata.com/learn/kafka-cdc-postgres/, but it is running the docker commands directly using the link option and not using a docker-compose.yml file.
I attempted to convert these:
docker run -it --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka
to the below docker-compose.yml file. However, it appears it is completely ignoring the KAFKA_ZOOKEEPER_CONNECT environment variable as this is what I see in the log:
Using ZOOKEEPER_CONNECT=0.0.0.0:2181
Even though, the documentation https://github.com/debezium/docker-images/tree/master/kafka/1.5 indicates it should work.
When I follow the tutorial using docker run and not creating a docker-compose file it works completely fine. It shows my local computer's IP address with port 2181 instead of 0.0.0.0:2181.
docker-compose.yml:
version: "3.7"
services:
postgres:
image: debezium/postgres:10
container_name: postgres
ports:
- "5000:5432"
environment:
POSTGRES_HOST_AUTH_METHOD: trust
# POSTGRES_USER: db_user
# POSTGRES_PASSWORD: db_password
zookeeper:
image: debezium/zookeeper:1.5
container_name: zookeeper
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
kafka:
image: debezium/kafka:1.5
container_name: kafka
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
connect:
image: debezium/connect:1.5
container_name: connect
ports:
- "8083:8083"
environment:
GROUP_ID: 1
CONFIG_STORAGE_TOPIC: my-connect-configs
OFFSET_STORAGE_TOPIC: my-connect-offsets
depends_on:
- postgres
- kafka
- zookeeper
networks:
default:
name: kafkaCDC
zoo.cfg on the Zookeeper container:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/txns
clientPort=2181
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
Been looking at this issue for several; however, I am getting totally lost. Especially, since so many examples are using links.
This is the GitHub post that made me think to use KAFKA_ZOOKEEPER_CONNECT.
https://github.com/wurstmeister/kafka-docker/issues/512#issuecomment-505905161
Apart of me feels like it is something wrong with https://github.com/debezium/docker-images/blob/master/kafka/1.5/docker-entrypoint.sh that is ignoring the environment variable, but it is probably just me not understanding something and having a conf error.
For debezium/kafka:1.5 image to work in docker compose, you can try passing following environment variable:
ZOOKEEPER_CONNECT: "zookeeper:2181"
It addressed my problem, sample docker compose yaml below:
version: "3.9"
services:
zookeeper:
image: debezium/zookeeper:1.5
ports:
- "2181:2181"
- "2888:2888"
kafka:
image: debezium/kafka:1.5
ports:
- "9092:9092"
environment:
ZOOKEEPER_CONNECT: "zookeeper:2181"
depends_on:
- zookeeper
Don't prefix your ENVIRONMENT variables with KAFKA_
Here is my working cluster :
version: '2'
services:
postgres:
image: debezium/postgres:13-alpine
container_name: postgres
hostname: postgres
environment:
POSTGRES_USER: nikamooz
POSTGRES_PASSWORD: nikamooz
ports:
- 5432:5432
zookeeper:
image: debezium/zookeeper
container_name: zookeeper
hostname: zookeeper
environment:
ZOOKEEPER_SERVER_ID: 1
ports:
- 2182:2181
- 2888:2888
- 3888:3888
volumes:
- ./data/zoo/data:/zookeeper/data
- ./data/zoo/log:/zookeeper/txns
kafka:
image: debezium/kafka
container_name: kafka
hostname: kafka
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
ZOOKEEPER_CONNECT: zookeeper:2181
BOOTSTRAP_SERVERS: kafka:9092
volumes:
- ./data/kafka/data:/kafka/data
- ./data/kafka/logs:/kafka/logs
connect:
image: debezium/connect
container_name: connect
hostname: connect
depends_on:
- kafka
- postgres
ports:
- 8083:8083
environment:
GROUP_ID: holding_group
CONFIG_STORAGE_TOPIC: holding_storage_topic
OFFSET_STORAGE_TOPIC: holding_offset_topic
BOOTSTRAP_SERVERS: kafka:9092
I was able to fix it by setting the zookeeper connect address to the docker container IP address
To get the IP address run
docker inspect <container-name> --format='{{ .NetworkSettings.IPAddress }}'
and start start kafka as follows
docker run --name some-kafka -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=<zookeeper-ip>:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc/cp-kafka
This github comment helped me find out what I was missing
I'm trying to use microservices Spring Boot with Kafka, but my Spring Boot containers can not connect to the Kafka container.
docker-compose.yml:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
restart: always
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka
container_name: kafka
restart: always
ports:
- 9092:9092
depends_on:
- zookeeper
links:
- zookeeper:zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
consumer:
image: consumer
container_name: consumer
depends_on:
- kafka
restart: always
ports:
- 8084:8080
depends_on:
- kafka
links:
- kafka:kafka
producer:
image: producer
container_name: producer
depends_on:
- kafka
restart: always
ports:
- 8085:8080
depends_on:
- kafka
links:
- kafka:kafka
application.properties in Consumer:
spring.kafka.consumer.bootstrap-servers=kafka:9092
spring.kafka.consumer.group-id=WorkUnitApp
spring.kafka.consumer.topic=kafka_topic
application.properties in Producer:
spring.kafka.producer.bootstrap-servers=kafka:9092
But if I run the Kafka in a container and the Spring Boot microservices locally it works.
application.properties in Consumer:
spring.kafka.consumer.bootstrap-servers=0.0.0.0:9092
spring.kafka.consumer.group-id=WorkUnitApp
spring.kafka.consumer.topic=kafka_topic
application.properties in Producer:
spring.kafka.producer.bootstrap-servers=0.0.0.0:9092
What's the problem, why does the links from the docker not work ?
p.s. 0.0.0.0 because mac os
Edited
I added in docker-compose.yml environments to kafka but it still does not work either
- KAFKA_ADVERTISED_PORT=9092
You need to advertise your Kafka broker as kafka, which is the effective hostname for all linking containers (i.e. the hostname that the client needs to connect to from the Kafka protocol perspective, and so kafka:9092 is correct, not 0.0.0.0):
kafka:
...
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
I have written docker-compose.yml file to create the following containers:
Confluent-Zookeeper
Confluent-Kafka
Confluent-Schema Registry
I want a single docker-compose file to spun up the necessary containers, expose required ports and interconnect the dependent containers. The goal is to have
I am using the official confluent images from Docker Hub.
My docker-compose file looks like this:
zookeeper:
image: confluent/zookeeper
container_name: confluent-zookeeper
hostname: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "2181:2181"
kafka:
environment:
KAFKA_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
image: confluent/kafka
container_name: confluent-kafka
hostname: kafka
links:
- zookeeper
ports:
- "9092:9092"
schema-registry:
image: confluent/schema-registry
container_name: confluent-schema_registry
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
SCHEMA_REGISTRY_HOSTNAME: schema-registry
SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8081
SCHEMA_REGISTRY_DEBUG: 'true'
SCHEMA_REGISTRY_KAFKASTORE_TOPIC_REPLICATION_FACTOR: '1'
links:
- kafka
- zookeeper
ports:
- "8081:8081"
Now when I run docker-compose up, all these containers will be created and launched. But Schema Registry container exits immediately. docker logs gives the following output:
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
[2017-05-17 06:06:33,415] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)
org.apache.kafka.common.config.ConfigException: Only plaintext and SSL Kafka endpoints are supported and none are configured.
at io.confluent.kafka.schemaregistry.storage.KafkaStore.getBrokerEndpoints(KafkaStore.java:254)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.<init>(KafkaStore.java:111)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:136)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:53)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)
at io.confluent.rest.Application.createServer(Application.java:117)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)
I searched for this issue but nothing helped. I tried various other configurations like providing KAFKA_ADVERTISED_HOSTNAME, changing SCHEMA_REGISTRY_LISTENERS value, etc. but no luck.
Can anybody point out the exact configuration issue why Schema Registry container is failing?
Those are old and deprecated docker images. Use the latest supported docker images from confluentinc https://hub.docker.com/u/confluentinc/
You can find a full compose file here - confluentinc/cp-docker-images
You're missing the hostname (hostname: schema-registry) entry in the failing container. By default Docker will populate a container's /etc/hosts with the linked containers' aliases and names, plus the hostname of self.
The question is old, though it might be helpful to leave a solution that worked for me. I am using docker-compose:
version: '3.3'
services:
zookeeper:
image: confluent/zookeeper:3.4.6-cp1
hostname: "zookeeper"
networks:
- test-net
ports:
- 2181:2181
environment:
zk_id: "1"
kafka:
image: confluent/kafka:0.10.0.0-cp1
hostname: "kafka"
depends_on:
- zookeeper
networks:
- test-net
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: "kafka"
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_BROKER_ID: "0"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
schema-registry:
image: confluent/schema-registry:3.0.0
hostname: "schema-registry"
depends_on:
- kafka
- zookeeper
networks:
- test-net
ports:
- 8081:8081
environment:
SR_HOSTNAME: schema-registry
SR_LISTENERS: http://schema-registry:8081
SR_DEBUG: 'true'
SR_KAFKASTORE_TOPIC_REPLICATION_FACTOR: '1'
SR_KAFKASTORE_TOPIC_SERVERS: PLAINTEXT://kafka:9092
networks:
test-net:
driver: bridge`
I am running wurstmeister/kafka in a ubuntu 14.04 LTS machine. Kafka container is running fine. However I am not able to connect my microservice to this container.
Kafka docker-compose.yml:
version: '2'
services:
zookeeper:
image: user1/zookeeper:3.4.9
ports:
- "2181:2181"
kafka:
image: my-kafka
ports:
- "9092:9092"
hostname: kafka-01
environment:
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ADVERTISED_HOST_NAME: "kafka-01"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "topic1:1:1,topic2:1:1"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
docker-compose for microservice(myapp):
version: '2'
services:
myapp:
network_mode: 'bridge'
extends:
file: myapp.base.yml
service: myapp
links:
- kafka
zookeeper:
network_mode: 'bridge'
extends:
file: zookeeper.yml
service: zookeeper
kafka:
network_mode: 'bridge'
extends:
file: kafka.yml
service: kafka
links:
- zookeeper
I have linked kafka in an env:
kafkaHost=kafka-01:9092
Please let me know what am I doing wrong or if more info is required. Thanks in advance.
The hostname should be what you specified in "links", ie. "kafka". As #dnephin said the hostname is not needed. I think what you want is actually KAFKA_ADVERTISED_HOST_NAME environment variable.
What version of docker do you have? I am using docker 1.12.3 on Mac OS with no issues.
You might also want to try the Confluent docker images:
https://hub.docker.com/r/confluentinc/cp-kafka/
https://hub.docker.com/r/confluentinc/cp-zookeeper/
Here's my sample docker compose file:
version: "2"
services:
web:
image: nginx:latest
links:
- kafka
zookeeper:
extends:
file: kafka-services.yml
service: zookeeper
kafka:
extends:
file: kafka-services.yml
service: kafka
depends_on:
- zookeeper
My base kafka services compose file (kafka-services.yml):
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
ports:
- "32181:32181"
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:latest
ports:
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092
The hostname field does not set the public hostname, it sets the internal container hostname. It's generally not that useful.
Hostnames are automatically provided by docker networking, use kafka as the hostname.
When i was running the kafka and zookeeper without Docker, I could see the topic partitions log files in the /tmp/kafka-logs directory. Now with Docker, even though i specify the log directory in the Volumes section in docker-compose.yml, i cant see the files in the docker VM, like "TOPICNAME-PARTITIONNUMBER".. Is there anything I'm missing here ? Any idea on where i could find these directories in Docker VMs..
zookeeper:
image: confluent/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
- "15001:15000"
environment:
ZK_SERVER_ID: 1
volumes:
- /tmp/docker/zk1/logs:/logs
- /tmp/docker/zk1/data:/data
kafka1:
image: confluent/kafka
container_name: kafka1
ports:
- "9092:9092"
- "15002:15000"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_STORAGE: kafka
# This is Container IP
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
volumes:
- /tmp/docker/kafka1/logs:/logs
- /tmp/docker/kafka1/data:/data
this is how we configured for logs in our compose file and it has the log files in it. You should jump onto the container to see the '/var/lib/kafka/data', directory and the data inside it
volumes:
-kb1_data:/var/lib/kafka/data
Remember that 1st parameter in list for volumes, ports and other fields about sharing resources in docker-compose is about host, the 2nd is about the container.
So you should change the order for your volumes values.
zookeeper:
image: confluent/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
- "15001:15000"
environment:
ZK_SERVER_ID: 1
volumes:
#- ./host/folder:/container/folder
- ./logs:/tmp/docker/zk1/logs
- ./data:/tmp/docker/zk1/data
kafka1:
image: confluent/kafka
container_name: kafka1
ports:
#- "host-port:container-port"
- "9092:9092"
- "15002:15000"
links:
- zookeeper
environment:
- KAFKA_BROKER_ID: 1
- KAFKA_OFFSETS_STORAGE: kafka
- # This is Container IP
- KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
volumes:
#- ./host/folder:/container/folder
- ./logs:/tmp/docker/kafka1/logs
- ./data:/tmp/docker/kafka1/data