I am trying to understand a docker-compose.yml (also shown at the bottom of this post).
Question: confluent:2181 is used the following line
KAFKA_ZOOKEEPER_CONNECT: "confluent:2181"
How was this confluent hostname defined? If I understand hostnames in Docker correctly, the only container hostnames are zookeeper, kafka, rest-proxy and schema-registry.
docker-compose.yml
version: "2"
services:
zookeeper:
image: confluent/zookeeper
ports:
- "2181:2181"
environment:
zk_id: "1"
network_mode: "host"
kafka:
image: confluent/kafka
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: "confluent:2181"
network_mode: "host"
rest-proxy:
image: confluent/rest-proxy
depends_on:
- zookeeper
- kafka
- schema-registry
ports:
- "8082:8082"
environment:
RP_ZOOKEEPER_CONNECT: "confluent:2181"
RP_SCHEMA_REGISTRY_URL: "http://confluent:8081"
network_mode: "host"
schema-registry:
image: confluent/schema-registry
depends_on:
- kafka
- zookeeper
ports:
- "8081:8081"
environment:
SR_KAFKASTORE_CONNECTION_URL: "confluent:2181"
network_mode: "host"
You're right. According to the docker-compose.yml the services being defined will provide DNS resolution only for zookeeper, kafka, rest-proxy and schema-registry, and not confluent.
However if you take a look at the documenttion from confluentinc they require you to modify your hosts file in your host machine:
Edit your hosts file and add a host entry for the docker machine.
192.168.99.100 confluent
That's why you can use confluent and get name resolution. It doesn't have anything to do with docker or compose.
P.S: Beware that you're using deprecated images according to their documentation
Yes, you are right.
According to the README, you were supposed to run confluent firstly.
Related
---
version: "3.6"
services:
postgres:
image: postgres:alpine
restart: on-failure
environment:
- POSTGRES_USER=${APP_POSTGRES_USER:-postgres}
- POSTGRES_PASSWORD=${APP_POSTGRES_PASS:-postgres}
- POSTGRES_DB=${APP_POSTGRES_DB:-my_proj}
ports:
- "5566:5432"
server:
container_name: my_proj_app
hostname: my_proj_app
build:
context: .
depends_on:
- postgres
network_mode: host
environment:
- PORT=8080
- HOST=my_proj_app
ports:
- "8080:8080"
Here is my docker-compose.yml
I can't ping google.com from my_proj_app container.
Have anybody ideas what I'm doing wrong?
The error is explained here https://docs.docker.com/network/host/ : you used the host network mode which cannot access outside network, unlike the NAT.
Host mode networking can be useful to optimize performance, and in situations where a container needs to handle a large range of ports, as it does not require network address translation (NAT), and no “userland-proxy” is created for each port.
Try commenting network_mode: host.
I'm trying to setup Kafka in a docker container for local development. My docker-compose.yml looks as follows:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181"
hostname: zookeeper
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092"
hostname: kafka
environment:
KAFKA_CREATE_TOPICS: "UploadFile:1:1,GetFile:1:1,TrackUpload:1:1,GetEmailContent:1:1" # topic:partition:replicas
KAFKA_ADVERTISED_HOST_NAME: kafka # docker-machine ip
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
Unfortunately my node app running on my localhost (not in docker) cannot connect to it. I used the url 'kafka:9092' and even 'localhost:9092'. Nothing works. Any idea what's happening?
Expose the host port 9092 for kafka service & you should be able to connect via "localhost:9092" from the app or host machine.
....
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092:9092"
....
I have written docker-compose.yml file to create the following containers:
Confluent-Zookeeper
Confluent-Kafka
Confluent-Schema Registry
I want a single docker-compose file to spun up the necessary containers, expose required ports and interconnect the dependent containers. The goal is to have
I am using the official confluent images from Docker Hub.
My docker-compose file looks like this:
zookeeper:
image: confluent/zookeeper
container_name: confluent-zookeeper
hostname: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "2181:2181"
kafka:
environment:
KAFKA_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
image: confluent/kafka
container_name: confluent-kafka
hostname: kafka
links:
- zookeeper
ports:
- "9092:9092"
schema-registry:
image: confluent/schema-registry
container_name: confluent-schema_registry
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
SCHEMA_REGISTRY_HOSTNAME: schema-registry
SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8081
SCHEMA_REGISTRY_DEBUG: 'true'
SCHEMA_REGISTRY_KAFKASTORE_TOPIC_REPLICATION_FACTOR: '1'
links:
- kafka
- zookeeper
ports:
- "8081:8081"
Now when I run docker-compose up, all these containers will be created and launched. But Schema Registry container exits immediately. docker logs gives the following output:
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
[2017-05-17 06:06:33,415] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)
org.apache.kafka.common.config.ConfigException: Only plaintext and SSL Kafka endpoints are supported and none are configured.
at io.confluent.kafka.schemaregistry.storage.KafkaStore.getBrokerEndpoints(KafkaStore.java:254)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.<init>(KafkaStore.java:111)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:136)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:53)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)
at io.confluent.rest.Application.createServer(Application.java:117)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)
I searched for this issue but nothing helped. I tried various other configurations like providing KAFKA_ADVERTISED_HOSTNAME, changing SCHEMA_REGISTRY_LISTENERS value, etc. but no luck.
Can anybody point out the exact configuration issue why Schema Registry container is failing?
Those are old and deprecated docker images. Use the latest supported docker images from confluentinc https://hub.docker.com/u/confluentinc/
You can find a full compose file here - confluentinc/cp-docker-images
You're missing the hostname (hostname: schema-registry) entry in the failing container. By default Docker will populate a container's /etc/hosts with the linked containers' aliases and names, plus the hostname of self.
The question is old, though it might be helpful to leave a solution that worked for me. I am using docker-compose:
version: '3.3'
services:
zookeeper:
image: confluent/zookeeper:3.4.6-cp1
hostname: "zookeeper"
networks:
- test-net
ports:
- 2181:2181
environment:
zk_id: "1"
kafka:
image: confluent/kafka:0.10.0.0-cp1
hostname: "kafka"
depends_on:
- zookeeper
networks:
- test-net
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: "kafka"
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_BROKER_ID: "0"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
schema-registry:
image: confluent/schema-registry:3.0.0
hostname: "schema-registry"
depends_on:
- kafka
- zookeeper
networks:
- test-net
ports:
- 8081:8081
environment:
SR_HOSTNAME: schema-registry
SR_LISTENERS: http://schema-registry:8081
SR_DEBUG: 'true'
SR_KAFKASTORE_TOPIC_REPLICATION_FACTOR: '1'
SR_KAFKASTORE_TOPIC_SERVERS: PLAINTEXT://kafka:9092
networks:
test-net:
driver: bridge`
I am running wurstmeister/kafka in a ubuntu 14.04 LTS machine. Kafka container is running fine. However I am not able to connect my microservice to this container.
Kafka docker-compose.yml:
version: '2'
services:
zookeeper:
image: user1/zookeeper:3.4.9
ports:
- "2181:2181"
kafka:
image: my-kafka
ports:
- "9092:9092"
hostname: kafka-01
environment:
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ADVERTISED_HOST_NAME: "kafka-01"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "topic1:1:1,topic2:1:1"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
docker-compose for microservice(myapp):
version: '2'
services:
myapp:
network_mode: 'bridge'
extends:
file: myapp.base.yml
service: myapp
links:
- kafka
zookeeper:
network_mode: 'bridge'
extends:
file: zookeeper.yml
service: zookeeper
kafka:
network_mode: 'bridge'
extends:
file: kafka.yml
service: kafka
links:
- zookeeper
I have linked kafka in an env:
kafkaHost=kafka-01:9092
Please let me know what am I doing wrong or if more info is required. Thanks in advance.
The hostname should be what you specified in "links", ie. "kafka". As #dnephin said the hostname is not needed. I think what you want is actually KAFKA_ADVERTISED_HOST_NAME environment variable.
What version of docker do you have? I am using docker 1.12.3 on Mac OS with no issues.
You might also want to try the Confluent docker images:
https://hub.docker.com/r/confluentinc/cp-kafka/
https://hub.docker.com/r/confluentinc/cp-zookeeper/
Here's my sample docker compose file:
version: "2"
services:
web:
image: nginx:latest
links:
- kafka
zookeeper:
extends:
file: kafka-services.yml
service: zookeeper
kafka:
extends:
file: kafka-services.yml
service: kafka
depends_on:
- zookeeper
My base kafka services compose file (kafka-services.yml):
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
ports:
- "32181:32181"
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:latest
ports:
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092
The hostname field does not set the public hostname, it sets the internal container hostname. It's generally not that useful.
Hostnames are automatically provided by docker networking, use kafka as the hostname.
When i was running the kafka and zookeeper without Docker, I could see the topic partitions log files in the /tmp/kafka-logs directory. Now with Docker, even though i specify the log directory in the Volumes section in docker-compose.yml, i cant see the files in the docker VM, like "TOPICNAME-PARTITIONNUMBER".. Is there anything I'm missing here ? Any idea on where i could find these directories in Docker VMs..
zookeeper:
image: confluent/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
- "15001:15000"
environment:
ZK_SERVER_ID: 1
volumes:
- /tmp/docker/zk1/logs:/logs
- /tmp/docker/zk1/data:/data
kafka1:
image: confluent/kafka
container_name: kafka1
ports:
- "9092:9092"
- "15002:15000"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_STORAGE: kafka
# This is Container IP
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
volumes:
- /tmp/docker/kafka1/logs:/logs
- /tmp/docker/kafka1/data:/data
this is how we configured for logs in our compose file and it has the log files in it. You should jump onto the container to see the '/var/lib/kafka/data', directory and the data inside it
volumes:
-kb1_data:/var/lib/kafka/data
Remember that 1st parameter in list for volumes, ports and other fields about sharing resources in docker-compose is about host, the 2nd is about the container.
So you should change the order for your volumes values.
zookeeper:
image: confluent/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
- "15001:15000"
environment:
ZK_SERVER_ID: 1
volumes:
#- ./host/folder:/container/folder
- ./logs:/tmp/docker/zk1/logs
- ./data:/tmp/docker/zk1/data
kafka1:
image: confluent/kafka
container_name: kafka1
ports:
#- "host-port:container-port"
- "9092:9092"
- "15002:15000"
links:
- zookeeper
environment:
- KAFKA_BROKER_ID: 1
- KAFKA_OFFSETS_STORAGE: kafka
- # This is Container IP
- KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
volumes:
#- ./host/folder:/container/folder
- ./logs:/tmp/docker/kafka1/logs
- ./data:/tmp/docker/kafka1/data