I have a docker-compose file with several service-container definitions. One of the services communicates with Apache Kafka within the same docker-compose run.
So I have the kafka docker definition like this:
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9092
I have my service definition in the same docker-compose file. In the startup script of the service I have to figure out somehow the IP address of the Kafka instance.
I know, I can use something like docker inspect to find out which IP address is used by a container.
But how can I do it dynamically in a docker-compose environment?
EDIT
So, the right configuration should be (thank you, #nwinkler):
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: kafka
ADVERTISED_PORT: 9092
myservice:
image: foo
links:
- kafka:kafka
Don't forget to set the ADVERTISED_HOST to kafka (or how you named your kafka container within docker-compose).
You can use the Docker Compose Links feature for this. If you provide a link to the kafka container from your other container, Docker Compose will ensure that your other container can access the Kafka container through its hostname - you will not have to know its IP address.
Example:
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9092
myservice:
image: foo
links:
- kafka:kafka
This will allow your myservice container to access the Kafka container through the kafka hostname. So from your myservice container, you can do something like curl http://kafka:9092 to access the service on the Kafka container.
Docker-Compose does this through DNS, it creates a hostname/IP mapping in your container allowing you to access the container without knowing its IP address.
The ip of your container will be the ip you are looking for.
Append the port number (9092 in your case) to the ip of the container to get whatever kafka is serving.
Related
I'im fairly new to docker and docker compose.
I have a simple scenario, based on three applications (app1, app2, app3) that I want to connect to my host's network. The purpose is having an internet connection also inside the container.
Here is my docker-compose file:
version: '3.9'
services:
app1container:
image: app1img
build: ./app1
networks:
network_comp:
ipv4_address: 192.168.1.1
extra_hosts:
anotherpc: 192.168.1.44
ports:
- 80:80
- 8080:8080
app2container:
depends_on:
- "app1container"
image: app2img
build: ./app2
networks:
network_comp:
ipv4_address: 192.168.1.2
ports:
- 3100:3100
app3container:
depends_on:
- "app1container"
image: app3img
build: ./app3
networks:
network_comp:
ipv4_address: 192.168.1.3
ports:
- 9080:9080
networks:
network_comp:
driver: ""
ipam:
driver: ""
config:
- subnet: 192.168.0.0/24
gateway: 192.168.1.254
I already read the docker-compose documentation, which says that there is no a bridge driver for Windows OS. Is there anyway a solution to this issue?
You shouldn't usually need to do any special setup for this to work. When your Compose service has ports:, that makes a port available on the host's IP address. The essential rules for this are:
The service inside the container must listen on the special 0.0.0.0 "all interfaces" address (not 127.0.0.1 "this container only"), on some (usually fixed) port.
The container must be started with Compose ports: (or docker run -p). You choose the first port number, the second port number must match the port inside the container.
The service can be reached via the host's IP address on the first port number (or, if you're using the older Docker Toolbox setup, on the docker-machine ip address).
http://host.example.com:12345 (from other hosts)
|
v
ports: ['12345:8080'] (in the `docker-compose.yml`)
|
v
./my_server -bind 0.0.0.0:8080 (the main container command)
You can remove all of the manual networks: configuration in this file. In particular, it's problematic if you try to specify the Docker network to have the same IP address range as the host network, since these are two separate networks. Compose automatically provides a network named default that should work for most practical applications.
I have created a docker compose file where my application wants to use kafka.
docker-compose.yaml is:
version: '3.7'
services:
api:
depends_on:
- kafka
restart: on-failure
build:
context: .
dockerfile: Dockerfile
ports:
- 8080:8080
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.7
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "mytopic:1:1"
192.168.1.7 is my ip that i got from ifconfig.
In my service i am giving broker as 192.168.1.7:9092.
When i do docker ps and exec to my kafka container. I am not able to access the 192.168.1.0
What am i doing wrong here though the strange thing is in my application logs i see that the topic is created.
When i try to create the topic:
You don't need IP addresses other than 127.0.0.1
192.168.1.7 seems like your host IP, not the docker IP, and yet you are not using network_mode: host, and so the network is not allowing you to connect to the broker.
I recommend finding existing, functional Docker Compose files such as ones in this answer
As posted above by #oneCricketeer you don't have to hardcode any of your host ip addresses.
You can connect to broker using "broker" name inside your api itself. And same can be set to advertise host name as well.
i created a new docker-stack where i would need several influxdb instances, which i can’t connect to my grafana container atm. Here is a port of my docker-compose.yml
services:
grafana:
image: grafana/grafana
container_name: grafana
restart: always
ports:
- 3000:3000
networks:
- monitoring
volumes:
- grafana-volume:/var/lib/grafana
influxdb:
image: influxdb
container_name: influxdb
restart: always
ports:
- 8086:8086
networks:
- monitoring
volumes:
- influxdb-volume:/var/lib/influxdb
influxdb-2:
image: influxdb
container_name: influxdb-2
restart: always
ports:
- 12380:12380
networks:
- monitoring
volumes:
- influxdb-volume-2:/var/lib/influxdb
When i try to create a new influxdb datasource in grafana with influxdb-2 i get a Network Error: Bad Gateway(502), the logfile is showing:
2782ca98a4d7_grafana | 2019/10/05 13:18:50 http: proxy error: dial tcp 172.20.0.4:12380: connect: connection refused
Any ideas?
Thanks
#hmm provides the answer.
When you create services within Docker Compose, you:
are able to access containers by the service name. Grafana will reference influxdb-2 by that name.
are not able to change the ports a container exposes. Per #hmm, influxdb-2 must still be referenced on port 8086 because that's the port the container exposes; you can't change it unless you change the image.
you may (but you don't need to) expose the containers' ports to the host (using --ports: [[HOST-PORT]]:[[CONTAINER-PORT]]
Long and the short of it is that the InfluxDB service in influxdb-2 should be referenced as influxdb-2:8086. If you want to expose this service to the host (!), you could do ports: - 12380:8086. You may change the value of 12380 to something available on your host but you cannot change the value of the container port (8086).
The main reason that you would include the --ports: flag on influxdb-2 is for debugging from the host. But the grafana service does not require this. It will access the influxdb-2 service through the network provisioned by Docker Compose on port 8086.
You do want to expose the grafana service on the host because, otherwise, it would be inaccessible to you (from the host). It's akin to public|private. grafana is host public but the influxdb* services may be host private because they are generally only needed by the grafana service.
HTH!
I am hosting 3 services using docker-compose.
version: '3.3'
services:
service-a:
container_name: service-a
network_mode: default
ports:
- 8001:8001
- 8080:8080
service-b:
container_name: service-b
network_mode: default
ports:
- 8180:8080
links:
- service-a:srv_a
service-api:
container_name: service-api
environment:
- SERVER_URL=http://localhost:8180/myserver
- 8001:8001
links:
- service-b: srv_b
However the service-api which is a spring boot application can't access the
service-b despite the link.
I can do that when using the browser.
What can I do to investigate the reasons for the lack of connectivity?
Should the link be somehow used in the server_url variable?
Each Docker container has it's own IP address. From the service-api container perspective, localhost resolve to its own IP address.
Docker-compose provides your containers with the ability to resolve other containers IP addresses from the docker compose service names.
Try:
service-api:
environment:
- SERVER_URL=http://service-b:8080/myserver
note that you need to connect to the container internal port (8080) and not the matching port published on the docker host (8180).
I use kafka in docker container.
One of the requirements is that the kafka will be available to a producer natively running on the host machine.
This is why I set the KAFKA_ADVERTISED_HOST_NAME to my host ip.
my docker-compose.yml looks like this:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.10
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERES: PLAINTEXT://localhost:9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
and it works.
The problem is, I want to be able to use this docker-compose file also on other machines, and I don't know what their IP might be.
trying to change the ip into a name like 'kafka' caused it to be unavailable to the host machine (although still available from other containers).
Is there a way to use the host IP in the docker-compose file without "hardcoding" it (so that it will be a different IP address on different machines)?
Is there another way of addressing this issue?
did you try using the property? It not recommend for production usage for whatever reason but it might work for you
host.docker.internal
https://docs.docker.com/docker-for-mac/networking/#there-is-no-docker0-bridge-on-macos#i-want-to-connect-from-a-container-to-a-service-on-the-host
https://docs.docker.com/docker-for-windows/networking/#there-is-no-docker0-bridge-on-windows#i-want-to-connect-from-a-container-to-a-service-on-the-host