I have created a docker compose file where my application wants to use kafka.
docker-compose.yaml is:
version: '3.7'
services:
api:
depends_on:
- kafka
restart: on-failure
build:
context: .
dockerfile: Dockerfile
ports:
- 8080:8080
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.7
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "mytopic:1:1"
192.168.1.7 is my ip that i got from ifconfig.
In my service i am giving broker as 192.168.1.7:9092.
When i do docker ps and exec to my kafka container. I am not able to access the 192.168.1.0
What am i doing wrong here though the strange thing is in my application logs i see that the topic is created.
When i try to create the topic:
You don't need IP addresses other than 127.0.0.1
192.168.1.7 seems like your host IP, not the docker IP, and yet you are not using network_mode: host, and so the network is not allowing you to connect to the broker.
I recommend finding existing, functional Docker Compose files such as ones in this answer
As posted above by #oneCricketeer you don't have to hardcode any of your host ip addresses.
You can connect to broker using "broker" name inside your api itself. And same can be set to advertise host name as well.
Related
I have 3 containers with my bot, server and db. after docker-compose up, server and db are working. telegram bot does get-request and takes this error:
Get "http://localhost:8080/user/": dial tcp 127.0.0.1:8080: connect: connection refused
docker-compose.yml
version: "3"
services:
db:
image: postgres
container_name: todo_postgres
restart: always
ports:
- "5432:5432"
environment:
# TODO: Change it to environment variables
POSTGRES_USER: user
POSTGRES_DB: somedb
POSTGRES_PASSWORD: pass
server:
depends_on:
- db
build: .
restart: always
ports:
- 8080:8080
environment:
DB_NAME: somedb
DB_USERNAME: user
DB_PASSWORD: pass
bot:
depends_on:
- server
build:
./src/telegram_bot
environment:
BOT_TOKEN: TOKEN
restart: always
links:
- server
When using compose, try using the containers hostname.. in the case your bot should try to connect to
server:8080
Compose will handle the name resolution to the IP you need
What you try is to access localhost within your container (service) bot.
Maybe this answer will help you to solve the problem. It sound similar to your problem.
But I want to provide you another solution to your problem:
In case it's not needed to access the containers form outside (from your host), one appraoch would be making use of the expose functionality and a docker network.
See docs.docker.com: network.
The expose functionality allows to access your other containers within your network
See docs.docker.com: expose
Expose ports without publishing them to the host machine - they’ll only be accessible to linked services. Only the internal port can be specified.
Example
What is this example doing?
A couple of steps that are not mandatory
Set a static ip within your docker container
These Steps are not needed and can be omitted. However, I like to do this, since you have now a better control over the network. You can access the containers by their hostname (which is the container name or service name) as well.
The steps that are needed are the following:
This exposes port 8080, but do not publish it.
expose:
- 8080
The network which allows static ip configuration
networks:
vpcbr:
driver: bridge
ipam:
config:
- subnet: 10.5.0.0/16
A complete file could look similar to this:
version: "3.8"
services:
first-service:
image: <your-image>
networks:
vpcbr:
ipv4_address: 10.5.0.2
expose:
- 8080
second-service:
image: <your-image>
networks:
vpcbr:
ipv4_address: 10.5.0.3
depends_on:
- first-service
networks:
vpcbr:
driver: bridge
ipam:
config:
- subnet: 10.5.0.0/16
Your bot container is up before your server & db containers.
When you use depends_on it's not accually waiting them to finish setup themeselves.
You should try some tricky algorithem for waiting the other container finish setup.
I remmember that when I used Nginx proxy I used something called wait-for-it.sh
Current compose yaml:
version: '3.7'
networks:
app-tier:
driver: bridge
services:
php:
container_name: docker_php
build: .docker/php73
volumes:
- .:/srv/
networks:
- app-tier
rabbitmq:
container_name: docker_rabbitmq
image: "rabbitmq:3-management"
hostname: "rabbitmq-localhost"
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
ports:
- "15672:15672"
- "5672:5672"
networks:
- app-tier
My target is to reach docker_rabbitmq container from docker_php within localhost:
#bash php_container
telnet loaclhost 15672
How can I configure a network that:
container A has port mapping on localhost to Container B?
you're limited by the inner port, which means if your two containers are in the same docker defined network, you can use the internally opened ports of the respective container. For the hostname to be defined for a container in a different one, you can use the links attribute in the service definition inside your docker-compose.yml.
Consider a micro service which you want to be only accessed by only the containers on that network therefore exposing the ports on the host wouldn't make sense. Now assuming rabbitmq is the service that you want to access from php service, you need to define a link to rabbitmq in your php service definition( please not the link/host-definition is not bi-directional, if you need php in your rabbitmq you need to define a link in rabbitmq for php)
version: '3.7'
networks:
app-tier:
driver: bridge
services:
php:
container_name: docker_php
build: .docker/php73
volumes:
- .:/srv/
networks:
- app-tier
links:
- rabbitmq
rabbitmq:
container_name: docker_rabbitmq
image: "rabbitmq:3-management"
hostname: "rabbitmq-localhost"
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
networks:
- app-tier
Now you can access the internal ports of the rabbitmq from php but note the expternal ports are not accessible, those are for the host.
# inside your `php` container `bash`
telnet rabbitmq <internal_port>
Also not that I got rid of the ports in rabbitmq by removing
now these ports of rabbitmq are not accessible from the host.
Update
if you want to access the ports, such that the ports opened in rabbitmq are accessible in php on localhost. the easiest and the simplest way would be to configure rabbitmq to run in container network mode on the network of php to do this simple add
network_mode: "container:[container name/id]"
rabbitmq:
container_name: docker_rabbitmq
image: "rabbitmq:3-management"
hostname: "rabbitmq-localhost"
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
network_mode: "container:php"
ports:
- "15672:15672"
- "5672:5672"
I am trying to access a docker container from another container using localhost address.
The compose file is pretty simple. Both containers ports are exposed.
There are no problems when building.
In my host machine I can successfully execute curl http://localhost:8124/ and get a response.
But inside the django_container when trying the same command I get Connection refused error.
I tried adding them in the same network, still result didn't change.
Well if I try to execute with the internal ip of that container like curl 'http://172.27.0.2:8123/' I get the response.
Is this the default behavior? How can I reach clickhouse_container using localhost?
version: '3'
services:
django:
container_name: django_container
build: ./django
ports:
- "8007:8000"
links:
- clickhouse:clickhouse
volumes:
- ./django:/usr/src/run
command: bash /usr/src/run/run.sh
clickhouse:
container_name: clickhouse_container
build: ./clickhouse
ports:
- "9001:9000"
- "8124:8123"
- "9010:9009"
So with this line here - "8124:8123" you're mapping the port of clickhouse container to localhost 8124. Which allows you to access clickhouse from localhost at port 8124.
If you want to hit clickhouse container from within the dockerhost network you have to use the hostname for the container. This is what I like to do:
version: '3'
services:
django:
hostname: djano
container_name: django
build: ./django
ports:
- "8007:8000"
links:
- clickhouse:clickhouse
volumes:
- ./django:/usr/src/run
command: bash /usr/src/run/run.sh
clickhouse:
hostname: clickhouse
container_name: clickhouse
build: ./clickhouse
ports:
- "9001:9000"
- "8124:8123"
- "9010:9009"
If you make the changes like I have made above you should be able to access clickhouse from within the django container like this curl http://clickhouse:8123.
As in #Billy Ferguson's answer, you can visit using localhost in host machine just because: you define a port mapping to route localhost:8124 to clickhouse:8123.
But when from other container(django), you can't. But if you insist, there is a ugly workaround: share host's network namespace with network_mode, but with this the django container will just share all network of host.
services:
django:
hostname: djano
container_name: django
build: ./django
ports:
- "8007:8000"
links:
- clickhouse:clickhouse
volumes:
- ./django:/usr/src/run
command: bash /usr/src/run/run.sh
network_mode: "host"
It depends of config.xml settings. If in config.xml <listen_host> 0.0.0.0</listen_host> you can use clickhouse-client -h your_ip --port 9001
I use kafka in docker container.
One of the requirements is that the kafka will be available to a producer natively running on the host machine.
This is why I set the KAFKA_ADVERTISED_HOST_NAME to my host ip.
my docker-compose.yml looks like this:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.10
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERES: PLAINTEXT://localhost:9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
and it works.
The problem is, I want to be able to use this docker-compose file also on other machines, and I don't know what their IP might be.
trying to change the ip into a name like 'kafka' caused it to be unavailable to the host machine (although still available from other containers).
Is there a way to use the host IP in the docker-compose file without "hardcoding" it (so that it will be a different IP address on different machines)?
Is there another way of addressing this issue?
did you try using the property? It not recommend for production usage for whatever reason but it might work for you
host.docker.internal
https://docs.docker.com/docker-for-mac/networking/#there-is-no-docker0-bridge-on-macos#i-want-to-connect-from-a-container-to-a-service-on-the-host
https://docs.docker.com/docker-for-windows/networking/#there-is-no-docker0-bridge-on-windows#i-want-to-connect-from-a-container-to-a-service-on-the-host
I have a docker-compose file with several service-container definitions. One of the services communicates with Apache Kafka within the same docker-compose run.
So I have the kafka docker definition like this:
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9092
I have my service definition in the same docker-compose file. In the startup script of the service I have to figure out somehow the IP address of the Kafka instance.
I know, I can use something like docker inspect to find out which IP address is used by a container.
But how can I do it dynamically in a docker-compose environment?
EDIT
So, the right configuration should be (thank you, #nwinkler):
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: kafka
ADVERTISED_PORT: 9092
myservice:
image: foo
links:
- kafka:kafka
Don't forget to set the ADVERTISED_HOST to kafka (or how you named your kafka container within docker-compose).
You can use the Docker Compose Links feature for this. If you provide a link to the kafka container from your other container, Docker Compose will ensure that your other container can access the Kafka container through its hostname - you will not have to know its IP address.
Example:
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9092
myservice:
image: foo
links:
- kafka:kafka
This will allow your myservice container to access the Kafka container through the kafka hostname. So from your myservice container, you can do something like curl http://kafka:9092 to access the service on the Kafka container.
Docker-Compose does this through DNS, it creates a hostname/IP mapping in your container allowing you to access the container without knowing its IP address.
The ip of your container will be the ip you are looking for.
Append the port number (9092 in your case) to the ip of the container to get whatever kafka is serving.