I'd like my web Docker container to access Redis on 127.0.0.1:6379 from within the web container. I've setup my Docker Compose file as the following. I get ECONNREFUSED though:
version: "3"
services:
web:
build: .
ports:
- 8080:8080
command: ["test"]
links:
- redis:127.0.0.1
redis:
image: redis:alpine
ports:
- 6379
Any ideas?
The short answer to this is "don't". Docker containers each get their own loopback interface, 127.0.0.1, that is separate from the host loopback and from that of other containers. You can't redefine 127.0.0.1, and if you could, that would almost certainly break other things.
There is a technically possible way to do it by either running all containers directly on the host, with:
network_mode: "host"
However, that removes the docker network isolation that you'll want with containers.
You can also attach one container to the network of another container (so they have the same loopback interface) with:
docker run --net container:$container_id ...
but I'm not sure if there's a syntax to do this in docker-compose and it's not available in swarm mode since containers may run on different nodes. The main use I've had for this syntax is attach network debugging tools like nicolaka/netshoot.
What you should do instead is make the location of the redis database a configuration parameter to your webapp container. Pass the location in as an environment variable, config file, or command line parameter. If the web app can't support this directly, update the configuration with an entrypoint script that runs before you start your web app. This would change your compose yml file to look like:
version: "3"
services:
web:
# you should include an image name
image: your_webapp_image_name
build: .
ports:
- 8080:8080
command: ["test"]
environment:
- REDIS_URL=redis:6379
# no need to link, it's deprecated, use dns and the network docker creates
#links:
# - redis:127.0.0.1
redis:
image: redis:alpine
# no need to publish the port if you don't need external access
#ports:
# - 6379
Related
Consider a part of some base docker-compose.yml file:
services:
foo:
image: bar
network_mode: host
.
.
.
Then, consider a docker-compose.prod.yml file which would override the base file's network mode and also set ports:
services:
foo:
ports:
- 'xxxx:yyyy'
network_mode: ?
I am looking for a value of ? such that the network_mode is considered unset. In other words, setting it to none or bridge doesn't seem to work, so I want it to just disappear, or use a value with such an effect (I don't think there is a default).
An alternative solution to this problem is to define three docker-compose files: docker-compose.yml, docker-compose.prod.yml, and docker-compose.dev.yml (or something equivalent, doesn't matter). It works fine (see below), but I would rather have 2 files only, and override the dev file with the prod file, rather than the other way around (it feels more natural this way).
A working version using three files:
docker-compose.yml
services:
foo:
image: bar
.
.
.
docker-compose.dev.yml
services:
foo:
network_mode: host
docker-compose.prod.yml
services:
foo:
ports:
- 'xxxx:yyyy'
Notes:
All files are using docker-compose version 3.
The specific setup which doesn't work with bridge network mode in my case is a collection of three services - one for running web stuff (exposed to public), one for celery workers (internal), and one for Redis (internal). Using bridge in web and/or celery results in being unable to connect to the Redis service.
I don't know how to remove a option from former docker-compose.yaml, but the bridge really works, you may want to have a double confirm.
If we have a look for docker default network, you could see bridge really there, if we do not set --net, default docker will use bridge:
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
32a9a31082ae bridge bridge local
dcb12f4cc711 host host local
16c73acab8c9 none null local
I give a small example to prove it:
docker-compose.yaml:
version: "3"
services:
web:
image: python:3.7
network_mode: host
command: python -m http.server 9000
docker-compose.prod.yaml:
version: "3"
services:
web:
ports:
- 10000:9000
network_mode: bridge
docker-compose up
With this command, we will in fact just use docker-compose.yaml, at that time, if we open browser we can see could successfully visit e.g.: http://$ip:9000, this is because the network is host.
docker-compose -f docker-compose.yaml -f docker-compose.prod.yaml up
With this command, we will use 2 compose files, the latter one will override the duplicate option in the former one.
If you open browser, you will find you can't visit http://$ip/9000 now, you could just visit http://$ip/10000. This can prove that host loss effect, and the bridge override host successfully.
(NOTE: you will have to assure docker-compose.prod.yaml after docker-compose.yaml, the sequence order is important.)
UPDATE 20211017 based on your new comments:
If you want to addtional visit other container in your compose, you will have to define network_mode in that target service to use the same network of source service with network_mode: service:web, see network mode:
docker-compose.yaml:
version: "3"
services:
web:
image: python:3.7-alpine3.13
command: python -m http.server 9000
network_mode: host
db:
image: python:3.7-alpine3.13
command: python -m http.server 8000
network_mode: service:web
docker-compose.yaml:
version: "3"
services:
web:
ports:
- 10000:9000
network_mode: bridge
Then, after up, you could use something like next to directly visit the service in db:
docker-compose exec web wget http://localhost:8000 -O -
NOTE:
with network_mode: bridge in web, the web container nolonger use the default network which compose set up for you. As a result, you won't benifit from the auto dns setup by compose, which means in web you won't be able to access db container using the service name db.
to conquer this issue, you now could let db container use the network of web explicitly with network_mode: service:web. This means the 2 containers now share the same network namespace, then you no need to visit db with service name, but to use localhost. In web, you now could access db's 8000 port just with http://localhost:8000.
TL;DR
docker-compose.prod.yml
services:
foo:
network_mode: unset # Can be any string other than 'host'.
networks: [ default ]
ports: [ 80:80 ]
Docker Compose Default Network
This is the default behavior: if there's no networks: and no network_mode: defined for the service then it's attached to the default network.
Unfortunately an empty string is ignored by the YAML parser, so we can't override network_mode: back to '', what we can do is set networks: [ default ], but that won't work with network_mode: host so what can we set network_mode: to?
network_mode: is equivalent to Docker --network, if we set it to a network name networks: will override it, so we can use any value other than host which apparently triggers some special behavior. It's probably a good idea to set it to something that will fail without networks: and avoid default, bridge or none.
To restore the default behavior we need to add default to networks: and set network_mode: to something other than host.
Note that network_mode: default is equivalent to --network default or --network bridge which uses the default Docker network: bridge, and not the default project-wide Compose network: projectName_default.
I have one application running on http://home.local:8180 in container A. And the other container B is running on http://data.local:9010. Container B is using container A to hit the API. If I specify container A hostname as http://host.docker.internal:8180 in container B then it works. What I would have to do if I want to use the hostname as is (home.local:8180)
Following is the docker-compose file:
home_app:
hostname: "home.local"
image: "home-app"
ports:
- "8180:8080"
environment:
data_app:
hostname: "data.local"
image: "data-app"
links:
- "home_app"
ports:
- "9010:9010"
Just use "home.local:8080". 8180 is just on the host machine and forwards to 8080 on the container, whereas based on your docker-compose, 8080 is the port of your application on home_app container, so within the docker-compose network, other containers should be able to access it via hostname (home.local) and the actual ports (8080).
You need to configure your application to use the Compose service name home_app as a host name, and the port number that the process inside the container is using. Neither hostname: nor ports: has any effect on connections between containers. You don't need to (and can't) specify a custom DNS suffix. See Networking in Compose in the Docker documentation for additional details.
So I might specify:
version: '3.8'
services:
home_app:
image: "home-app"
ports:
- "8180:8080" # optional, only for access from outside Docker
data_app:
image: "data-app"
ports:
- "9010:9010"
environment:
HOME_APP_URL: 'http://home_app:8080'
You don't need hostname:, which only affects what a container thinks its own hostname is and has no effect on anything outside the container; and you don't need links:, which is an obsolete option from first-generation Docker networking.
I have 2 docker images, one for my backend and one for a mock database. I want to spin up these two images separately and link the backend to the database. To do this I have a connection string in my backend like so Data Source=192.168.99.100;Catalog=DB name;Integrated Security=True;MultipleActiveResultSets=True"; where 192.168.99.100 is the IP of my default Docker machine where the database container is running. So on my Windows machine this works perfectly and the backend container can communicate with the database which is running on another container. However, when some of my colleagues who use Mac and Linux use the same images they can't get the link to work because they obviously don't have the same IP for their Docker machine.
Is there any way to reference the database in the connection string so that it is the same no matter where it is running? For example use the name of the database container, instead of the IP or something similar?
You can also do this using plain docker. Basically you just need to create a bridge network, and then attach both containers to it.
Eg:
docker network create --driver=bridge mynetwork
docker run --network=mynetwork --name mydb mydb:latest
docker run --network=mynetwork --name myapp myapp:latest
Then inside the myapp container you can reference the database container using the hostname mydb (same as with docker-compose). You can still expose ports in the myapp container to your host using -p 3000:3000, etc
Further reading: https://docs.docker.com/network/bridge/
You can use docker-compose services to achieve what you are looking for. Here is a simplified example docker-compose.yml file:
version: "3.5"
services:
db:
container_name: mock_db
restart: "no"
build: ./mock_db
expose:
- 5432 (or whatever your port is)
env_file: .env
command: your-command
server:
container_name: my_server
build: ./server
env_file: .env
ports:
- "8443:8443"
command: your-command
You can then reference the service name (in this case db) as the ip/url part of your connection string.
You can read more about docker-compose configuration options here
I am running a Java app inside a Docker container which is supposed to connect MySQL inside the other container. Trying multiple options suggested in the forms, nothing really works. Here is my Docker Compose file:
version: "3"
services:
app:
build:
context: ./
dockerfile: /src/main/docker/Dockerfile
image: app1
environment:
- DB_HOST=Imrans-MacBook-Pro.local
- DB_PORT=3306
ports:
- 8080:8080
networks:
- backend
depends_on:
- mysql
mysql:
image: mysql:5.7.20
hostname: mysql
environment:
- MYSQL_USER=root
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
- MYSQL_DATABASE=app1
ports:
- 3306:3306
command: mysqld --lower_case_table_names=1 --skip-ssl --character_set_server=utf8 --explicit_defaults_for_timestamp
networks:
- backend
networks:
backend:
driver: bridge
Where DB_HOST=Imrans-MacBook-Pro.local is my laptop's name. This did not work. Some suggest that the container name can be used so tried DB_HOST= mysql, never worked.
The only thing works from times to time when I pass the laptop's IP address, which is not I want to do. So, what is a good way to create communication between those containers?
The mysql is running in the container so there are two things that you should consider here:
If the mysql is running in the container then you will need to link the app container to the mysql container. This will allow them to talk to
each other using docker's inter container communication. The containers talk to each other using hostnames to resolve their respective internal IP addresses. See later in my answer I will show you how to get the two containers to communicate with each other using a compose file.
The mysql container should make use of a docker volume to store the database. This will allow you to store the database and related files on the file system of the host (server or machine where the containers are running on). The docker volume will then be mounted as a directory in the container. Thus the container can now read and write to a directory on the machine where the docker containers are running on. This means that even if the containers are all deleted or removed you will still have the database data persist. Here is a nice beginner friendly article on docker volumes and using them with MySQL:
https://severalnines.com/blog/mysql-docker-containers-understanding-basics
Container communication using only docker without compose:
You have container "app" and "mysql", you want to be able to access "app" on localhost and you want "app" to be able to connect to mysql. How are you gonna do this?
1. You need to expose a port for container "app" so we can access it on localhost. The docker containers have their own internal network and it is closed to you unless you expose some ports with docker.
You need to link the "mysql" container to "app" without exposing "mysql" 's ports to the rest of the world.
This config should work for what you want to achieve:
version: "2"
services:
app:
build:
context: ./
dockerfile: /src/main/docker/Dockerfile
image: app1:latest
links:
- mysql
environment:
- DB_HOST=mysql
# This is the hostname that app will reach the mysql container on.
# If you do with app container:
# docker exec -it <app container id> bash
# # apt-get update -y && apt-get install iputils-ping -y
#
# Then you should be able to ping mysql container with:
#
# # ping -c 2 mysql
- DB_PORT=3306
ports:
- 8080:8080
# You will access "app" on localhost:8080 in your browser. If this is running on your own machine.
mysql: #hostname actually gets set here so no need to set it later
image: mysql:5.7.20
environment:
- MYSQL_USER=root
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
- MYSQL_DATABASE=app1
# Remember to use a volume if you would like this container's data to persist or if you would like
# to restore a database backup.
command: mysqld --lower_case_table_names=1 --skip-ssl --character_set_server=utf8 --explicit_defaults_for_timestamp
Now you can just start it up with:
$ docker-compose up
If you ran this before then just make sure to run this first before running docker-compose up:
$ docker-compose down
Let me know if that helps.
I have, in the past, gotten this to work without explicitly setting the host networking part in Docker Compose. Because Docker images inside a Docker Compose File are put into a Docker Network with each other, you really shouldn't have to do anything to get this to work: by default you should be able to attach into the container for your Spring app and be able to ping mysql and have it work out.
DB host should be localhost or 127.0.0.1
This question already has answers here:
Communication between multiple docker-compose projects
(20 answers)
Closed 4 months ago.
I have a dockerized application with a few services running using docker-compose. I'd like to connect this application with ElasticSearch/Logstash/Kibana (ELK) using another docker-compose application, docker-elk. Both of them are running in the same docker machine in development. In production, that will probably not be the case.
How can I configure my application's docker-compose.yml to link to the ELK stack?
Update Jun 2016
The answer below is outdated starting with docker 1.10. See this other similar answer for the new solution.
https://stackoverflow.com/a/34476794/1556338
Old answer
Create a network:
$ docker network create --driver bridge my-net
Reference that network as an environment variable (${NETWORK})in the docker-compose.yml files. Eg:
pg:
image: postgres:9.4.4
container_name: pg
net: ${NETWORK}
ports:
- "5432"
myapp:
image: quay.io/myco/myapp
container_name: myapp
environment:
DATABASE_URL: "http://pg:5432"
net: ${NETWORK}
ports:
- "3000:3000"
Note that pg in http://pg:5432 will resolve to the ip address of the pg service (container). No need to hardcode ip addresses; An entry for pg is automatically added to the /etc/host of the myapp container.
Call docker-compose, passing it the network you created:
$ NETWORK=my-net docker-compose up -d -f docker-compose.yml -f other-compose.yml
I've created a bridge network above which only works within one node (host). Good for dev. If you need to get two nodes to talk to each other, you need to create an overlay network. Same principle though. You pass the network name to the docker-compose up command.
You could also create a network with docker outside your docker-compose :
docker network create my-shared-network
And in your docker-compose.yml :
version: '2'
services:
pg:
image: postgres:9.4.4
container_name: pg
expose:
- "5432"
networks:
default:
external:
name: my-shared-network
And in your second docker-compose.yml :
version: '2'
myapp:
image: quay.io/myco/myapp
container_name: myapp
environment:
DATABASE_URL: "http://pg:5432"
net: ${NETWORK}
expose:
- "3000"
networks:
default:
external:
name: my-shared-network
And both instances will be able to see each other, without open ports on host, you just need to expose ports, and there will see each other through the network : "my-shared-network".
If you set a predictable project name for the first composition you can use external_links to reference external containers by name from a different compose file.
In the next docker-compose release (1.6) you will be able to use user defined networks, and have both compositions join the same network.
Take a look at multi-host docker networking
Networking is a feature of Docker Engine that allows you to create
virtual networks and attach containers to them so you can create the
network topology that is right for your application. The networked
containers can even span multiple hosts, so you don’t have to worry
about what host your container lands on. They seamlessly communicate
with each other wherever they are – thus enabling true distributed
applications.
I didn't find any complete answer, so decided to explain it in a complete and simple way.
To connect two docker-compose you need a network and putting both docker-composes in that network,
you could create netwrok with docker network create name-of-network,
or you could simply put network declaration in networks option of docker-compose file and when you run docker-compose (docker-compose up) the network would be created automatically.
put the below lines in both docker-compose files
networks:
net-for-alpine:
name: test-db-net
Note: net-for-alpine is internal name of the network and it will be used inside of the docker-compose files and could be different,
test-db-net is external name of the network and must be same in two docker-compose files.
Assume we have docker-compose.db.yml and docker-compose.alpine.yml
docker-compose.apline.yml would be:
version: '3.8'
services:
alpine:
image: alpine:3.14
container_name: alpine
networks:
- net-for-alpine
# these two command keeps apline container running
stdin_open: true # docker run -i
tty: true # docker run -t
networks:
net-for-alpine:
name: test-db-net
docker-compose.db.yml would be:
version: '3.8'
services:
db:
image: postgres:13.4-alpine
container_name: psql
networks:
- net-for-db
networks:
net-for-db:
name: test-db-net
To test the network, go inside alpine container
docker exec -it alpine sh
then with following commands you can check the network
# if it returns 0 or see nothing as a result, network is established
nc -z psql (container name)
or
ping pgsql