I have two different Docker containers and each has a different image. Each app in the containers uses non-conflicting ports. See the docker-compose.yml:
version: "2"
services:
service_a:
container_name: service_a.dev
image: service_a.dev
ports:
- "6473:6473"
- "6474:6474"
- "1812:1812"
depends_on:
- postgres
volumes:
- ../configs/service_a/var/conf:/opt/services/service_a/var/conf
postgres:
container_name: postgres.dev
hostname: postgres.dev
image: postgres:9.6
ports:
- "5432:5432"
volumes:
- ../configs/postgres/scripts:/docker-entrypoint-initdb.d/
I can cURL each image successfully from the host machine (Mac OS), e.g. curl -k https://localhost:6473/service_a/api/version works. What I'd like to do is to be able to refer to postgres container from the service_a container via localhost as if these two containers were one and they share the same localhost. I know that it's possible if I use the hostname postgres.dev from inside the service_a container, but I'd like to be able to use localhost. Is this possible? Please note that I am not very well versed in networking or Docker.
Mac version: 10.12.4
Docker version: Docker version 17.03.0-ce, build 60ccb22
I have done quite some prior research, but couldn't find a solution.
Relevant: https://forums.docker.com/t/localhost-and-docker-compose-networking-issue/23100/2
The right way: don't use localhost. Instead use docker's built in DNS networking and reference the containers by their service name. You shouldn't even be setting the container name since that breaks scaling.
The bad way: if you don't want to use the docker networking feature, then you can switch to host networking, but that turns off a very key feature and other docker capabilities like the option to connect containers together in their own isolated networks will no longer work. With that disclaimer, the result would look like:
version: "2"
services:
service_a:
container_name: service_a.dev
image: service_a.dev
network_mode: "host"
depends_on:
- postgres
volumes:
- ../configs/service_a/var/conf:/opt/services/service_a/var/conf
postgres:
container_name: postgres.dev
image: postgres:9.6
network_mode: "host"
volumes:
- ../configs/postgres/scripts:/docker-entrypoint-initdb.d/
Note that I removed port publishing from the container to the host, since you're no longer in a container network. And I removed the hostname setting since you shouldn't change the hostname of the host itself from a docker container.
The linked forum posts you reference show how when this is a VM, the host cannot communicate with the containers as localhost. This is an expected limitation, but the containers themselves will be able to talk to each other as localhost. If you use a VirtualBox based install with docker-toolbox, you should be able to talk to the containers by the virtualbox IP.
The really wrong way: abuse the container network mode. The mode is available for debugging container networking issues and specialized use cases and really shouldn't be used to avoid reconfiguring an application to use DNS. And when you stop the database, you'll break your other container since it will lose its network namespace.
For this, you'll likely need to run two separate docker-compose.yml files because docker-compose will check for the existence of the network before taking any action. Start with the postgres container:
version: "2"
services:
postgres:
container_name: postgres.dev
image: postgres:9.6
ports:
- "5432:5432"
volumes:
- ../configs/postgres/scripts:/docker-entrypoint-initdb.d/
Then you can make a second service in that same network namespace:
version: "2"
services:
service_a:
container_name: service_a.dev
image: service_a.dev
network_mode: "container:postgres.dev"
ports:
- "6473:6473"
- "6474:6474"
- "1812:1812"
volumes:
- ../configs/service_a/var/conf:/opt/services/service_a/var/conf
Specifically for Mac and during local testing, I managed to get the multiple containers working using docker.for.mac.localhost approach. I documented it http://nileshgule.blogspot.sg/2017/12/docker-tip-workaround-for-accessing.html
Related
My docker-compose.yml contains this:
version: '3.2'
services:
mysql:
image: mysql:latest
container_name: mysql
restart: always
network_mode: "host"
hostname: localhost
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
volumes:
- $HOME/data/datasql:/var/lib/mysql
ports:
- 3306:3306
user-management-service:
build: user-management-service/
container_name: user-management-service
restart: always
depends_on:
- mysql
- rabbitmq
- eureka
network_mode: "host"
hostname: localhost
ports:
- 8089:8089
When I try to do docker-compose up, I get the following error:
"host" network_mode is incompatible with port_bindings
Can anyone help me with the solution?
network_mode: host is almost never necessary. For straightforward servers, like the MySQL server you show or what looks like a normal HTTP application, it's enough to use normal (bridged) Docker networking and ports:, like you show.
If you do set up host networking, it completely disables Docker's networking stack. You can't call to other containers using their host name, and you can't remap a container's port using ports: (or choose to not publish it at all).
You should delete the network_mode: lines you show in your docker-compose.yml file. The container_name: and hostname: lines are also unnecessary, and you can delete those too (specific exception: RabbitMQ needs a fixed hostname:).
I feel like the two places I see host networking are endorsed are either to call back to the host machine (see From inside of a Docker container, how do I connect to the localhost of the machine?), or because the application code has hard-coded localhost as the host name of the database or other components (in which case Docker and a non-Docker development setup fundamentally act differently, and you should configure these locations using environment variable or another mechanism).
Quick solution:
Downgrade the docker-compose version and you'll be fine. The issue is with the latest docker-compose version and network_mode: "host"
I faced the same issue on v1.29.2 and while everything worked smooth on v1.27.4.
I had the same problem with network_mode: 'host'.
When downgrading docker-compose from 1.29.2 to 1.25.4, it worked fine. Maybe some bug added in new versions?
Get rid of the param ports in your services containing network_mode its like doing mapping twice.
mysql:
image: mysql:latest
container_name: mysql
restart: always
network_mode: "host"
hostname: localhost
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
volumes:
- $HOME/data/datasql:/var/lib/mysql
....
....
To access the host's http://localhost inside your docker, you need to replace:
network_mode: host
with:
ports:
- 80:80
You can do the same with any other port.
If you want to connect to a local database then, when connecting to that database, don't use "localhost" or "127.0.0.1". Instead use "host.docker.internal" and that will allow traffic between your container to the database.
I am trying to understand how I access containers between each other through their container name. Specifically when using a pgadmin container and connecting to a postgresql container through dns.
In docker-compose V3 , I cannot link them, nor does networks: seem to be available either.
The main reason to need this is when the containers spin up they don't have a static IP address, so in pgadmin I can't connect to the postgresql DB using the same IP every time , so a dns name would work better (ie: the container name).
Can we do this with docker-compose or at least set a static ip address for a specific container?
I have tried creating a user defined network:
networks:
backed:
and then using it in the service:
app:
networks:
- backend
This causes a docker-compose error regarding an invalid option of "networks" in the app.
docker-compose.yml
version: "0.1"
services:
devapi:
container_name: devapi
restart: always
build: .
ports:
- "3000:3000"
api-postgres-pgadmin:
container_name: api-postgres-pgadmin
image: dpage/pgadmin4:latest
ports:
- "5050:80"
environment:
- PGADMIN_DEFAULT_EMAIL=stuff#stuff.com
- PGADMIN_DEFAULT_PASSWORD=12345
api-postgres:
container_name: api-postgres
image: postgres:10
volumes:
- ./data:/data/db
ports:
- "15432:5432"
environment:
- POSTGRES_PASSWORD=12345
Actually, I spot one immediate problem:
version: "0.1"
Why are you doing this? The current version of the compose file format is 3.x. E.g:
version: "3"
See e.g. the Compose file version 3 reference.
The version determines which feature are available. It's entirely possible that by setting version: "0.1" you are explicitly disabling support for the networks parameter. You'll note that the reference shows examples using the networks attribute.
As an aside, unless there is a particular reason you ened it, I would drop the use of the container_name in your compose file, since this makes it impossible to run multiple instances of the same compose file on your host.
networks are available from docker-compose version 3 but you are using version:"0.1" in your docker-compose file.
Change the version: "0.1" to version: "3" in docker-compose.yml.
My app has 2 dependencies which I specify in my docker-compose, a postgres and kafka service:
services:
postgres:
image: postgres:9.6-alpine
ports:
- "5432:5432"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
I run my code and tests outside the docker network, and use these two containers as my dependencies.
As these both expose ports, I can configure my app to hit them via: localhost:5432, localhost:9092. This works.
The problem I have is when I want to test the app image itself, I add this as a service to the docker-compose file:
app:
image: myapp
links:
- postgres
- kafka
The app is still configured to use localhost, so I allow the app container to access my network using --net=host
Whilst the app container can now access localhost:5432 and localhost:9092 (confirmed by curling from inside the container), the host names fail to resolve when the code runs and the dependencies are unreachable - possibly as a result of using localhost from inside the container and confusing the client libraries? I'm really not sure.
It feels like the use of localhost in the app configuration isn't the right approach here. Is it possible to refer to the service names 'postgres' and 'kafka' from outside the docker network?
Why do you want to continue using localhost:xxx in your app?
The best approach for you is to change connection strings in your application when it is being launched from docker-compose. You just use postgres:5432 and kafka:9092 and everything will work, because inside docker-compose network all machines are visible to each other under their service names.
If for some great reasons you insist on using localhost as a connection target, you need to turn all services into host mode. But remember - in this case ports are not exposed, so you access services with their original port values.
version: '3'
services:
postgres:
image: postgres:9.6-alpine
network_mode: "host"
kafka:
image: wurstmeister/kafka
network_mode: "host"
app:
image: myapp
network_mode: "host"
And by the way, forget about links. They are deprecated.
I'm fairly new to docker and docker compose, so forgive me if this is a stupid question...
I have a compose file with 2 containers. A homeassistant container with port 8123 exposed and a database with 5432. The homeassistant can access the database using the url postgresql://user:password#db:5432/homeassistant_db. I think that this is because docker has created a db binding on the host and that's why I can connect to db.
However I need to bind the homeassistant to the host, which I can do with network_mode: "host" which you can see commented out in my config. When I do this I can indeed bind to the host and homeassistant can do it's discovery of network devices etc...
Unfortunately this breaks the connection with the database so that I can't use the postgresql://user:password#db:5432/homeassistant_db url any longer.
How do I attach homeassistant to the host AND keep the database connection working? I guess I could change the database host from db to the pi's url or network name (eg. postgresql://user:password#192.168.0.100:5432/homeassistant_db or postgresql://user:password#homeassistant.local:5432/homeassistant_db) but this doesn't feel as clean or as robust as it could be.
I don't really understand the network bindings so I wan to try and learn so I can fix this myself going forward.
compose file below:
version: '3'
services:
db:
restart: always
container_name: "homeassistant_db_container"
# image: postgres:latest
image: tobi312/rpi-postgresql
ports:
- "5432:5432"
volumes:
- ./data/postgres/data:/var/lib/postgresql/data/pgdata
env_file:
- ./envs/database.env
home_assistant:
container_name: "homeassistant_container"
restart: always
image: homeassistant/raspberrypi3-homeassistant
ports:
- "8123:8123"
# network_mode: "host"
env_file:
- ./envs/homeassistant.env
volumes:
- ./configs/homeassistant:/config
depends_on:
- db
volumes:
data:
driver_opts:
type: none
o: bind
device: "${PWD}/data/postgres"
You can add both the containers in the same network as shown below. Then you could use the way you want to. Just add below code to your compose file. Then it will create a network and add both these containers there. This will also give you a security layer, so that no other containers can talk to your db container.
Second, remove container_name. You are confusing yourself. Services get their host names equal to service names by default.
networks:
default:
external:
name: "tools"
I'm running a mongo instance with docker-compose and traefik.
myapp-mongo:
build: ../images/myapp-mongo
restart: always
ports:
- "27017:27017"
labels:
- "traefik.ports=27017,27018"
- "traefik.backend=myapp-mongo"
- "traefik.frontend.rule=Host:myapp-mongo.docker.localhost"
networks:
- development
environment:
- MONGO_USER=${MONGO_USER}
- MONGO_PASSWD=${MONGO_PASSWD}
- MONGO_AUTHDB=${MONGO_AUTHDB}
Mongo is running fine and I can connect using 127.0.0.1 from my Mac.
The problem is that I can't connect using hostname myapp-mongo.docker.localhost. It only works using IP 127.0.0.1.
Trying to ping the IP 127.0.0.1 responds ok, but trying to ping the hostname doesn't work.
I've already added 127.0.0.1 proxy.docker.localhost into /etc/hosts to get traefik working.
All other web apps has hostnames working fine like eg myapp.docker.localhost. This problem is only happening with this mongodb container.
Probably because Træfik is HTTP proxy and so will only support HTTP/HTTPS connections.
I believe #bpatel is right (see comment I left on his answer with link to github conversation) Traefik at the time of writing only supports HTTP/HTTPS.
Solution using native docker networks
However, you can get around this issue! Since you are using docker, you can work around by using the container name in your code (assuming mongo and your mongo accessing code are both running in containers on a shared docker network. This will be the case if the containers are spun up with docker-compose). Run the following to see if your containers are linked up correctly:
run docker ps to get your container names running (under the NAMES column)
run docker network ls to see your network names
run docker network inspect <target_network_name> to verify your containers from step 1 are on the same network.
I run docker-compose from three separate compose files, so you should be able to cover most cases from the following (apologies for any syntax errors, the following are stripped down code examples):
Entire docker-compose file that that starts up traefik (under directory name 'proxy')
version: '2'
services:
traefik:
image: traefik
command: --web --docker --docker.domain=docker.localhost --logLevel=DEBUG
networks:
- webgateway
ports:
- "80:80"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /dev/null:/traefik.toml
networks:
webgateway:
driver: bridge
snippet from my docker-compose file that spins up mongo
version: '2'
services:
database:
image: mongo
ports:
- "27017:27017"
networks:
- web
networks:
web:
external:
name: proxy_webgateway
snippet from docker-compose that has mongo accessing code
version: '2'
services:
topicOntologyBuilder:
image: topic-ontology-builder
labels:
- "traefik.backend=topicOntologyBuilder"
- "traefik.port=80"
- "traefik.frontend.rule=Host:topic-ontology.docker.localhost"
networks:
- web
volumes:
- ./:/home
networks:
web:
external:
name: proxy_webgateway
Connection in Code
Not certain what language you're using, this is what the following js code looked like for me to connect to mongo (inside that 'topicOntologyBuilder` container, while using traefik as the proxy (again, this works because we're making the most of docker networks):
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect('mongodb://<MONGO_CONTAINER_NAME>/<DB_NAME>', function(err, db) {
//insert code here to interact with mongo
})
Why this works
This works because docker does some clever DNS stuff within the containers so that each container knows the IP of other containers, by looking it up in their DNS entry, by the container names
Extra intel
If your containers are on separate computers/vm's, you'll probably want to play around with a service discovery tool (Consul plays well with Traefik) or do something fancy with a docker network overlay which is specific for containers in a cluster.
If using raw docker networks, you can assign container aliases (this doesn't work with Traefik though, or at least it didn't a couple months back).