Currently I have a rabbitmq message broker and multiple celery workers that need to be containerized. My problem is, how can I fire up containers using different docker-compose.yml? My goal is to start the rabbitmq once and for all, and never touch it again.
Currently I have a docker-compose.yml for the rabbitmq:
version: '2'
services:
rabbit:
hostname: rabbit
image: rabbitmq:latest
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
ports:
- "5672:5672"
expose:
- "5672"
And another docker-compose.yml for celery workers:
version: '2'
services:
worker:
build:
context: .
dockerfile: dockerfile
volumes:
- .:/app
environment:
- CELERY_BROKER_URL=amqp://admin:mypass#rabbit:5672
links:
- rabbit
However, when I do docker-compose up for celery workers, I keep getting the following error:
ERROR/MainProcess] consumer: Cannot connect to
amqp://admin:**#rabbit:5672//: failed to resolve broker hostname.
Can anyone take a look if there is anything wrong with my code? Thanks.
the domain name rabbit in your second docker-compose.yml file does not resolve because there is no service with that name in that docker-compose.yml file.
As stated in the comments, one solution is to put both the rabbit service and worker service in the same docker-compose.yml file. In such a setup, all containers started for those services would join the same docker network and those service names could be resolved to the IP adresses of their containers.
Since having a single docker-compose.yml file is not convenient in your case, you have to find an other way to have the containers originating from different docker-compose.yml files join a same docker network.
To do so, you need to create a dedicated docker network for that purpose:
docker network create rabbitNetwork
Then, in each docker-compose.yml file, you need to refer to this network in the services definitions:
version: '2'
services:
rabbit:
hostname: rabbit
image: rabbitmq:latest
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
# ports:
# - "5672:5672" # there is no need to publish ports on the docker host anymore
expose:
- "5672"
networks:
- rabbitNet
networks:
rabbitNet:
external:
name: rabbitNetwork
version: '2'
services:
worker:
build:
context: .
dockerfile: dockerfile
volumes:
- .:/app
environment:
- CELERY_BROKER_URL=amqp://admin:mypass#rabbit:5672
networks:
- rabbitNet
networks:
rabbitNet:
external:
name: rabbitNetwork
You can use any file as service definition.
docker-compose.yml is default file name but any other name can be passed using -f argument.
docker-compose -f rabbit-compose.yml COMMAND
Related
I have two conteiners:
docker-compose.yml
version: '3.8'
services:
db:
image: postgres:14.1
container_name: postgres
volumes:
- postgres_data:/var/lib/postgresql/data/
......
network_mode: bridge
web:
container_name: web
build: .
........
network_mode: bridge
external_links:
- postgres
depends_on:
- db
volumes:
postgres_data:
name: postgres_data
After docker-compose up, when i recreate only one container - "db", all works, but i can not connect to conteiner "web", i get error: "Failure
Cannot link to a non running container: /postgres AS /web/postgres".
In conteiner "web" i call db as host=postgres.
What am I doing wrong?
The external_links: setting is obsolete and you don't need it. You can just remove it with no adverse consequences.
network_mode: bridge and container_name: are also unnecessary, though they shouldn't specifically cause problems; still, I'd delete them. What you show can be reduced to
version: '3.8'
services:
db:
image: postgres:14.1
volumes:
- postgres_data:/var/lib/postgresql/data/
......
web:
build: .
........
depends_on:
- db
volumes:
postgres_data: # empty
Since Compose creates a network named default for you and attaches containers to it, your application container can still reach the database container using the hostname db. Networking in Compose in the Docker documentation describes this further.
Hello I have multiple projects that have there own dockerfiles and docker-compose.yml files. I am not too familiar on how I would setup the networking between these projects. So they could share the same databases and the project would be able to talk to on another. Does anyone have suggests?
Right now, In one of the projects I am just pulling in all the dockerfile into a docker-compose.yml and setting-up all the services I need from all the other projects in this yml file. I do not think this is ideal and there is a high level a coupling between the services.
version: "3"
services:
db:
image: mysql/mysql-server
ports:
- 3306:3306
mongo:
image: mongo
restart: always
rails_app:
build:
context: ${RAILS_APP_PATH}
dockerfile: Dockerfile
volumes:
- ${RAILS_APP_PATH}:/application
ports:
- 4000:4000
depends_on:
- db
- mongo
links:
- db
- mongo
frontend:
build:
context: ${FRONTEND_PATH}
ports:
- ${EXPOSED_PORT}:${EXPOSED_PORT}
depends_on:
- go_services
links:
- go_services
go_services:
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
depends_on:
- db
- mongo
- rails_app
links:
- db
- mongo
- rails_app
The trick is to use an External Docker Network.
Set up the network and the Containers can talk to each other by their Service Names.
Setup the the network on the Host
docker network create my-net
First compose file
version: '3.9'
services:
mymongo:
image: mongo:latest
restart: unless-stopped
container_name: mongo
environment:
MONGO_INITDB_DATABASE: mymongo
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: password
volumes:
- ./database:/data/db
ports:
- "27017:27017"
networks:
default:
external: true
name: my-net
Second compose file
version: '3.9'
services:
ui:
build:
context: ./build
dockerfile: Dockerfile_ui
image: ui
restart: "no"
container_name: ui
ports:
- "8005:3000"
command: ["npm", "start"]
networks:
default:
external: true
name: my-net
You can do this without any special Compose setup, if:
each project is self-contained (they do not share databases)
the service locations are configurable via environment variables
you don't mind communicating via the host
If you're thinking about scaling up this project at all, this approach can look attractive. It will work even if you're running each Compose file on a different host, and it translates well into clustered environments like Kubernetes.
Go ahead and break up your Compose file into several independent ones:
# rails/docker-compose.yml
version: '3.8'
services:
db:
image: mysql/mysql-server
app:
build: .
ports: ['4000:4000']
depends_on: [db]
# go/docker-compose.yml
services:
mongo:
image: mongo
service:
build: .
ports: ['8080:8080']
depends_on: [mongo]
environment:
- RAILS_APP_URL
The very last line here passes the RAILS_APP_URL environment variable from the host environment into the container.
You can start the Rails application independently:
docker-compose -f ./rails/docker-compose.yml up -d
You need to find some hostname where the container can call back to the host. On MacOS and Windows hosts, Docker provides a special hostname host.docker.internal for this. You can then connect the client container to the published port of its server:
export RAILS_APP_URL=http://host.docker.internal:4000
docker-compose -f ./go/docker-compose.yml up
If you're doing development, you can run the service you're working on locally, and its dependencies in containers, and point the environment variable at the container
go build -o ./server ./cmd/server
export RAILS_APP_URL=http://localhost:4000
./server
If you want to run this setup on multiple hosts but without using a dedicated cluster manager like Docker Swarm or Kubernetes, set the environment variable to point at the DNS name of the host running the service. If you did want to translate this to Kubernetes, a Helm "chart" would be analogous, containing the Deployment, Service, etc. and dependencies for a single component, and you could configure the other service's URL through Helm values.
I have 2 different services in 2 distinct docker-compose.yml files in 2 different locations.
Service 1: wordpress
version: "3.7"
services:
# Wordpress
wordpress:
depends_on:
- db
container_name: wordpress
image: wordpress:latest
ports:
- '8000:80'
restart: unless-stopped
volumes: ['./:/var/www/html']
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_USER: wordpress
WORDPRESS_DB_PASSWORD: wordpress
networks:
- wpsite
networks:
wpsite:
driver: bridge
Service 2: frontend
version: "3.7"
services:
frontend:
depends_on:
- wordpress
container_name: frontend
restart: unless-stopped
stdin_open: true
build:
context: ../realm-frontend
volumes:
- static:/realm-frontend/build
networks:
- cms_wpsite
networks:
cms_wpsite:
external: true
I have a shell script that goes to the 2 locations and runs the docker-compose files to create the containers.
Script
cd ~/cms;
docker-compose -f docker-compose.yml up -d --build --force-recreate
cd ../frontend;
docker-compose -f docker-compose.yml up -d --build --force-recreate
As you can see I have created the link between the networks using bridge.
When I docker network inspect {network id} I can see both the containers wordpress and frontend are in the network. However, when the second container is created with the depends_on command, I get the following error.
ERROR: Service 'frontend' depends on service 'wordpress' which is undefined.
I am not sure why this is, given that they're in the same network.
I'd appreciate any help. Thanks!
Depends_on only works on services within the same compose file, so to do what you want, you would need to use something like wait-for-it.sh. Take a look here for more information: https://docs.docker.com/compose/startup-order/
Something like this may work for you or you can create a custom wait-for-it script as well:
services:
frontend:
container_name: frontend
restart: unless-stopped
stdin_open: true
build:
context: ../realm-frontend
volumes:
- static:/realm-frontend/build
command: ["./wait-for-it.sh", "wordpress:80", "--", "yourfrontendcmd"]
networks:
- cms_wpsite
I think you misunderstanding there.
depends_on: only works in a docker-compose file and only says which order to start and stop a container.
https://docs.docker.com/compose/compose-file/#depends_on
I have the following configuration in my docker-composer.yml file.
version: '3.3'
services:
service-1:
container_name: 'service-1'
build: './service-1'
depends_on:
- 'mongo'
- 'consul'
networks:
backend:
aliases:
- service-1
service-2:
build: './service-2'
ports:
- '8825:8825'
- '8835:8835'
networks:
frontend:
backend:
aliases:
- service-2
depends_on:
- 'mongo'
- 'consul'
consul:
image: 'consul:latest'
networks:
backend:
aliases:
- consul
mongo:
image: 'mongo:latest'
networks:
backend:
aliases:
- mongo
networks:
frontend:
backend:
internal: true
When my containers start they are not able to communicate between each other using host names.
Most of containers use the mongo db container, but they are not able even reach it and I am getting the following error.
Error connecting to mongo : no reachable servers
Please help me to solve the problem, I got stuck.
Thanks.
You've got a lot of unneeded settings in the compose file, here's a stripped down version that would work just as well:
version: '3.3'
services:
service-1:
build: './service-1'
networks:
- backend
service-2:
build: './service-2'
ports:
- '8825:8825'
- '8835:8835'
networks:
- frontend
- backend
consul:
image: 'consul:latest'
networks:
- backend
mongo:
image: 'mongo:latest'
networks:
- backend
networks:
frontend:
backend:
internal: true
You automatically get the alias of the service name for each container, no need to duplicate that. You also lose the ability to scale a service if you give it a container name. I'd also recommend moving the build step out of the compose file and use an image name for the apps you're building locally.
Now for the likely issue, you have a depends_on in your compose file. At best, this will not do what you're looking for. All it checks that the other container has been created and started, but not that the application inside is ready to serve traffic, and a DB may take time to become available. At worst, you'll get an error that it's unsupported if you try to move this into swarm mode.
Instead of depending on docker for this, update your application entrypoint to check for the external dependencies and wait a minute or two for them to become available before failing. A very simple example tool for this is wait-for-it that is written as a bash shell script.
I'm learning docker. I see those two terms that make me confused. For example here is a docker-compose that defined two services redis and web-app.
services:
redis:
container_name: redis
image: redis:latest
ports:
- "6379:6379"
networks:
- lognet
app:
container_name: web-app
build:
context: .
dockerfile: Dockerfile
ports:
- "3000:3000"
volumes:
- ".:/webapp"
links:
- redis
networks:
- lognet
networks:
lognet:
driver: bridge
This docker-compose file defines a bridge network named lognet and all services will connect to this network. As I understand, this action makes those services see others. So why app service still needs to link to redis service in the above case?
Thanks
Links have been replaced by networks. Docker describes them as a legacy feature that you should avoid using. You can safely remove the link and the two containers will be able to refer to each other by their service name (or container_name).
With compose, links do have a side effect of creating an implied dependency. You should replace this with a more explicit depends_on section so that the app doesn't attempt to run without or before redis starts.
As an aside, I'm not a fan of hard coding container_name unless you are certain that this is the only container that will exist with that name on the host and you need to refer to it from the docker cli by name. Without the container name, docker-compose will give it a less intuitive name, but it will also give it an alias of redis on the network, which is exactly what you need for container to container networking. So the end result with these suggestions is:
version: '2'
# do not forget the version line, this file syntax is invalid without it
services:
redis:
image: redis:latest
ports:
- "6379:6379"
networks:
- lognet
app:
container_name: web-app
build:
context: .
dockerfile: Dockerfile
ports:
- "3000:3000"
volumes:
- ".:/webapp"
depends_on:
- redis
networks:
- lognet
networks:
lognet:
driver: bridge