Is it possible to create services in a loop with docker-compose rather than typing all the services by hand? (See example below of creating 100 workers with appropriate ports)
version: '3'
services:
redis:
image: redis
worker1:
build: .
ports:
- "5001:5001"
worker2:
build: .
ports:
- "5002:5002"
worker3:
build: .
ports:
- "5003:5003"
...
worker100:
build: .
ports:
- "5100:5100"
You can probably do it with the --scale option, so if you run docker-compose up --scale worker=100 it should do exactly what you want.
The documentation for docker-compose up references this as follows:
--scale SERVICE=NUM Scale SERVICE to NUM instances. Overrides the
`scale` setting in the Compose file if present.
Related
I have a docker-compose.yml
services:
nextjs:
container_name: next_app
build:
context: ./
restart: on-failure
command: npm run dev
volumes:
- ./:/app
- /app/node_modules
- /app/.next
ports:
- "3000:3000"
cypress:
image: "cypress/included:9.4.1"
depends_on:
- next_app
environment:
- CYPRESS_baseUrl=http://nextjs:3000
working_dir: /e2e
volumes:
- ./e2e:/e2e
I want to change env_file for next_app from cypress service. I found solution like this
cypress:
image: "cypress/included:9.4.1"
depends_on:
- next_app
environment:
- CYPRESS_baseUrl=http://nextjs:3000
working_dir: /e2e
volumes:
- ./e2e:/e2e
next_app:
env_file: .env.test
But this solution does not work. Is it even possible ?
Try something like cp .env #docker/.env
No. In Compose (or Docker, or even more generally in Linux/Unix) there is no way for one container (process) to specify environment variables for another.
You can think of a docker-compose.yml file as a set of instructions only for running containers. If you need a specific set of containers for a specific context – you don't normally need to run Cypress in production, but this is an integration-test setup – it's fine to write a separate Compose file just for that setup.
# docker-compose.cypress.yml
# Used only for integration testing
version: '3.8'
services:
nextjs:
build: .
restart: on-failure
ports:
- "3000:3000"
env_file: .env.test # <-- specific to this test-oriented Compose file
cypress:
build: ./e2e
depends_on:
- nextjs
environment:
- CYPRESS_baseUrl=http://nextjs:3000
docker-compose -f docker-compose.cypress.yml up --build
This can also be a case where using multiple Compose files together can be a reasonable option. You can define a "standard" Compose setup that only defines the main service, and then an e2e-test Compose file that adds the Cypress container and the environment settings.
# docker-compose.yml
version: '3.8'
services:
nextjs:
image: registry.example.com/nextjs:${NEXTJS_TAG:-latest}
restart: on-failure
ports:
- '3000:3000'
# docker-compose.e2e.yaml
version: '3.8'
services:
nextjs:
# These add to the definitions in the base `docker-compose.yml`
build: .
env_file: .env.test
cypress:
# This is a brand new container for this specific setup
depends_on: [nextjs]
et: cetera # copy from question or previous Compose setup
docker-compose \
-f docker-compose.yml \
-f docker-compose.e2e.yml \
up --build
Hello I have multiple projects that have there own dockerfiles and docker-compose.yml files. I am not too familiar on how I would setup the networking between these projects. So they could share the same databases and the project would be able to talk to on another. Does anyone have suggests?
Right now, In one of the projects I am just pulling in all the dockerfile into a docker-compose.yml and setting-up all the services I need from all the other projects in this yml file. I do not think this is ideal and there is a high level a coupling between the services.
version: "3"
services:
db:
image: mysql/mysql-server
ports:
- 3306:3306
mongo:
image: mongo
restart: always
rails_app:
build:
context: ${RAILS_APP_PATH}
dockerfile: Dockerfile
volumes:
- ${RAILS_APP_PATH}:/application
ports:
- 4000:4000
depends_on:
- db
- mongo
links:
- db
- mongo
frontend:
build:
context: ${FRONTEND_PATH}
ports:
- ${EXPOSED_PORT}:${EXPOSED_PORT}
depends_on:
- go_services
links:
- go_services
go_services:
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
depends_on:
- db
- mongo
- rails_app
links:
- db
- mongo
- rails_app
The trick is to use an External Docker Network.
Set up the network and the Containers can talk to each other by their Service Names.
Setup the the network on the Host
docker network create my-net
First compose file
version: '3.9'
services:
mymongo:
image: mongo:latest
restart: unless-stopped
container_name: mongo
environment:
MONGO_INITDB_DATABASE: mymongo
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: password
volumes:
- ./database:/data/db
ports:
- "27017:27017"
networks:
default:
external: true
name: my-net
Second compose file
version: '3.9'
services:
ui:
build:
context: ./build
dockerfile: Dockerfile_ui
image: ui
restart: "no"
container_name: ui
ports:
- "8005:3000"
command: ["npm", "start"]
networks:
default:
external: true
name: my-net
You can do this without any special Compose setup, if:
each project is self-contained (they do not share databases)
the service locations are configurable via environment variables
you don't mind communicating via the host
If you're thinking about scaling up this project at all, this approach can look attractive. It will work even if you're running each Compose file on a different host, and it translates well into clustered environments like Kubernetes.
Go ahead and break up your Compose file into several independent ones:
# rails/docker-compose.yml
version: '3.8'
services:
db:
image: mysql/mysql-server
app:
build: .
ports: ['4000:4000']
depends_on: [db]
# go/docker-compose.yml
services:
mongo:
image: mongo
service:
build: .
ports: ['8080:8080']
depends_on: [mongo]
environment:
- RAILS_APP_URL
The very last line here passes the RAILS_APP_URL environment variable from the host environment into the container.
You can start the Rails application independently:
docker-compose -f ./rails/docker-compose.yml up -d
You need to find some hostname where the container can call back to the host. On MacOS and Windows hosts, Docker provides a special hostname host.docker.internal for this. You can then connect the client container to the published port of its server:
export RAILS_APP_URL=http://host.docker.internal:4000
docker-compose -f ./go/docker-compose.yml up
If you're doing development, you can run the service you're working on locally, and its dependencies in containers, and point the environment variable at the container
go build -o ./server ./cmd/server
export RAILS_APP_URL=http://localhost:4000
./server
If you want to run this setup on multiple hosts but without using a dedicated cluster manager like Docker Swarm or Kubernetes, set the environment variable to point at the DNS name of the host running the service. If you did want to translate this to Kubernetes, a Helm "chart" would be analogous, containing the Deployment, Service, etc. and dependencies for a single component, and you could configure the other service's URL through Helm values.
Currently I have a rabbitmq message broker and multiple celery workers that need to be containerized. My problem is, how can I fire up containers using different docker-compose.yml? My goal is to start the rabbitmq once and for all, and never touch it again.
Currently I have a docker-compose.yml for the rabbitmq:
version: '2'
services:
rabbit:
hostname: rabbit
image: rabbitmq:latest
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
ports:
- "5672:5672"
expose:
- "5672"
And another docker-compose.yml for celery workers:
version: '2'
services:
worker:
build:
context: .
dockerfile: dockerfile
volumes:
- .:/app
environment:
- CELERY_BROKER_URL=amqp://admin:mypass#rabbit:5672
links:
- rabbit
However, when I do docker-compose up for celery workers, I keep getting the following error:
ERROR/MainProcess] consumer: Cannot connect to
amqp://admin:**#rabbit:5672//: failed to resolve broker hostname.
Can anyone take a look if there is anything wrong with my code? Thanks.
the domain name rabbit in your second docker-compose.yml file does not resolve because there is no service with that name in that docker-compose.yml file.
As stated in the comments, one solution is to put both the rabbit service and worker service in the same docker-compose.yml file. In such a setup, all containers started for those services would join the same docker network and those service names could be resolved to the IP adresses of their containers.
Since having a single docker-compose.yml file is not convenient in your case, you have to find an other way to have the containers originating from different docker-compose.yml files join a same docker network.
To do so, you need to create a dedicated docker network for that purpose:
docker network create rabbitNetwork
Then, in each docker-compose.yml file, you need to refer to this network in the services definitions:
version: '2'
services:
rabbit:
hostname: rabbit
image: rabbitmq:latest
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
# ports:
# - "5672:5672" # there is no need to publish ports on the docker host anymore
expose:
- "5672"
networks:
- rabbitNet
networks:
rabbitNet:
external:
name: rabbitNetwork
version: '2'
services:
worker:
build:
context: .
dockerfile: dockerfile
volumes:
- .:/app
environment:
- CELERY_BROKER_URL=amqp://admin:mypass#rabbit:5672
networks:
- rabbitNet
networks:
rabbitNet:
external:
name: rabbitNetwork
You can use any file as service definition.
docker-compose.yml is default file name but any other name can be passed using -f argument.
docker-compose -f rabbit-compose.yml COMMAND
I want to have two docker-compose files, where one overrides another.
(The motivation comes from Docker Compose Docs)
The use case comes from the buildbot environment. The first docker-compose file should define a simple service. This is a service that is going to be tested. Let's take
version: '2'
services:
service-node:
build:
context: ./res
dockerfile: Dockerfile
image: my/server
env_file: .env
The second docker-compose file (let's name it docker-compose.test.yml) overrides the service-node to add a buildbot worker feature, and creates the second container, i.e. buildbot master node, that is going to control testing machinery. Let's take
version: '2'
services:
service-node:
build:
context: ./res
dockerfile: buildbot.worker.Dockerfile
image: my/buildbot-worker
container_name: bb-worker
env_file: ./res/buildbot.worker.env
environment:
- BB_RES_DIR=/var/lib/buildbot
networks:
testlab:
aliases:
- bb-worker
volumes:
- ./vol/bldbot/worker:/home/bldbotworker
depends_on:
- bb-master
bb-master:
build:
context: ./res
dockerfile: buildbot.master.Dockerfile
image: my/buildbot-master
container_name: bb-master
env_file: ./res/buildbot.master.env
environment:
- BB_RES_DIR=/var/lib/buildbot
networks:
- testlab
expose:
- "9989"
volumes:
- ./vol/bldbot/master:/var/lib/buildbot
networks:
testlab:
driver: bridge
Generally this configuration works, i.e. the command
docker-compose -f docker-compose.yml -f docker-compose.test.yml up -d
builds both images and runs both containers, but there is one shortcoming, i.e. the command
docker-compose ps
shows only one service, bb-worker. At the same time
docker ps
shows both.
Furthermore, the command
docker-compose down
stops only one service, and outputs the message/warning Found orphan containers. Of course, the message refers to bb-master.
How can I override the basic docker-compose.yml file to be able to add additional non-orphan service?
You need to run all docker-compose commands with the flags, e.g.:
docker-compose -f docker-compose.yml -f docker-compose.test.yml down
Alternatively, you can make this the default by writing the following to a .env file in the same folder:
COMPOSE_FILE=docker-compose.yml:docker-compose.test.yml
NOTE:
In windows you need tu use ";" as the separator (#louisvno)
I am completely new to Docker. I have a sonarqube image. I wrote a sample Docker Compose file and ran the image. Everything is working fine. Now I want to create 5-6 containers using Docker Compose.
How can I do this?
This is my sample YAML file:
version: '2'
services:
web:
build: .
ports:
- "9000:9000"
depends_on:
- my_image
my_image:
image: mySonarApp
I also would be interested in knowing if it is possible to create them only with my docker config file.
Just add more service definitions
version: '2'
services:
weba:
build: .
ports:
- "9000:9000"
weba:
build: .
ports:
- "9001:9000"
webc:
build: .
ports:
- "9002:9000"
You can use the scale command to create more instances of the service:
docker-compose scale web=6