I have a docker-compose project which (simplified) looks like:
version: '3'
services:
main:
image: main-image
depends_on:
- my_service
my_service:
image: very-big-image
The functionality supplied by my-service is also available in the cloud. Running it locally is faster, but demands extreme CPU and disk resources. Therefore, I will sometimes want to run just main, and other times will want to run both containers.
I don't want to duplicate the docker-compose code; the full container definitions are much more complex than the simplified version above.
I thought I could put each service in its own .yml file and then do docker-compose -f main.yml -f my_service.yml up, but that fails on the depends-on line, which seems to require the dependency to be in the same file.
For now, I have an ugly solution. I split into two (with the second files starting with the depends_on line. I then use a shell script that knows which configuration I want to run, and either runs the first file or concatenates the two files together. This works, I guess; but is ugly.
What is the right way to do this?
Try this:
docker-compose start <container_name>
This command only starts a selected container.
What is the right way to do this?
I think you've already identified it: split the service definitions between two files, so that sometimes you can start both:
docker-compose -f main.yml -f service.yml up
And sometimes you can start just the service container:
docker-compose -f service.yml up
This works fine, and the depends_on key does not require that all definitions be in the same file. The dictionaries are merged before docker-compose tries to resolve dependencies.
For example, if I have in main.yml:
---
version: "3"
services:
main:
image: alpine
command: sleep 30
depends_on:
- service
And in service.yml:
---
version: "3"
services:
service:
image: alpine
command: sleep 30
I can start just the service container like this:
$ docker-compose -f service.yml up
Starting compose_service_1 ... done
Attaching to compose_service_1
Or I can start both the main and service containers:
$ docker-compose -f main.yml -f service.yml up
Starting compose_service_1 ... done
Starting compose_main_1 ... done
Attaching to compose_service_1, compose_main_1
If you're seeing behavior that appears to contradict this, it would help if you were to update your question with minimal examples that reproduce the problem, and include any error output from running docker-compose.
Related
I'm writing an automated test that involves running several containers at once. The test submits some workload to the tested service, and expects a callback from it after a time.
To run the whole system, I use docker compose run with the following docker-compose file:
version: "3.9"
services:
service:
build: ...
ports: ...
tester:
image: alpine
depends_on:
- service
profiles:
- testing
The problem is, I can see "service" from "tester", but not the other way around, so the callback from the service could not land to "tester":
$ docker compose -f .docker/docker-compose.yaml run --rm tester \
nslookup service
Name: service
Address 1: ...
$ docker compose -f .docker/docker-compose.yaml run --rm service \
nslookup tester
** server can't find tester: NXDOMAIN
I tried specifying the same network for them, and giving them "links", but the result is the same.
It seems like a very basic issue, so perhaps I'm missing something?
When you docker-compose run some-container, it starts a temporary container based on that description plus the things it depends_on:. So, when you docker-compose run service ..., it doesn't depends_on: anything, and Compose only starts the temporary container, which is why the tester container doesn't exist at that point.
If you need the whole stack up to make connections both ways between containers, you need to run docker-compose up -d. You can still docker-compose run temporary containers on top of these.
Let's say I have the following docker-compose.yml file:
version: '3'
services:
load-balancer:
...
...
web-application:
...
...
If I want to run this with 5 replicas of web-application, I have to issue this command:
docker-compose up --scale web-application=5
Is there any way to tell Docker to do the --scale web-application=5 bit from within the docker-compose.yml file?
You can specify the number of replicas in docker-compose only when working in swarm mode.
See the deploy directive.
Example:
services:
redis:
image: redis:latest
deploy:
replicas: 2
From the documentation:
This only takes effect when deploying to a swarm with
docker stack deploy, and is ignored by docker-compose up
and docker-compose run.
Lastly, there is some discussion about this feature (since it used to be possible) in this GitHub issue, and here is the mention in the most recent compose spec.
I tested replicas in docker compose file, the code that worked for me is the following.
You should use docker-compose up -d to execute the configuración
You should see the next results.
The results in web browser should be the following:
I have a large Docker project with Dockerfiles for nginx, apache2, varnish, redis configured and working well after weeks of changes and testing.
I am now at a point where I setup the projects to use docker-compose and override.yml files for easy setup:
I am trying to use the same docker-compose setup for multiple projects (websites)
Normal startup (using docker-compose.yml and optional docker-compose.override.yml)
docker-compose up -d
Custom startup (using specific docker-compose files)
docker-compose -f docker-compose.yml -f custom/docker-compose.website1.yml up -d
Both these methods starts up fine:
docker-compose ps
Ignore the fact that they are Exit 0 - I stopped them using docker-compose stop, the containers work fine
nginx-proxy /usr/bin/supervisord Exit 0
redis-cache /usr/bin/supervisord Exit 0
varnish-cache /usr/bin/supervisord Exit 0
web-server-apache2 /usr/bin/supervisord Exit 0
Now I want a second project (website) to use the same docker/docker-compose configuration setup:
docker-compose -f docker-compose.yml -f anothercustomfolder/docker-compose.website2.yml up -d
To my surprise docker-compose recreated containers and do not create a new set of containers:
See 'current setup' section for how I setup things.
Creating network "delete-network-frontend" with the default driver
Recreating nginx-proxy ... done
Recreating varnish-cache ... done
Recreating web-server ... done
Recreating redis-cache ... done
When running docker-compose ps in the second setup folder:
Note the names are not the same as above (this is the second test setup)
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy-delete /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
It appears docker-compose did two things : 1. Recreate (replace) the project 1 containers, used the project 1 container names to mention that they were 'recreated', and 2. Remove the project 1 containers, renamed it to project 2 containers.
Current setup
I created a full Dockerfile project configured with docker-compose.yml and two override docker-compose files (docker-compose.website1.yml and docker-compose.website2.yml`).
I made a complete copy of the working Dockerfile / docker-compose.yml project and created a new folder: In other words both these will use the same docker setup but use different docker-compose.yml override files.
/var/www/docker/site1
/var/www/docker/site2
Question
TLDR: How do I use a working docker-compose project on the same host operating system for multiple projects... without it replacing another project's containers.
I want to be able to see (use both) at the same time, and for instance be able to see this:
Ignore the fact that the ports are the same here, I am aware they won't run at the same time, I will update the project docker-compose.yml custom files when this works
docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
If anyone asks: Why not just put the websites into the same (one) container??
For the possibility someone might ask this, I know I can add multiple websites into the /etc/apache2/sites-enabled (or nginx) and add custom configuration files using ADD in Dockerfile for each site, but using that method I cannot test different slight setups.
Different setups that can be used by referencing another different image in the 'override docker-compose files'
For instance I can create a Dockerfile that installs all php7.3 libraries required and run Magento 2.3 on it, then have another Dockerfile to test php7.4, and have another to run an older Magento 1 site on a PHP5.6 installation and so on.
Thanks to advice from David Maze, I struggled further with configuring the docker-compose setup to work with multiple projects.
Information based on docker-compose v1.25.0 (July 2020)
This discussion is especially important when you want to re-use (persist) your containers (start/stop instead of just up/down - deleting)
As initially pointed out in my question - if you try to create containers using docker-compose up -d there are some pitfalls which the tool simply does not support right at the moment.
Pitfalls
PITFALLS OF CURRENT DOCKER-COMPOSE IMPLEMENTATION:
If you just use overridden docker-compose*.yml with different container_names (per 'project') with files in the same folder
docker-compose up will simply replace existing containers as explained in my question.
You can do the following: docker-compose -p CUSTOM_PROJECT_NAME -f file1.yml -f file2.yml up -d, but:
This on its own is useless - these containers will only work until you want stop them. As soon as you want to do docker-compose start (to restart existing container set) it will simply fail with Error: No containers to start
If you use two different folders with the same docker-compose project (ie cloned project): for instance ./dc-project1 ./dc-project2 but using container_name field inside docker-compose.*.yml file:
When you try to run docker-compose -f f1.yml -f f2.yml up -d inside ./dc-project1 and the same inside ./dc-project2 folder, you will get the following error: You have to remove (or rename) that container to be able to reuse that name.
Similar issues with your Docker network will occur with docker-compose when you use overridden files:
Removed most of the custom settings to make the network setting clearer:
Network will be attached correctly from your overridden file on docker-compose up, but as soon as you want to docker-compose start it looks for your default network name: in the default docker-compose.yml or even the docker-compose.override.yml file if it exists. In other words - it ignores your custom docker-compose override files (see example below):
docker-compose.yml:
networks:
network_frontend:
name: stage6-network-frontend
customfolder/docker-compose.custom.yml:
networks:
network_frontend:
name: magento2.3-network-frontend
SOLUTION
Example
Objective : to get docker-compose start/stop to work correctly with multiple setups (aka projects/websites/tools)using the same docker-compose project.
Suppose you have the following docker-compose files:
**Main file: ** docker-compose.yml:
web_server:
image: current_timezone/full-supervisord-web-server/php7.3:1.00
container_name: web-server-apache2
networks:
- network_frontend
build:
context: "./all-services/"
dockerfile: ./web-server/Dockerfile.webserver.apache2
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
name: stage6-network-frontend
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: docker-custom # Seems limit of 15 characters only
and then an override file: customfolder/magento2.override.yml:
web_server:
container_name: web-server-apache2-magento2.3.5
networks:
- network_frontend
build:
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: d-glo-femag2_35 # Seems limit of 15 characters only
name: glo-magento2.3-network-frontend
Do the following:
Copy the full Docker project (Dockerfiles/ADDs/docker-compose.yml files etc) into a new seperate folder:
/var/docker/project1
/var/docker/project2
Make sure that the container_name entries in your override docker-compose.yml are unique between the two projects.
In project project1 run docker-compose -f docker-compose.yml -f customfolder/magento2.override.yml up -d && docker-compose stop, navigate to project2 and do the same.
Using -p flag as David Maze suggested does not work on its own, JSON files are still sourced as ./foldername on docker-compose start/stop
Since networks are having similar issues on start/stop , before you can correctly use your custom name defined in your override file, .... unfortunately you need to update the main base docker-compose.yml to the overridden file!
Extended explanation: There is no way to call the correct custom network name from docker-compose start, so since docker-compose ignores the overridden files on start, you need to make sure to update the base file docker-compose.yml or docker-compose.override.yml has your custom network name.
In case you have not updated the names before using up -d, you will need to replace the content of each /var/lib/docker/containers/*/config.v2.json.
For example you could do this: you have to stop docker first
sudo service docker stop
find /var/lib/docker/containers/ -type f -name "config.v2.json" -exec sed -i "s|wrong-network-name|overridden-network-name|g" '{}' \;
sudo service docker start
IF done correctly, you should have unique container names, and each folder can be accessed separately correctly now without it breaking each other's containers: docker-compose start, docker-compose stop, docker-compose ps
NOTE: You still need to navigate to the seperate folder to run those commands
I have an application that performs elaboration over a data feed. The process is divided into tasks so I structured a docker-compose.yml file like this:
task1-service:
image: task1-image
task2-service:
image: task2-image
task3-service:
image: task3-image
Each task-service is triggered by the end of the previous and triggers the next, then it can exit. So there's no point to keep each service running.
I wonder if there's a solution to keep them all stopped, and start each service on demand when needed.
I don't know if docker compose is the correct solution, but I like the idea ok keeping the system described into one yml file. Anyway, other solutions are appreciated.
Thanks
Is possible to solve your approach in different ways, and one of them is with docker-compose.
First, you can start one concrete service (taskX-service) using docker-compose up -d <service_name>
You have more details in docker-compose up for only certain containers
Second, docker-compose also allows you to configure dependencies between containers. If you want to run them in order, you can specify it in depends_on: structure.
For example, to execute tasks 1, 2, and 3 in order you could use:
task1-service:
image: task1-image
task2-service:
image: task2-image
depends_on:
- task1-service
task3-service:
image: task3-image
depends_on:
- task2-service
Furthermore, this docker-compose.yml is compatible with first I said:
docker-compose up -d task1-service
docker-compose up -d task2-service (also launchs task1-service)
docker-compose up -d task3-service (also launchs task2 and 1 service)
If you don't specify any container, with docker-compose down stops all containers in compose file.
I hope it's useful for you.
I am new to Docker and have docker-compose.yml which is containing many services and iI need to start one particular service. I have docker-compose.yml file with information:
version: '2'
services:
postgres:
image: ${ARTIFACTORY_URL}/datahub/postgres:${BUILD_NUMBER}
restart: "no"
volumes:
- /etc/passwd:/etc/passwd
volumes_from:
- libs
depends_on:
- libs
setup:
image: ${ARTIFACTORY_URL}/setup:${B_N}
restart: "no"
volumes:
- ${HOME}:/usr/local/
I am able to call docker-compose.yml file using command:
docker-compose -f docker-compose.yml up -d --no-build
But I need to start "setup service" in docker-compose file:
How can I do this?
It's very easy:
docker compose up <service-name>
In your case:
docker compose -f docker-compose.yml up setup -d
To stop the service, then you don't need to specify the service name:
docker compose down
will do.
Little side note: if you are in the directory where the docker-compose.yml file is located, then docker-compose will use it implicitly, there's no need to add it as a parameter.
You need to provide it in the following situations:
the file is not in your current directory
the file name is different from the default one, eg. myconfig.yml
As far as I understand your question, you have multiple services in docker-compose but want to deploy only one.
docker-compose should be used for multi-container Docker applications. From official docs :
Compose is a tool for defining and running multi-container Docker
applications.
IMHO, you should run your service image separately with docker run command.
PS: If you are asking about recreating only the container whose image is changed among the multiple services in your docker-compose file, then docker-compose handles that for you.