I'm using docker-compose to start a job on a remote server:
version: "3.9"
services:
solution-app:
container_name: solution_new_name_v3
build: .
I ran the following command to deploy the container:
docker-compose -H "ssh://..." up -d --build
This works great, now I wanted to start another job in parallel using the same docker compose file, but with some minor changes in the source code. So I changed the container_name in the compose config and assumed that it should start a new container. However, instead of starting a new container my previous container was stopped and replaced with a new one with this new name.
So my question is: how can I start few containers running in parallel using same docker-compose config?
I figure out the way:
so Docker compose maps all services based on the project name, which is by default a directory name. To override the project name one can use:
docker-compose -p NAME
Alternatively one can set COMPOSE_PROJECT_NAME env variable
Related
Is there any proper way of restarting an entire docker compose stack from within one of its containers?
One workaround involves mounting the docker socket:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
and then use the Docker Engine SDKs (https://docs.docker.com/engine/api/sdk/examples/).
However, this solution only allows restarting the containers itselves. There seems to be no way to send compose commands, like docker compose restart, docker compose up, etc.
The only solution I've found to send docker compose commands is to open a terminal on the host from the container using ssh, like this: access host's ssh tunnel from docker container
This is partly related to How to run shell script on host from docker container? , but I'm actually looking for a more specific solution to only send docker compose commands.
I tried with this simple docker-compose.yml file
version: '3'
services:
nginx:
image: nginx
ports:
- 3000:80
Then I started a docker container using
docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/work docker
Then, inside the container, I did
cd /work
docker-compose up -d
and it started the container up on the host.
Please note that you have an error in your socket mapping. It needs to be
- /var/run/docker.sock:/var/run/docker.sock
(you have a period instead of a slash at one point)
As mentioned by #BMitch in the comments, compose project name was the reason why I wasn't able to run docker compose commands inside the running container.
By default the compose project name is set to the directory name, so if the docker-compose.yml is run from a host directory named folder1, then the commands inside the container should be run as:
docker-compose -p folder1 ...
So now, for example, restarting the stack works:
docker-compose -p folder1 restart
Just as a reference, a fixed project name for your compose can be set using name: ... as a top-level attribute of the .yml file, but requires docker compose v2.3.3 : Set $PROJECT_NAME in docker-compose file
I wish to use custom container names in docker-compose.yml. I used the clause container_name like the example below:
...
services:
mysql:
image: "mysql"
container_name: mysql
...
When I execute docker-compose up -d It works fine, all the containers in .yml starts and set all up.
But when I execute docker-compose down, the project name defaults to my project folder name and it can't locate the custom named containers I wrote in docker-compose.yml.
How can I use docker-container down with custom container names?
Thanks to Eranga Heshan's comment I retested and the command run successfully. It was not a error of docker compose error as I inquired and thought it was.
I have a large Docker project with Dockerfiles for nginx, apache2, varnish, redis configured and working well after weeks of changes and testing.
I am now at a point where I setup the projects to use docker-compose and override.yml files for easy setup:
I am trying to use the same docker-compose setup for multiple projects (websites)
Normal startup (using docker-compose.yml and optional docker-compose.override.yml)
docker-compose up -d
Custom startup (using specific docker-compose files)
docker-compose -f docker-compose.yml -f custom/docker-compose.website1.yml up -d
Both these methods starts up fine:
docker-compose ps
Ignore the fact that they are Exit 0 - I stopped them using docker-compose stop, the containers work fine
nginx-proxy /usr/bin/supervisord Exit 0
redis-cache /usr/bin/supervisord Exit 0
varnish-cache /usr/bin/supervisord Exit 0
web-server-apache2 /usr/bin/supervisord Exit 0
Now I want a second project (website) to use the same docker/docker-compose configuration setup:
docker-compose -f docker-compose.yml -f anothercustomfolder/docker-compose.website2.yml up -d
To my surprise docker-compose recreated containers and do not create a new set of containers:
See 'current setup' section for how I setup things.
Creating network "delete-network-frontend" with the default driver
Recreating nginx-proxy ... done
Recreating varnish-cache ... done
Recreating web-server ... done
Recreating redis-cache ... done
When running docker-compose ps in the second setup folder:
Note the names are not the same as above (this is the second test setup)
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy-delete /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
It appears docker-compose did two things : 1. Recreate (replace) the project 1 containers, used the project 1 container names to mention that they were 'recreated', and 2. Remove the project 1 containers, renamed it to project 2 containers.
Current setup
I created a full Dockerfile project configured with docker-compose.yml and two override docker-compose files (docker-compose.website1.yml and docker-compose.website2.yml`).
I made a complete copy of the working Dockerfile / docker-compose.yml project and created a new folder: In other words both these will use the same docker setup but use different docker-compose.yml override files.
/var/www/docker/site1
/var/www/docker/site2
Question
TLDR: How do I use a working docker-compose project on the same host operating system for multiple projects... without it replacing another project's containers.
I want to be able to see (use both) at the same time, and for instance be able to see this:
Ignore the fact that the ports are the same here, I am aware they won't run at the same time, I will update the project docker-compose.yml custom files when this works
docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
If anyone asks: Why not just put the websites into the same (one) container??
For the possibility someone might ask this, I know I can add multiple websites into the /etc/apache2/sites-enabled (or nginx) and add custom configuration files using ADD in Dockerfile for each site, but using that method I cannot test different slight setups.
Different setups that can be used by referencing another different image in the 'override docker-compose files'
For instance I can create a Dockerfile that installs all php7.3 libraries required and run Magento 2.3 on it, then have another Dockerfile to test php7.4, and have another to run an older Magento 1 site on a PHP5.6 installation and so on.
Thanks to advice from David Maze, I struggled further with configuring the docker-compose setup to work with multiple projects.
Information based on docker-compose v1.25.0 (July 2020)
This discussion is especially important when you want to re-use (persist) your containers (start/stop instead of just up/down - deleting)
As initially pointed out in my question - if you try to create containers using docker-compose up -d there are some pitfalls which the tool simply does not support right at the moment.
Pitfalls
PITFALLS OF CURRENT DOCKER-COMPOSE IMPLEMENTATION:
If you just use overridden docker-compose*.yml with different container_names (per 'project') with files in the same folder
docker-compose up will simply replace existing containers as explained in my question.
You can do the following: docker-compose -p CUSTOM_PROJECT_NAME -f file1.yml -f file2.yml up -d, but:
This on its own is useless - these containers will only work until you want stop them. As soon as you want to do docker-compose start (to restart existing container set) it will simply fail with Error: No containers to start
If you use two different folders with the same docker-compose project (ie cloned project): for instance ./dc-project1 ./dc-project2 but using container_name field inside docker-compose.*.yml file:
When you try to run docker-compose -f f1.yml -f f2.yml up -d inside ./dc-project1 and the same inside ./dc-project2 folder, you will get the following error: You have to remove (or rename) that container to be able to reuse that name.
Similar issues with your Docker network will occur with docker-compose when you use overridden files:
Removed most of the custom settings to make the network setting clearer:
Network will be attached correctly from your overridden file on docker-compose up, but as soon as you want to docker-compose start it looks for your default network name: in the default docker-compose.yml or even the docker-compose.override.yml file if it exists. In other words - it ignores your custom docker-compose override files (see example below):
docker-compose.yml:
networks:
network_frontend:
name: stage6-network-frontend
customfolder/docker-compose.custom.yml:
networks:
network_frontend:
name: magento2.3-network-frontend
SOLUTION
Example
Objective : to get docker-compose start/stop to work correctly with multiple setups (aka projects/websites/tools)using the same docker-compose project.
Suppose you have the following docker-compose files:
**Main file: ** docker-compose.yml:
web_server:
image: current_timezone/full-supervisord-web-server/php7.3:1.00
container_name: web-server-apache2
networks:
- network_frontend
build:
context: "./all-services/"
dockerfile: ./web-server/Dockerfile.webserver.apache2
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
name: stage6-network-frontend
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: docker-custom # Seems limit of 15 characters only
and then an override file: customfolder/magento2.override.yml:
web_server:
container_name: web-server-apache2-magento2.3.5
networks:
- network_frontend
build:
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: d-glo-femag2_35 # Seems limit of 15 characters only
name: glo-magento2.3-network-frontend
Do the following:
Copy the full Docker project (Dockerfiles/ADDs/docker-compose.yml files etc) into a new seperate folder:
/var/docker/project1
/var/docker/project2
Make sure that the container_name entries in your override docker-compose.yml are unique between the two projects.
In project project1 run docker-compose -f docker-compose.yml -f customfolder/magento2.override.yml up -d && docker-compose stop, navigate to project2 and do the same.
Using -p flag as David Maze suggested does not work on its own, JSON files are still sourced as ./foldername on docker-compose start/stop
Since networks are having similar issues on start/stop , before you can correctly use your custom name defined in your override file, .... unfortunately you need to update the main base docker-compose.yml to the overridden file!
Extended explanation: There is no way to call the correct custom network name from docker-compose start, so since docker-compose ignores the overridden files on start, you need to make sure to update the base file docker-compose.yml or docker-compose.override.yml has your custom network name.
In case you have not updated the names before using up -d, you will need to replace the content of each /var/lib/docker/containers/*/config.v2.json.
For example you could do this: you have to stop docker first
sudo service docker stop
find /var/lib/docker/containers/ -type f -name "config.v2.json" -exec sed -i "s|wrong-network-name|overridden-network-name|g" '{}' \;
sudo service docker start
IF done correctly, you should have unique container names, and each folder can be accessed separately correctly now without it breaking each other's containers: docker-compose start, docker-compose stop, docker-compose ps
NOTE: You still need to navigate to the seperate folder to run those commands
When calling docker-compose in different directories, I get conflict errors and problems with networking:
Problem with conflicts
docker-compose.yml
version: '3'
services:
redis:
image: "redis:alpine"
container_name: redis
I. create and start docker container by docker-compose => OK
$ docker-compose up --force-recreate -d
Creating redis ... done
II. recreate and start docker container by docker-compose => OK
$ docker-compose up --force-recreate -d
Recreating redis ... done
III. copy docker-compose.yml to other directory.
Then try to recreate from other directory => ERROR
$ cp docker-compose.yml red2/
$ cd red2/
$ docker-compose up --force-recreate -d
Creating redis ... error
ERROR: for redis Cannot create container for service redis: Conflict. The container name "/redis" is already in use by container "1ba060b545f716731ac1c5992b680e4d4b3639fc0ffeb291899c712f0839d23a". You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.
Different Networks
Containers created from docker-compose in different directories also do not share the same network.
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
4a4af52e89cd red2_default bridge local
57695428bd9d redis_default bridge local
Usecase
My usecase for that szenario:
Call docker-compose from different deployment jobs.
Start containers for testing
Questions
Why is there the directory dependency? Is there an option to switch it off?
Does docker ps show which directory was used?
Answer for 1:
The directory name is used as the default project name.
You should better specify the project name:
docker-compose -p myproject up --force-recreate -d
Question 2 still open
docker and docker-compose seem to be interacting with the same dockerFile, what is the difference between the two tools?
The docker cli is used when managing individual containers on a docker engine. It is the client command line to access the docker daemon api.
The docker-compose cli can be used to manage a multi-container application. It also moves many of the options you would enter on the docker run cli into the docker-compose.yml file for easier reuse. It works as a front end "script" on top of the same docker api used by docker, so you can do everything docker-compose does with docker commands and a lot of shell scripting. See this documentation on docker-compose for more details.
Update for Swarm Mode
Since this answer was posted, docker has added a second use of docker-compose.yml files. Starting with the version 3 yml format and docker 1.13, you can use the yml with docker-compose and also to define a stack in docker's swarm mode. To do the latter you need to use docker stack deploy -c docker-compose.yml $stack_name instead of docker-compose up and then manage the stack with docker commands instead of docker-compose commands. The mapping is a one for one between the two uses:
Compose Project -> Swarm Stack: A group of services for a specific purpose
Compose Service -> Swarm Service: One image and it's configuration, possibly scaled up.
Compose Container -> Swarm Task: A single container in a service
For more details on swarm mode, see docker's swarm mode documentation.
docker manages single containers
docker-compose manages multiple container applications
Usage of docker-compose requires 3 steps:
Define the app environment with a Dockerfile
Define the app services in docker-compose.yml
Run docker-compose up to start and run app
Below is a docker-compose.yml example taken from the docker docs:
services:
web:
build: .
ports:
- "5000:5000"
volumes:
- .:/code
- logvolume01:/var/log
links:
- redis
redis:
image: redis
volumes:
logvolume01: {}
A Dockerfile is a text document that contains all the commands/Instruction a user could call on the command line to assemble an image.
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. By default, docker-compose expects the name of the Compose file as docker-compose.yml or docker-compose.yaml. If the compose file has a different name we can specify it with -f flag.
Check here for more details
docker or more specifically docker engine is used when we want to handle only one container whereas the docker-compose is used when we have multiple containers to handle. We would need multiple containers when we have more than one service to be taken care of, like we have an application that has a client server model. We need a container for the server model and one more container for the client model. Docker compose usually requires each container to have its own dockerfile and then a yml file that incorporates all the containers.