When calling docker-compose in different directories, I get conflict errors and problems with networking:
Problem with conflicts
docker-compose.yml
version: '3'
services:
redis:
image: "redis:alpine"
container_name: redis
I. create and start docker container by docker-compose => OK
$ docker-compose up --force-recreate -d
Creating redis ... done
II. recreate and start docker container by docker-compose => OK
$ docker-compose up --force-recreate -d
Recreating redis ... done
III. copy docker-compose.yml to other directory.
Then try to recreate from other directory => ERROR
$ cp docker-compose.yml red2/
$ cd red2/
$ docker-compose up --force-recreate -d
Creating redis ... error
ERROR: for redis Cannot create container for service redis: Conflict. The container name "/redis" is already in use by container "1ba060b545f716731ac1c5992b680e4d4b3639fc0ffeb291899c712f0839d23a". You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.
Different Networks
Containers created from docker-compose in different directories also do not share the same network.
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
4a4af52e89cd red2_default bridge local
57695428bd9d redis_default bridge local
Usecase
My usecase for that szenario:
Call docker-compose from different deployment jobs.
Start containers for testing
Questions
Why is there the directory dependency? Is there an option to switch it off?
Does docker ps show which directory was used?
Answer for 1:
The directory name is used as the default project name.
You should better specify the project name:
docker-compose -p myproject up --force-recreate -d
Question 2 still open
Related
Is there any proper way of restarting an entire docker compose stack from within one of its containers?
One workaround involves mounting the docker socket:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
and then use the Docker Engine SDKs (https://docs.docker.com/engine/api/sdk/examples/).
However, this solution only allows restarting the containers itselves. There seems to be no way to send compose commands, like docker compose restart, docker compose up, etc.
The only solution I've found to send docker compose commands is to open a terminal on the host from the container using ssh, like this: access host's ssh tunnel from docker container
This is partly related to How to run shell script on host from docker container? , but I'm actually looking for a more specific solution to only send docker compose commands.
I tried with this simple docker-compose.yml file
version: '3'
services:
nginx:
image: nginx
ports:
- 3000:80
Then I started a docker container using
docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/work docker
Then, inside the container, I did
cd /work
docker-compose up -d
and it started the container up on the host.
Please note that you have an error in your socket mapping. It needs to be
- /var/run/docker.sock:/var/run/docker.sock
(you have a period instead of a slash at one point)
As mentioned by #BMitch in the comments, compose project name was the reason why I wasn't able to run docker compose commands inside the running container.
By default the compose project name is set to the directory name, so if the docker-compose.yml is run from a host directory named folder1, then the commands inside the container should be run as:
docker-compose -p folder1 ...
So now, for example, restarting the stack works:
docker-compose -p folder1 restart
Just as a reference, a fixed project name for your compose can be set using name: ... as a top-level attribute of the .yml file, but requires docker compose v2.3.3 : Set $PROJECT_NAME in docker-compose file
I hope I didn't miss anything simple from the manual.
The structure is:
/home/user
/foo1/bar1/docker/
Dockerfile
docker-compose.yml
/foo2/bar2/docker/
Dockerfile
docker-compose.yml
docker-compose.yml
version: '3.9'
services:
foo1-bar1:
build:
context: .
args:
DOCKER_SERVER_ROOT_DIR: ${DOCKER_SERVER_ROOT_DIR}
dockerfile: Dockerfile
image: foo1-bar1:v1
container_name: foo1-bar1-v1
The same is for foo-bar-v2.
Both of them I successfully run as:
cd /foo1/bar1/docker/
docker-compose up -d
[+] Running 1/1
⠿ Container foo1-bar1-v1 Started
cd /foo2/bar2/docker/
docker-compose up -d
[+] Running 1/1
⠿ Container foo2-bar2-v1 Started
The question is, why does it stop both of them when I try to stop only 1? Service names, container names, image names are different...
user#vm:~/foo1/bar1/docker$ docker-compose stop
[+] Running 2/2
⠿ Container foo1-bar1-v1 Stopped
⠿ Container foo2-bar2-v2 Stopped
docker-compose has the concept of projects. Run docker-compose --help and you will see:
--project-directory string Specify an alternate working directory
(default: the path of the, first specified, Compose file)
-p, --project-name string Project name
So in your case, both your services belong to the same project named docker.
You can actually run docker-compose -p docker ps and you will see both your services.
You can also override this by specifying your own project name independent of the directory name.
My version of docker-compose (Docker Compose version v2.10.2 MacOS) does warn me that there are orphan containers in this project when I replicate your setup. Also it doesn't automatically stop "orphan" services and gives a warning that the network could not be removed either.
This is also another interesting fact: both services run on the same network (docker_default) only because the project name (folder name) is the same.
I hope this explains it.
You have to specify the service to stop. Otherwise it will stop all services.
docker compose stop [OPTIONS] [SERVICE...]
here : docker-compose stop foo1-bar1
I'm using docker-compose to start a job on a remote server:
version: "3.9"
services:
solution-app:
container_name: solution_new_name_v3
build: .
I ran the following command to deploy the container:
docker-compose -H "ssh://..." up -d --build
This works great, now I wanted to start another job in parallel using the same docker compose file, but with some minor changes in the source code. So I changed the container_name in the compose config and assumed that it should start a new container. However, instead of starting a new container my previous container was stopped and replaced with a new one with this new name.
So my question is: how can I start few containers running in parallel using same docker-compose config?
I figure out the way:
so Docker compose maps all services based on the project name, which is by default a directory name. To override the project name one can use:
docker-compose -p NAME
Alternatively one can set COMPOSE_PROJECT_NAME env variable
I have a large Docker project with Dockerfiles for nginx, apache2, varnish, redis configured and working well after weeks of changes and testing.
I am now at a point where I setup the projects to use docker-compose and override.yml files for easy setup:
I am trying to use the same docker-compose setup for multiple projects (websites)
Normal startup (using docker-compose.yml and optional docker-compose.override.yml)
docker-compose up -d
Custom startup (using specific docker-compose files)
docker-compose -f docker-compose.yml -f custom/docker-compose.website1.yml up -d
Both these methods starts up fine:
docker-compose ps
Ignore the fact that they are Exit 0 - I stopped them using docker-compose stop, the containers work fine
nginx-proxy /usr/bin/supervisord Exit 0
redis-cache /usr/bin/supervisord Exit 0
varnish-cache /usr/bin/supervisord Exit 0
web-server-apache2 /usr/bin/supervisord Exit 0
Now I want a second project (website) to use the same docker/docker-compose configuration setup:
docker-compose -f docker-compose.yml -f anothercustomfolder/docker-compose.website2.yml up -d
To my surprise docker-compose recreated containers and do not create a new set of containers:
See 'current setup' section for how I setup things.
Creating network "delete-network-frontend" with the default driver
Recreating nginx-proxy ... done
Recreating varnish-cache ... done
Recreating web-server ... done
Recreating redis-cache ... done
When running docker-compose ps in the second setup folder:
Note the names are not the same as above (this is the second test setup)
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy-delete /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
It appears docker-compose did two things : 1. Recreate (replace) the project 1 containers, used the project 1 container names to mention that they were 'recreated', and 2. Remove the project 1 containers, renamed it to project 2 containers.
Current setup
I created a full Dockerfile project configured with docker-compose.yml and two override docker-compose files (docker-compose.website1.yml and docker-compose.website2.yml`).
I made a complete copy of the working Dockerfile / docker-compose.yml project and created a new folder: In other words both these will use the same docker setup but use different docker-compose.yml override files.
/var/www/docker/site1
/var/www/docker/site2
Question
TLDR: How do I use a working docker-compose project on the same host operating system for multiple projects... without it replacing another project's containers.
I want to be able to see (use both) at the same time, and for instance be able to see this:
Ignore the fact that the ports are the same here, I am aware they won't run at the same time, I will update the project docker-compose.yml custom files when this works
docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
If anyone asks: Why not just put the websites into the same (one) container??
For the possibility someone might ask this, I know I can add multiple websites into the /etc/apache2/sites-enabled (or nginx) and add custom configuration files using ADD in Dockerfile for each site, but using that method I cannot test different slight setups.
Different setups that can be used by referencing another different image in the 'override docker-compose files'
For instance I can create a Dockerfile that installs all php7.3 libraries required and run Magento 2.3 on it, then have another Dockerfile to test php7.4, and have another to run an older Magento 1 site on a PHP5.6 installation and so on.
Thanks to advice from David Maze, I struggled further with configuring the docker-compose setup to work with multiple projects.
Information based on docker-compose v1.25.0 (July 2020)
This discussion is especially important when you want to re-use (persist) your containers (start/stop instead of just up/down - deleting)
As initially pointed out in my question - if you try to create containers using docker-compose up -d there are some pitfalls which the tool simply does not support right at the moment.
Pitfalls
PITFALLS OF CURRENT DOCKER-COMPOSE IMPLEMENTATION:
If you just use overridden docker-compose*.yml with different container_names (per 'project') with files in the same folder
docker-compose up will simply replace existing containers as explained in my question.
You can do the following: docker-compose -p CUSTOM_PROJECT_NAME -f file1.yml -f file2.yml up -d, but:
This on its own is useless - these containers will only work until you want stop them. As soon as you want to do docker-compose start (to restart existing container set) it will simply fail with Error: No containers to start
If you use two different folders with the same docker-compose project (ie cloned project): for instance ./dc-project1 ./dc-project2 but using container_name field inside docker-compose.*.yml file:
When you try to run docker-compose -f f1.yml -f f2.yml up -d inside ./dc-project1 and the same inside ./dc-project2 folder, you will get the following error: You have to remove (or rename) that container to be able to reuse that name.
Similar issues with your Docker network will occur with docker-compose when you use overridden files:
Removed most of the custom settings to make the network setting clearer:
Network will be attached correctly from your overridden file on docker-compose up, but as soon as you want to docker-compose start it looks for your default network name: in the default docker-compose.yml or even the docker-compose.override.yml file if it exists. In other words - it ignores your custom docker-compose override files (see example below):
docker-compose.yml:
networks:
network_frontend:
name: stage6-network-frontend
customfolder/docker-compose.custom.yml:
networks:
network_frontend:
name: magento2.3-network-frontend
SOLUTION
Example
Objective : to get docker-compose start/stop to work correctly with multiple setups (aka projects/websites/tools)using the same docker-compose project.
Suppose you have the following docker-compose files:
**Main file: ** docker-compose.yml:
web_server:
image: current_timezone/full-supervisord-web-server/php7.3:1.00
container_name: web-server-apache2
networks:
- network_frontend
build:
context: "./all-services/"
dockerfile: ./web-server/Dockerfile.webserver.apache2
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
name: stage6-network-frontend
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: docker-custom # Seems limit of 15 characters only
and then an override file: customfolder/magento2.override.yml:
web_server:
container_name: web-server-apache2-magento2.3.5
networks:
- network_frontend
build:
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: d-glo-femag2_35 # Seems limit of 15 characters only
name: glo-magento2.3-network-frontend
Do the following:
Copy the full Docker project (Dockerfiles/ADDs/docker-compose.yml files etc) into a new seperate folder:
/var/docker/project1
/var/docker/project2
Make sure that the container_name entries in your override docker-compose.yml are unique between the two projects.
In project project1 run docker-compose -f docker-compose.yml -f customfolder/magento2.override.yml up -d && docker-compose stop, navigate to project2 and do the same.
Using -p flag as David Maze suggested does not work on its own, JSON files are still sourced as ./foldername on docker-compose start/stop
Since networks are having similar issues on start/stop , before you can correctly use your custom name defined in your override file, .... unfortunately you need to update the main base docker-compose.yml to the overridden file!
Extended explanation: There is no way to call the correct custom network name from docker-compose start, so since docker-compose ignores the overridden files on start, you need to make sure to update the base file docker-compose.yml or docker-compose.override.yml has your custom network name.
In case you have not updated the names before using up -d, you will need to replace the content of each /var/lib/docker/containers/*/config.v2.json.
For example you could do this: you have to stop docker first
sudo service docker stop
find /var/lib/docker/containers/ -type f -name "config.v2.json" -exec sed -i "s|wrong-network-name|overridden-network-name|g" '{}' \;
sudo service docker start
IF done correctly, you should have unique container names, and each folder can be accessed separately correctly now without it breaking each other's containers: docker-compose start, docker-compose stop, docker-compose ps
NOTE: You still need to navigate to the seperate folder to run those commands
I am trying to run bamboo on server using docker containers. When i running on local machine work normally and volume save datas successfully. But when i run same docker compose file on server, volume data not save my datas.
docker-compose.yml
version: '3.2'
services:
bamboo:
container_name: bamboo-server_test
image: atlassian/bamboo-server
volumes:
- ./volumes/bamboo_test_vol:/var/atlassian/application-data/bamboo
ports:
- 8085:8085
volumes:
bamboo_test_vol:
Run this compose file on local machine
$ docker-compose up -d
Creating network "test_default" with the default driver
Creating volume "test_bamboo_test_vol" with default driver
Creating bamboo-server_test ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
916c98ca1a9d atlassian/bamboo-server "/entrypoint.sh" 24 minutes ago Up 24 minutes 0.0.0.0:8085->8085/tcp, 54663/tcp bamboo-server_test
$ ls
docker-compose.yml volumes
$ cd volumes/bamboo_test_vol/
$ ls
bamboo.cfg.xml logs
localhost:8085
Run this compose file on server
$ ssh <name>#<ip_address>
password for <name>:
$ docker-compose up -d
Creating network "test_default" with the default driver
Creating volume "test_bamboo_test_vol" with default driver
Creating bamboo-server_test ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38b77e1b736f atlassian/bamboo-server "/entrypoint.sh" 12 seconds ago Up 11 seconds 0.0.0.0:8085->8085/tcp, 54663/tcp bamboo-server_test
$ ls
docker-compose.yml volumes
$ cd volumes/
$ cd bamboo_test_vol/
$ ls
$ # VOLUME PATH IS EMPTY
server_ip:8085
I didn't have this problem when I tried the same process for jira-software. Why can't it work through the bamboo server even though I use the exact same compose file?
I had the same problem when I wanted to upgrade my Bamboo server instance with my mounted host volume for the bamboo-home directory.
The following was in my docker-compose file:
version: '2.2'
bamboo-server:
image: atlassian/bamboo-server:${BAMBOO_VERSION}
container_name: bamboo-server
environment:
TZ: 'Europe/Berlin'
restart: always
init: true
volumes:
- ./bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo
ports:
- "8085:8085"
- "54663:54663"
When i started with docker-compose up -d bamboo-server, the container never took the files from the host system. So I tried it first without docker-compose, following the instructions of Atlassian Bamboo with the following command:
docker run -v ./bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo --name="bamboo-server" --init -d -p 54663:54663 -p 8085:8085 atlassian/bamboo-server:${BAMBOO_VERSION}
The following error message was displayed:
docker: Error response from daemon: create ./bamboo/bamboo-server/data: "./bamboo/bamboo-server/data" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
So I converted the error message and took the absolute path:
docker run -v /var/project/bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo --name="bamboo-server" --init -d -p 54663:54663 -p 8085:8085 atlassian/bamboo-server:${BAMBOO_VERSION}
After the successful start, I switched to the docker container via SSH and all files were as usual in the docker directory.
I transferred the whole thing to the docker-compose file and took the absolute path in the volumes section. Subsequently it also worked with the docker-compose file.
My docker-compose file then looked like this:
[...]
init: true
volumes:
- /var/project/bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo
ports:
[...]
Setting up a containerized Bamboo Server is not supported for these reasons;
Repository-stored Specs (RSS) are no longer processed in Docker by default. Running RSS in Docker was not possible because;
there is no Docker capability added on the Bamboo server by default,
the setup would require running Docker in Docker.