Docker - Reuse docker-compose configurations for different projects - docker

Question

I have a large Docker project with Dockerfiles for nginx, apache2, varnish, redis configured and working well after weeks of changes and testing.
I am now at a point where I setup the projects to use docker-compose and override.yml files for easy setup:
I am trying to use the same docker-compose setup for multiple projects (websites)
Normal startup (using docker-compose.yml and optional docker-compose.override.yml)
docker-compose up -d
Custom startup (using specific docker-compose files)
docker-compose -f docker-compose.yml -f custom/docker-compose.website1.yml up -d
Both these methods starts up fine:
docker-compose ps
Ignore the fact that they are Exit 0 - I stopped them using docker-compose stop, the containers work fine
nginx-proxy /usr/bin/supervisord Exit 0
redis-cache /usr/bin/supervisord Exit 0
varnish-cache /usr/bin/supervisord Exit 0
web-server-apache2 /usr/bin/supervisord Exit 0
Now I want a second project (website) to use the same docker/docker-compose configuration setup:
docker-compose -f docker-compose.yml -f anothercustomfolder/docker-compose.website2.yml up -d
To my surprise docker-compose recreated containers and do not create a new set of containers:
See 'current setup' section for how I setup things.
Creating network "delete-network-frontend" with the default driver
Recreating nginx-proxy ... done
Recreating varnish-cache ... done
Recreating web-server ... done
Recreating redis-cache ... done
When running docker-compose ps in the second setup folder:
Note the names are not the same as above (this is the second test setup)
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy-delete /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
It appears docker-compose did two things : 1. Recreate (replace) the project 1 containers, used the project 1 container names to mention that they were 'recreated', and 2. Remove the project 1 containers, renamed it to project 2 containers.
Current setup
I created a full Dockerfile project configured with docker-compose.yml and two override docker-compose files (docker-compose.website1.yml and docker-compose.website2.yml`).
I made a complete copy of the working Dockerfile / docker-compose.yml project and created a new folder: In other words both these will use the same docker setup but use different docker-compose.yml override files.
/var/www/docker/site1
/var/www/docker/site2
Question
TLDR: How do I use a working docker-compose project on the same host operating system for multiple projects... without it replacing another project's containers.
I want to be able to see (use both) at the same time, and for instance be able to see this:
Ignore the fact that the ports are the same here, I am aware they won't run at the same time, I will update the project docker-compose.yml custom files when this works
docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2
nginx-proxy /usr/bin/supervisord Up 0.0.0.0:443->443/tcp,
0.0.0.0:80->80/tcp
redis-cache-delete /usr/bin/supervisord Up 0.0.0.0:6379->6379/tcp
varnish-cache-delete /usr/bin/supervisord Up 0.0.0.0:6081->6081/tcp,
0.0.0.0:6082->6082/tcp
web-server- /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp
apache2-delete
If anyone asks: Why not just put the websites into the same (one) container??
For the possibility someone might ask this, I know I can add multiple websites into the /etc/apache2/sites-enabled (or nginx) and add custom configuration files using ADD in Dockerfile for each site, but using that method I cannot test different slight setups.
Different setups that can be used by referencing another different image in the 'override docker-compose files'
For instance I can create a Dockerfile that installs all php7.3 libraries required and run Magento 2.3 on it, then have another Dockerfile to test php7.4, and have another to run an older Magento 1 site on a PHP5.6 installation and so on.

Answer 1

Thanks to advice from David Maze, I struggled further with configuring the docker-compose setup to work with multiple projects.
Information based on docker-compose v1.25.0 (July 2020)
This discussion is especially important when you want to re-use (persist) your containers (start/stop instead of just up/down - deleting)
As initially pointed out in my question - if you try to create containers using docker-compose up -d there are some pitfalls which the tool simply does not support right at the moment.
Pitfalls
PITFALLS OF CURRENT DOCKER-COMPOSE IMPLEMENTATION:
If you just use overridden docker-compose*.yml with different container_names (per 'project') with files in the same folder
docker-compose up will simply replace existing containers as explained in my question.
You can do the following: docker-compose -p CUSTOM_PROJECT_NAME -f file1.yml -f file2.yml up -d, but:
This on its own is useless - these containers will only work until you want stop them. As soon as you want to do docker-compose start (to restart existing container set) it will simply fail with Error: No containers to start
If you use two different folders with the same docker-compose project (ie cloned project): for instance ./dc-project1 ./dc-project2 but using container_name field inside docker-compose.*.yml file:
When you try to run docker-compose -f f1.yml -f f2.yml up -d inside ./dc-project1 and the same inside ./dc-project2 folder, you will get the following error: You have to remove (or rename) that container to be able to reuse that name.
Similar issues with your Docker network will occur with docker-compose when you use overridden files:
Removed most of the custom settings to make the network setting clearer:
Network will be attached correctly from your overridden file on docker-compose up, but as soon as you want to docker-compose start it looks for your default network name: in the default docker-compose.yml or even the docker-compose.override.yml file if it exists. In other words - it ignores your custom docker-compose override files (see example below):
docker-compose.yml:
networks:
network_frontend:
name: stage6-network-frontend
customfolder/docker-compose.custom.yml:
networks:
network_frontend:
name: magento2.3-network-frontend
SOLUTION
Example
Objective : to get docker-compose start/stop to work correctly with multiple setups (aka projects/websites/tools)using the same docker-compose project.
Suppose you have the following docker-compose files:
**Main file: ** docker-compose.yml:
web_server:
image: current_timezone/full-supervisord-web-server/php7.3:1.00
container_name: web-server-apache2
networks:
- network_frontend
build:
context: "./all-services/"
dockerfile: ./web-server/Dockerfile.webserver.apache2
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
name: stage6-network-frontend
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: docker-custom # Seems limit of 15 characters only
and then an override file: customfolder/magento2.override.yml:
web_server:
container_name: web-server-apache2-magento2.3.5
networks:
- network_frontend
build:
args:
volumes:
- website_data:/var/www/html
ports:
- "8080:8080"
networks:
network_frontend:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.100.0.0/16
driver_opts:
# Custom name for host-side network : for instance on Ubuntu : ip addr | ifconfig
com.docker.network.bridge.name: d-glo-femag2_35 # Seems limit of 15 characters only
name: glo-magento2.3-network-frontend
Do the following:
Copy the full Docker project (Dockerfiles/ADDs/docker-compose.yml files etc) into a new seperate folder:
/var/docker/project1
/var/docker/project2
Make sure that the container_name entries in your override docker-compose.yml are unique between the two projects.
In project project1 run docker-compose -f docker-compose.yml -f customfolder/magento2.override.yml up -d && docker-compose stop, navigate to project2 and do the same.
Using -p flag as David Maze suggested does not work on its own, JSON files are still sourced as ./foldername on docker-compose start/stop
Since networks are having similar issues on start/stop , before you can correctly use your custom name defined in your override file, .... unfortunately you need to update the main base docker-compose.yml to the overridden file!
Extended explanation: There is no way to call the correct custom network name from docker-compose start, so since docker-compose ignores the overridden files on start, you need to make sure to update the base file docker-compose.yml or docker-compose.override.yml has your custom network name.
In case you have not updated the names before using up -d, you will need to replace the content of each /var/lib/docker/containers/*/config.v2.json.
For example you could do this: you have to stop docker first
sudo service docker stop
find /var/lib/docker/containers/ -type f -name "config.v2.json" -exec sed -i "s|wrong-network-name|overridden-network-name|g" '{}' \;
sudo service docker start
IF done correctly, you should have unique container names, and each folder can be accessed separately correctly now without it breaking each other's containers: docker-compose start, docker-compose stop, docker-compose ps
NOTE: You still need to navigate to the seperate folder to run those commands