I have an issue. I used my docker-compose file for one project. Then I copied it to another directory in order to run another containers. But whenever I do that it recreates existing containers or in case I use the down command it also destroys containers from another directory, what could be wrong?
Here is my configuration.
version: '3.5'
services:
postgres:
image: postgres
environment:
POSTGRES_USER: user
POSTGRES_PASSWORD: password
PGDATA: /data/postgres
volumes:
- postgres:/data/postgres
ports:
- "5440:5432"
networks:
- postgres
restart: unless-stopped
pgadmin:
image: dpage/pgadmin4
environment:
PGADMIN_DEFAULT_EMAIL: admin#admin.com
PGADMIN_DEFAULT_PASSWORD: rootme
volumes:
- pgadmin:/root/.pgadmin
ports:
- "8440:80"
networks:
- postgres
restart: unless-stopped
networks:
postgres:
driver: bridge
volumes:
postgres:
pgadmin:
For example when I run docker-compose up -d from another directory it recreates containers
Recreating docker_postgres_1 ... done
Recreating docker_pgadmin_1 ... done
What is the issue?
Docker Compose attaches a name prefix to everything it creates, but the default prefix is just based on the basename of the current directory. If you have a layout like
projectA
+-- docker
| \-- docker-compose.yml
projectB
\-- docker
\-- docker-compose.yml
then both docker-compose instances will think the project name is just docker (the name of the directory containing docker-compose.yml) and create container names like, for example, docker_postgres_1.
You can get around this by either renaming one of the directories, using the docker-compose -p option, or setting a COMPOSE_PROJECT_NAME environment variable. It wouldn't be unusual to see a docker-compose.yml file in the top-level directory of a project and that might help disambiguate things.
I believe that it occurs because of the containers on the previous project continue running while you try up the new project.
Try to stop the containers on the previous folder before upon the current.
Related
I have been trying to install drupal using the official image from docker hub. I created a new folder in my D directory, for my Drupal project and created a docker-compose.yml file.
Drupal with PostgreSQL
Access via "http://localhost:8080"
(or "http://$(docker-machine ip):8080" if using docker-machine)
During initial Drupal setup,
Database type: PostgreSQL
Database name: postgres
Database username: postgres
Database password: example
ADVANCED OPTIONS; Database host: postgres
version: '3.1' services:
drupal:
image: drupal:8-apache ports:
- 8080:80
volumes:
- /var/www/html/modules
- /var/www/html/profiles
- /var/www/html/themes
this takes advantage of the feature in Docker that a new anonymous
volume (which is what we're creating here) will be initialized with the
existing content of the image at the same location
- /var/www/html/sites
restart: always
postgres:
image: postgres:10
environment:
POSTGRES_PASSWORD: example
restart: always
When I ran the docker-compose up -d command in a terminal from within the folder which constrong texttained docker-compose.yml file, my drupal container and its databse were successfully installed and running and I was able to access the site from http://localhost:8080 but I couldnt find their core files in the folder. It was just docker-compose.yml file in the folder.
I then removed the whole docker container and began with a fresh installation again with by editing the volume section in the docker-compose.yml file to point to the directory and folder where I want the core files of drupal to be populated.
Example D:/My Project/Drupal Project.
Drupal with PostgreSQL
Access via "http://localhost:8080"
(or "http://$(docker-machine ip):8080" if using docker-machine)
During initial Drupal setup,
Database type: PostgreSQL
Database name: postgres
Database username: postgres
Database password: example
ADVANCED OPTIONS; Database host: postgres
version: '3.1'
services:
drupal:
image: drupal:latest
ports:
- 8080:80
volumes:
- d:\projects\drupalsite/var/www/html/modules
- d:\projects\drupalsite/var/www/html/profiles
- d:\projects\drupal/var/www/html/themes
this takes advantage of the feature in Docker that a new anonymous
volume (which is what we're creating here) will be initialized with the
existing content of the image at the same location
- d:\projects\drupalsite/var/www/html/sites
restart: always
postgres:
image: postgres:10
environment:
POSTGRES_PASSWORD: example
restart: always
When I ran the docker-compose.yml command I received the error as shown below.
Container drupalsite_postgres_1 Created 3.2s
- Container drupalsite_drupal_1 Creating 3.2s
Error response from daemon: invalid mount config for type "volume": invalid mount path: 'z:/projects/drupalsite/var/www/html/sites' mount path must be absolute
PS Z:\Projects\drupalsite>
Please help me find a solution to this.
If these directories contain your application, they probably shouldn't be in volumes: at all. Create a file named Dockerfile that initializes your custom application:
FROM drupal:8-apache
COPY modules/ /var/www/html/modules/
COPY profiles/ /var/www/html/profiles/
COPY themes/ /var/www/html/themes/
COPY sites/ /var/www/html/sites/
# EXPOSE, CMD, etc. come from the base image
Then reference this in your docker-compose.yml file:
version: '3.8'
services:
drupal:
build: . # instead of image:
ports:
- 8080:80
restart: always
# no volumes:
postgres:
image: postgres:10
environment:
POSTGRES_PASSWORD: example
restart: always
volumes:
- pgdata:/var/lib/postgresql/data
volumes:
pgdata:
If you really want to use volumes: here, there are three forms of that directive. The form you have in the question with just a path creates an anonymous volume: it causes Compose to persist that directory, initialized from what's in the image, but disconnected from your host system. With a bare name and a path, it creates a named volume, which is similar but can be explicitly managed. With two paths, it creates a bind mount, which unconditionally replaces the container content with the host-system content (there is no initialization).
version: '3.8'
services:
something:
volumes:
- /path1 # anonymous volume
- named:/path2 # named volume
- /host/path:/path3 # bind mount
volumes: # named volumes referenced in containers only
named: # usually do not need any settings
So if you do want to replace the image's contents with host directories, you need to use the bind-mount syntax. Relative paths here are interpreted relative to the location of the docker-compose.yml file.
version: '3.8'
services:
drupal:
image: drupal:8-apache
volumes:
- ./modules:/var/www/html/modules
# etc.
A final comment on named volume initialization: your file has a comment about initializing anonymous volumes. There are two major problems with this approach, though. First, the second time you start the container, the content of the volume takes precedence, and any changes in the underlying images will be ignored. Second, this setup only works for Docker named and anonymous volumes, but not Docker bind mounts, volume mounts in Kubernetes, or other types of mount. I'd generally avoid relying on this "feature".
I have two Rails 6 application and I am trying to deploy in aws ec2 instance with different port 8080 and 8081 but when I trying to run docker-compose up -d it start one rails application successfully and if I tries to run docker-compose up -d for second application, It make first application down and make another application up on particular Port
Below is my docker configuration for two applications.
Application 1
version: "3.4"
services:
app:
image: "dockerhub_repo/a_api:${TAG}"
# build:
# context: .
# dockerfile: Dockerfile
container_name: a_api_container
depends_on:
- database
- redis
- sidekiq
ports:
- "8080:8080"
volumes:
- .:/app
env_file: .env
environment:
RAILS_ENV: staging
database:
image: postgres:12.1
container_name: a_database_container
restart: always
volumes:
- db_data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
sidekiq:
image: "dockerhub_repo/a_api:${STAG}"
container_name: a_sidekiq_container
environment:
RAILS_ENV: staging
env_file: .env
depends_on:
- redis
volumes:
- ".:/app"
redis:
image: redis:4.0-alpine
container_name: a_redis_container
volumes:
- "redis:/data"
volumes:
redis:
db_data:
Application 2
version: "3.4"
services:
app:
image: "dockerhub_repo/b_api:${PPTAG}"
build:
context: .
dockerfile: Dockerfile
container_name: b_api
depends_on:
- database
- redis
ports:
- "8081:8081"
volumes:
- .:/app
env_file: .env
environment:
RAILS_ENV: development
database:
image: postgres:12.1
container_name: pp_database
restart: always
volumes:
- db_data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
redis:
image: redis:4.0-alpine
container_name: pp_redis
volumes:
db_data:
This Configuration works very well in local machine. It start both application in local on different port but it has some issue on aws ec2. I am not sure is any thing wrong in configuration?
Compose has the notion of a project name. If you add or delete containers from a docker-compose.yml file, it looks for existing containers that are labeled with the project name to figure out what needs to change. The project name is also included in the Docker names of containers, networks, and volumes.
You can configure the project name with the COMPOSE_PROJECT_NAME environment variable or the docker-compose -p option. If you don't configure it, it defaults to the base name of the current directory.
You clarify in a comment that the two docker-compose.yml files are in directories app1/backend and app2/backend. Since the base name of those directories are both backend, they have the same project name; so if you run docker-compose up in the app2/backend directory, it finds the existing containers for the backend project, sees they don't match what's in the docker-compose.yml file, and deletes them (even though you as the operator think they belong to the other project).
There are a couple of ways to get around this:
Rename one or the other directory; maybe move the docker-compose.yml files up to the top-level app1 and app2 directories.
In one or both directories, create a .env file that sets COMPOSE_PROJECT_NAME=app1. (Note that file is checked in the current directory, not necessarily the directory that contains the docker-compose.yml file.)
Set and change an environment variable export COMPOSE_PROJECT_NAME=app1.
Consistently use an option docker-compose -p app1 ... with all Compose commands.
for some reason, I need to create the container with the same image, But when I started the second one, It just restarted the fist one's container
the first yml file:
version: "3.1"
services:
php:
image:php:php73-fpm
restart: always
ports:
- "9000:9000"
- "9501:9501"
volumes:
- $PWD/../:/var/www/html/
networks:
- app_net
container_name: php
networks:
app_net:
driver: bridge
the second yml file:
version: "3.1"
services:
php:
image:php:php73-fpm
restart: always
ports:
- "19000:19000"
- "19501:19501"
volumes:
- $PWD/../:/var/www/html/
networks:
- app_net2
container_name: php73
networks:
app_net2:
driver: bridge
when I run docker-compose up -d to start the first one:
$ cd ~/Document/php/work/docker/
$ docker-compose up -d
Creating network "docker_app_net" with driver "bridge"
Creating php ... done
then I switch the directory to the second yml file
$ cd ../../private/docker/
$ docker-compose up -d
Recreating php ... done
Compose has a notion of a project name. By default the project name is the basename of the directory containing the docker-compose.yml file. In your example both directories are named docker (even if they're in different parent directories) so Compose looks for a project named docker and a container named php, and finds a match.
There are four ways to override this:
Rename one of the directories.
Set the COMPOSE_PROJECT_NAME environment variable.
Create a .env file in the current directory, and set COMPOSE_PROJECT_NAME there.
Use the docker-compose -p option (on every docker-compose command).
Within your docker-compose.yml file, the second part of ports: needs to match what the container is listening on; this is allowed to be different from the first part. So use the same 9500/9501 in both files.
Another consequence of the Compose project naming is that the standard names of containers, volumes, and networks that Compose creates will be prefixed with the project name. If the project name (current directory name) is docker2, and you reduce the Compose file to
version: "3.1"
services:
php:
build: .
restart: always
ports:
- "19000:9000"
- "19501:9501"
# no manual container_name: or networks:
The container will be named docker2_php_1, and it will be attached to a network named docker2_default; these will be different from the container/network created in the docker1 project/directory.
You can't have two containers with the same name. Since both names are just php, Docker thought they were settings that were supposed to be merged for the same container. Rename one of them.
I am trying to allow nginx to proxy between multiple containers while also accessing the static files from those containers.
To share volumes between containers created using docker compose, the following works correctly:
version: '3.6'
services:
web:
build:
context: .
dockerfile: ./Dockerfile
image: webtest
command: ./start.sh
volumes:
- .:/code
- static-files:/static/teststaticfiles
nginx:
image: nginx:1.15.8-alpine
ports:
- "80:80"
volumes:
- ./nginx-config:/etc/nginx/conf.d
- static-files:/static/teststaticfiles
depends_on:
- web
volumes:
static-files:
However what I actually require is for the nginx compose file to be in a separate file and also in a completely different folder. In other words, the docker compose up commands would be run separately. I have tried the following:
First compose file:
version: '3.6'
services:
web:
build:
context: .
dockerfile: ./Dockerfile
image: webtest
command: ./start.sh
volumes:
- .:/code
- static-files:/static/teststaticfiles
networks:
- directorylocation-nginx_mynetwork
volumes:
static-files:
networks:
directorylocation-nginx_mynetwork:
external: true
Second compose file (ie: nginx):
version: '3.6'
services:
nginx:
image: nginx:1.15.8-alpine
ports:
- "80:80"
volumes:
- ./nginx-config:/etc/nginx/conf.d
- static-files:/static/teststaticfiles
networks:
- mynetwork
volumes:
static-files:
networks:
mynetwork:
The above two files work correctly in the sense that the site can be viewed. The problem is that the static files are not available in the nginx container. The site therefore displays without any images etc.
One work around which works correctly found here is to change the nginx container static files volume to instead be as follows:
- /var/lib/docker/volumes/directory_static-files/_data:/static/teststaticfiles
The above works correctly, but it seems 'hacky' and brittle. Is there another way to share volumes between containers which are housed in different compose files without needing to map the /var/lib/docker/volumes directory.
By separating the 2 docker-compose.yml files as you did in your question, 2 different volumes are actually created; that's the reason you don't see data from web service inside volume of nginx service, because there are just 2 different volumes.
Example : let's say you have the following structure :
example/
|- web/
|- docker-compose.yml # your first docker compose file
|- nginx/
|- docker-compose.yml # your second docker compose file
Running docker-compose up from web folder (or docker-compose -f web/docker-compose.yml up from example directory) will actually create a volume named web_static-files (name of the volume defined in docker-compose.yml file, prefixed by the folder where this file is located).
So, running docker-compose up from nginx folder will actually create nginx_static-files instead of re-using web_static-files as you want.
You can use the volume created by web/docker-compose.yml by specifying in the 2nd docker compose file (nginx/docker-compose.yml) that this is an external volume, and its name :
volumes:
static-files:
external:
name: web_static-files
Note that if you don't want the volume (and all resources) to be prefixed by the folder name (default), but by something else, you can add -p option to docker-compose command :
docker-compose \
-f web/docker-compose.yml \
-p abcd \
up
This command will now create a volume named abcd_static-files (that you can use in the 2nd docker compose file).
You can also define the volumes creation on its own docker-compose file (like volumes/docker-compose.yml) :
version: '3.6'
volumes:
static-files:
And reference this volume as external, with name volumes_static-files, in web and nginx docker-compose.yml files :
volumes:
volumes_static-files:
external: true
Unfortunately, you cannot set the volume name in docker compose, it will be automatically prefixed. If this is really a problem, you can also create the volume manually (docker volume create static-files) before running any docker-compose up command (I do not recommand this solution though because it adds a manual step that can be forgotten if you reproduce your deployment on another environment).
How to be with orphan images when you have 2 independent projects and you want them to work at the same time or at least to build running docker-compose up -d without --remove-orphans flag when images are already built for another project.
docker compose file1:
version: '2'
services:
applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project1
nginx:
build: ./images/nginx
image: project1/nginx:latest
ports:
- "80:80"
volumes_from:
- applications
networks:
appnet:
aliases:
- project1.app
- admin.project1.app
php:
image: project1/php:latest
ports:
- "7778:7778"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- applications
networks:
- appnet
mysql:
image: project1/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- mysqldata:/var/lib/mysql
networks:
- appnet
ports:
- "33066:3306"
workspace:
image: project1/workspace:latest
build:
context: ./images/workspace
volumes_from:
- applications
working_dir: /var/www/vhosts/project1
networks:
- appnet
networks:
appnet:
driver: "bridge"
volumes:
mysqldata:
driver: "local"
the second docker compose file:
version: '2'
services:
project2_applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project2
project2_nginx:
build: ./images/nginx
image: project2/nginx:latest
ports:
- "8080:80"
volumes_from:
- project2_applications
networks:
project2_appnet:
aliases:
- project2.app
- admin.project2.app
project2_php:
image: project2/php:latest
ports:
- "7777:7777"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- project2_applications
networks:
- project2_appnet
project2_mysql:
image: project2/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- project2_mysqldata:/var/lib/mysql
networks:
- project2_appnet
ports:
- "33067:3306"
project2_workspace:
image: project2/workspace:latest
build:
context: ./images/workspace
volumes_from:
- project2_applications
working_dir: /var/www/vhosts/videosite
networks:
- project2_appnet
networks:
project2_appnet:
driver: "bridge"
volumes:
project2_mysqldata:
driver: "local"
And now when I have already built project1 and trying to run docker-compose up -d for the second project I see warning:
WARNING: Found orphan containers (docker_workspace_1, docker_nginx_1, docker_php_1, docker_mysql_1, docker_memcached_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
I have a supposition that it's because container names for project1 should be more specific and I need to add some prefixes like I'm doing for project2, but project1 is in use by many other developers and I do not want to change it.
Is there any way to turn off orphan check?
And the second thing: is just a warning message but for some reason, after it appearing compose is failing with error:
ERROR: Encountered errors while bringing up the project.
And to make it work I need to run docker-compose up -d --remove-orphans
Compose uses the project name (which defaults to the basename of the project directory) internally to isolate projects from each other. The project name is used to create unique identifiers for all of the project's containers and other resources. For example, if your project name is myapp and it includes two services db and web, then Compose starts containers named myapp_db_1 and myapp_web_1 respectively.
You get the "Found orphan containers" warning because docker-compose detects some containers which belong to another project with the same name.
To prevent different projects from interfering with each other (and suppress the warning) you can set a custom project name by using any of the following options:
The -p command line option.
COMPOSE_PROJECT_NAME environment variable. This environment variable can also be set via an environment file (.env in the current working directory by default).
Top-level name element in the Compose file. Note: if you pass multiple files to docker-compose via the -f option, then the value from the last file will be used.
docker-compose takes the name of the directory it is in as the default project name.
You can set a different project name by using -p or --project-name.
https://docs.docker.com/compose/reference/#use--p-to-specify-a-project-name
I had a similar problem because my projects all had the docker/docker-compose.yml structure.
To build on other answers, I create a .env file with my docker compose projects. I have a number of projects that all use the docker directory but are different projects.
To use docker-compose -p is a bit error prone, so creating .env file in the same directory as the docker-compose.yml:
-rw-rw-r-- 1 auser auser 1692 Aug 22 20:34 docker-compose.yml
-rw-rw-r-- 1 auser auser 31 Aug 22 20:44 .env
alleviates the necessary overhead of remembering -p.
In the .env file, I can now set the COMPOSE_PROJECT_NAME variable:
COMPOSE_PROJECT_NAME=myproject
On running:
docker-compose up -d
the COMPOSE_PROJECT_NAME is substituted without the use of -p.
Reference:
https://docs.docker.com/compose/env-file/
docker-compose up --remove-orphans
you can run this command to clean orphan containers. As specified in the warning
If the orphaned containers are expected and not intended to remove, you can set COMPOSE_IGNORE_ORPHANS variable to true.
Consise but just right away working source is here.
One option is to put it as a line into .env file next to docker-compose.yml like this:
COMPOSE_IGNORE_ORPHANS=True
Another option is pass or set it as an environment variable.
sh:
COMPOSE_IGNORE_ORPHANS=True docker-compose up -d
or
export COMPOSE_IGNORE_ORPHANS=True
docker-compose up -d
cmd:
SET COMPOSE_IGNORE_ORPHANS=True&& docker-compose up -d
powershell:
$env:COMPOSE_IGNORE_ORPHANS = 'True'; & docker-compose up -d
TL;DR
You can also add a unique name: myproject to each of your compose files.
My journey
In case this helps anybody else scrounging around to find help for the above issue (This is in support of the already good comments here):
I have several config files in the same directory
redis.yml
mariadb.yml
...
and I kept getting the same error about orphan containers when I ran
docker-compose -f <one of my configs>.yml up
as of now you can simply put each yml file into a separate project. This is simply done using the command like parameter "-p my_project_name" as has already been mentioned before. BUT the name must be in all lowercase!
This got me a little closer but I also kept forgetting that to bring the docker container down using docker-compose I needed to include that parameter as well.
For example to start the container:
docker-compose -p myproject-d redis.yml up -d
and to destroy the container
docker-compose -p myproject-d redis.yml down
Today I found that I can simply add the name: bit into the yml config. Here is an example for redis:
version: '3.9'
name: redis
services:
redis_0:
...
Now I can simply start the container with the following and don't have to worry about project names again:
docker-compose -f redis.yml <up/down>
This happens when your docker-compose file has got updated. I received similar error on Docker startup and found out that another team member updated the docker-compose.yml as part of cleanup.
To fix this, I deleted the docker group using the Delete button in Docker Desktop and started it again. This fixed the error for me.
As a complement for the existing answers, if you're using docker-compose with the -f option, to my surprise docker-compose will use the name of the parent folder of the first file passed via -f as the project name.
For example, assuming the following folder structure:
/
└── Users/
└── papb/
├── a.yml
└── foo/
└── b.yml
If you're in /Users and run docker-compose -f papb/a.yml -f papb/foo/b.yml:
The project name will be inferred as papb
Any relative paths you have in both files will be resolved against /Users/papb
If you're in /Users and run docker-compose -f papb/foo/b.yml -f papb/a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo
If you're in /Users/papb and run docker-compose -f foo/b.yml -f a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo