I see lots of questions around setting/changing the COMPOSE_PROJECT_NAME or PROJECT_NAME using ENV variables.
I'm fine with the default project name, but I would like to reference it in my compose file.
version: "3.7"
services:
app:
build: DockerFile
container_name: app
volumes:
- ./:/var/app
networks:
- the-net
npm:
image: ${project_name}_app
volumes:
- ./:/var/app
depends_on:
- app
entrypoint: [ 'npm' ]
networks:
- the-net
npm here is arbitrary , hopefully the fact that could be run as its own container or in other ways does not distract from the questions.
is it possible to reference the project name with out setting it manually or first?
Unfortunately it is not possible.
As alluded to, you can create a .env file and populate it with COMPOSE_PROJECT_NAME=my_name, but the config option does not present itself in your environment by default.
Unfortunately the env substitution in docker-compose is fairly limited, meaning we cannot use the available PWD env variable and greedy match it at all
$ cd ~
$ pwd
/home/tqid
$ echo "Base Dir: ${PWD##*/}"
Base Dir: tqid
When we use this reference, compose has issues:
$ docker-compose up -d
ERROR: Invalid interpolation format for "image" option in service "demo": "${PWD##*/}"
It's probably better to be explicit anyway, the COMPOSE_PROJECT_NAME is based on your dir, and if someone clones to a new folder then it gets out of whack, including the .env file in source control would provide a re-usable and consistent place to reference the name
https://docs.docker.com/compose/reference/envvars/#compose_project_name
using the same image as another container was what I was after ... reuse the image and change the entry point.
Specify the same build: options for both containers.
This seems inefficient, in that it will trigger the build sequence twice and docker images will list both of them. However, the way Docker's layer caching works, if identical RUN commands are run on identical input images, the resulting layer will simply be reused, and the two final images will have the same image ID; they will literally be the same image with two names.
The context I've run into this the most is with a Python application where the same code base is used for a Django or Flask Web server, plus a Celery worker. The Docker-level setup is fairly language-independent, though: specify the same build: for both containers, and override the command: for the container(s) that need to do a non-default task.
version: '3.8'
services:
app:
build: .
ports: ['3000:3000']
environment:
REDIS_HOST: redis
worker:
build: . # <-- same as app
command: npm run worker # <-- overrides Dockerfile CMD
environment:
REDIS_HOST: redis
redis:
image: redis
It is also valid to specify build: and image: together in the docker-compose.yml file; this specifies the name of the image that will be built. It's frequently useful to explicitly specify this because you will need to point at a specific Docker Hub or other registry location to push the built image. If you do this, then you'll know the image name and don't need to depend on the context name.
version: '3.8'
services:
app:
build: .
image: registry.example.com/my/app:${TAG:-latest}
worker:
image: registry.example.com/my/app:${TAG:-latest}
command: npm run worker
You will need to manually docker-compose build in this setup. Compose's workflow doesn't have a way to specify that one container's build must run before a different container can start.
Related
I have three seperate Microservices, and for each of them in their directory I have Dockerfile.
I am beginner in Docker and I am a little confused.
for define Docker Compose file, I must define three docker-compose.yml files in the directory of each services?!
Or I must define just one docker-compose.yml file for all my services?! If yes, in which directory?
Docker compose is built for having multiple apps, with a Dockerfile it is very powerful.
To put it simply you can split a docker-compose file into things called 'services' and they act as different, separate apps/microservices, so say I wanted a nodejs app and a database within the same docker-compose file and Dockerfile:
Dockerfile:
FROM node:7.7.2-alpine
WORKDIR /usr/app
COPY package.json .
RUN npm install --quiet
COPY . .
Docker-compose:
version: '3.1'
services:
mongo:
image: mongo
name: database
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
web:
build: .
command: npm run dev
volumes:
- .:/usr/app/
- /usr/app/node_modules
ports:
- "3000:3000"
depends_on:
- mongo
If you ran that in the directory you want to work at, it will always stay in that directory . You can name each service it's own name. This example it's mongo and web. Once running, locally you can reference and connect to your services just by using their respective names.
I recommend these two YouTube video. Quick and simple. Here and here
You don't need to create separate compose file. Docker compose provides you the option to specify the location of Dockerfiles in order to setup the containers. In the root folder which contains this three app create a compose file.
For an example check this file https://github.com/dotnet-architecture/eShopOnContainers/blob/dev/src/docker-compose.yml
I am trying to use docker and want to create an Ubuntu base with three containers that do the following:
Container: Install Wildfly
Container: Install MySQL
Container: Other Required Packages
Does that mean, I have to create three Dockerfiles in three different directories containing each the following top line?:
FROM ubuntu:18.04
Orchestrate the containers with docker-compose
1- Create docker-compose.yml
2- Inside define:
version: '3'
services:
wildly:
build:
context: .
dockerfile: Dockerfile_Wildfly
mysql:
build:
context: .
dockerfile: Dockerfile_Mysql
anotherpackages:
build:
context: .
dockerfile: Dockerfile_AnotherPackages
it is not always the case that you need to write a docker file, for example for the database service you can simply pull the image from the docker hub and use it directly.
something like below
db:
image: mysql
3- Create files and both define commands you prefer:
Dockerfile_Wildfly
FROM wildfly
Dockerfile_Mysql
FROM mariadb
Dockerfile_AnotherPackages
FROM node
FROM nginx
You can create many Dockerfile and precise their name in the build command as suggested in another answer (#Krumelur's answer), but you can also use docker compose by calling directly the image from docker.io (if the base image for those dependecies in the hub match your neeed)
In this way you dont need any Dockerfile at all.
It should looks like this :
version: '3.3'
services:
wildfly:
#this image will be automatically downloaded from your registry (by default Docker hub)
image: jboss/wildfly
ports:
- '8080:8080'
- '9990:9990'
volumes:
- 'wildfly_data:/wildfly_data'
environment:
- WILDFLY_PASSWORD=password
mysql:
#this image will be automatically downloaded from your registry (by default Docker hub)
image: mysql:5.7
restart: always
environment:
MYSQL_DATABASE: 'db'
# So you don't have to use root, but you can if you like
MYSQL_USER: 'user'
# You can use whatever password you like
MYSQL_PASSWORD: 'password'
# Password for root access
MYSQL_ROOT_PASSWORD: 'password'
ports:
# <Port exposed> : < MySQL Port running inside container>
- '3306:3306'
expose:
# Opens port 3306 on the container
- '3306'
# Where our data will be persisted
volumes:
- my-db:/var/lib/mysql
otherService :
image: busybox
volumes:
my-db:
wildfly_data:
Then you just need to call the command : docker-compose up
You can have more than one Dockerfile in the same directory if desired. To specify the Dockerfile to use, use the -f argument, e.g
docker build -f wildfly.Dockerfile ./wildfly
docker build -f mysql.Dockerfile ./mysql
docker build -f other.Dockerfile ./other
In Compose, these arguments correspond to the dockerfile and context properties.
It is not always the case that you need to write a docker file, for example for the database service you can simply pull the image from the docker hub and use it directly.
something like below
db:
image: mysql
You can, of course, have them share the same context, e.g.
docker build -f wildfly.Dockerfile .
docker build -f mysql.Dockerfile .
docker build -f other.Dockerfile .
Just be aware that the context is sent in full to the daemon (respecting .dockerignore) so this might lead to longer build times if there is a lot of redundant data.
If there is a lot of reuse between the Dockerfiles, you can even have all of them in one file, e.g.
FROM ubuntu:20.04 as base
...
FROM base AS wildfly
(install wildfly)
FROM base AS mysql
(install mysql)
...
Then you can build the specific image with e.g.
docker build --target wildfly .
In Compose, these arguments correspond to the target and context properties.
This is called multi-stage builds, and is not always a good idea but is sometimes helpful to mitigate Docker's lack of support for #include.
Not sure if it will be a duplicate question but i tried to find out stuff but not sure if i have similar situation like others.
So i am new to docker and trying to setup a deployment for a small website.
So far i have a folder which has 3 files.
index.html - has basic html
Dockerfile - which has
FROM ubuntu:16.04
COPY . /var/www/html/
docker-compose.yml - which has
version: '2.1'
services:
app:
build: .
image: myname/myapp:1.0.0
nginx:
image: nginx
container_name: nginx
volumes:
- ./host-volumes:/cont-volumes
network_mode: "host"
phpfpm56:
image: php-fpm:5.6
container_name: phpfpm56
volumes:
- ./host-volumes:/cont-volumes
network_mode: "host"
mysql:
image: mysql:5.7
container_name: mysql
ports:
- "3306:3306"
volumes:
- mysql:/var/lib/mysql
volumes:
mysql:
Now i am using jenkins to create build, putting my all codes to host volumes to make it available to container and then i would run
docker-compose build
Now it creates an image and i push it to docker hub.
Then i login to remote server and pull the image and run. But that wont work because i still need to run docker-compose up inside the container.
Is this the right approach or i am missing something here?
The standard way to do this is to copy your code into the image. Do not bind-mount host folders containing your code; instead, use a Dockerfile COPY directive to copy in the application code (and in a compiled language, use a RUN command to build it). For example, your PHP container might have a corresponding Dockerfile that looks like (referencing this base Dockerfile)
FROM php-fpm:5.6
# Base Dockerfile defines a sensible WORKDIR
COPY . .
# Base Dockerfile sets EXPOSE 9000
# Base Dockerfile defines ENTRYPOINT, CMD
Then your docker-compose.yml would say, in part
version: '3'
service:
phpfpm56:
build: .
image: me/phpfpm56:2019-04-30
# No other settings
And then your nginx configuration would say, in part (using the Docker Compose service name as a hostname)
fastcgi_pass phpfpm56:9000
If you use this in production you need to comment out the build: lines I think.
If you're extremely set on a workflow where there is no hostname other than localhost and you do not need to rebuild Docker images to update code, you at least need to restart (some of) your containers after you've done the code push.
docker-compose stop app phpfpm56
docker-compose up -d
You might look into a system-automation tool like Ansible or Chef to automate the code-push mechanism. Those same tools can also just install nginx and PHP, and if you're trying to avoid the Docker image build sequence, you might have a simpler installation and deployment system running servers directly on the host.
docker-compose up should not be run inside a container but on a docker host. So this could be run via sh on a host but you need to have access to the composefile wherever you run the command.
Let's say we have the following docker-compose.yml:
version: '3'
services:
db:
image: "postgres"
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=mysecretpassword
web:
build: web
depends_on: [ db ]
ports:
- "80:80"
The first service, db, just runs a container with the official postgres image from Docker Hub.
The second service, web, first builds a new image based on the Dockerfile in a folder also called web, then runs a container with that image.
While developing, we now can (repeatedly) make changes to whatever is in the web folder, then run docker-compose up --build to run our app locally.
Let's say we now want to deploy to production. My understanding is that docker-compose.yml can now be used to "define a stack in Docker's swarm mode" (see this answer, for instance). However, for the build step of the web service, Docker's compose file documentation states that
This option is ignored when deploying a stack in swarm mode with a (version 3) Compose file. The docker stack command accepts only pre-built images.
It probably wouldn't be a great idea to build the image on the production machine anyways, as this would leave build artifacts (source code) behind; this should happen on a build server.
My question is, is there a recommended way to modify docker-compose.yml en route to production to swap out build: web with image: <id> somehow?
Nothing on Use Compose in production on that. Is there something wrong with my approach in general?
docker-compose.yml should only contain canonical service definitions.
Anything that's specific to the build environment (e.g. dev vs prod) should be declared in a separate file docker-compose.override.yml. Each build environment can have its own version of that file.
The build: web declaration doesn't belong into docker-compose.yml, as it's only supposed to run locally (and possibly on a build server), not in production.
Therefore, in the example above, this is what docker-compose.yml should look like:
version: '3'
services:
db:
image: "postgres"
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=mysecretpassword
web:
depends_on: [ db ]
ports:
- "80:80"
And this would be the default docker-compose.override.yml for local development:
version: '3'
services:
web:
build: web
Running docker-compose up --build -d will now build the latest code changes and launch our app locally.
There could also be another version docker-compose.override.build.yml, targeting a build/CI server:
version: '3'
services:
web:
build: web
image: mydockeruser/web
Running docker-compose -f docker-compose.yml -f docker-compose.override.build.yml push will build the latest code changes and push the image to its registry/repository.
Finally, there could be another version docker-compose.override.prod.yml:
version: '3'
services:
web:
image: mydockeruser/web
Deploying to production (just to a single Docker host, not a cluster) can now be as simple as copying over only docker-compose.yml and docker-compose.override.prod.yml and running docker-compose -f docker-compose.yml -f docker-compose.override.prod.yml up -d.
The correct way to do it (i.e. the way I do it :P) is to have different docker-compose files; for example, docker-compose.dev.yml and docker-compose.prod.yml. You can then push your production-ready image to a repository, say Docker Hub, and reference that image in docker-compose.prod.yml's web service. All the while you can use the dev docker-compose file (the one with the build option) for local development.
Also, in case you've thought about this, you cannot use env variables as keys in docker-compose (see here). So there is no way to conditionally set either image or build options.
How to be with orphan images when you have 2 independent projects and you want them to work at the same time or at least to build running docker-compose up -d without --remove-orphans flag when images are already built for another project.
docker compose file1:
version: '2'
services:
applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project1
nginx:
build: ./images/nginx
image: project1/nginx:latest
ports:
- "80:80"
volumes_from:
- applications
networks:
appnet:
aliases:
- project1.app
- admin.project1.app
php:
image: project1/php:latest
ports:
- "7778:7778"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- applications
networks:
- appnet
mysql:
image: project1/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- mysqldata:/var/lib/mysql
networks:
- appnet
ports:
- "33066:3306"
workspace:
image: project1/workspace:latest
build:
context: ./images/workspace
volumes_from:
- applications
working_dir: /var/www/vhosts/project1
networks:
- appnet
networks:
appnet:
driver: "bridge"
volumes:
mysqldata:
driver: "local"
the second docker compose file:
version: '2'
services:
project2_applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project2
project2_nginx:
build: ./images/nginx
image: project2/nginx:latest
ports:
- "8080:80"
volumes_from:
- project2_applications
networks:
project2_appnet:
aliases:
- project2.app
- admin.project2.app
project2_php:
image: project2/php:latest
ports:
- "7777:7777"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- project2_applications
networks:
- project2_appnet
project2_mysql:
image: project2/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- project2_mysqldata:/var/lib/mysql
networks:
- project2_appnet
ports:
- "33067:3306"
project2_workspace:
image: project2/workspace:latest
build:
context: ./images/workspace
volumes_from:
- project2_applications
working_dir: /var/www/vhosts/videosite
networks:
- project2_appnet
networks:
project2_appnet:
driver: "bridge"
volumes:
project2_mysqldata:
driver: "local"
And now when I have already built project1 and trying to run docker-compose up -d for the second project I see warning:
WARNING: Found orphan containers (docker_workspace_1, docker_nginx_1, docker_php_1, docker_mysql_1, docker_memcached_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
I have a supposition that it's because container names for project1 should be more specific and I need to add some prefixes like I'm doing for project2, but project1 is in use by many other developers and I do not want to change it.
Is there any way to turn off orphan check?
And the second thing: is just a warning message but for some reason, after it appearing compose is failing with error:
ERROR: Encountered errors while bringing up the project.
And to make it work I need to run docker-compose up -d --remove-orphans
Compose uses the project name (which defaults to the basename of the project directory) internally to isolate projects from each other. The project name is used to create unique identifiers for all of the project's containers and other resources. For example, if your project name is myapp and it includes two services db and web, then Compose starts containers named myapp_db_1 and myapp_web_1 respectively.
You get the "Found orphan containers" warning because docker-compose detects some containers which belong to another project with the same name.
To prevent different projects from interfering with each other (and suppress the warning) you can set a custom project name by using any of the following options:
The -p command line option.
COMPOSE_PROJECT_NAME environment variable. This environment variable can also be set via an environment file (.env in the current working directory by default).
Top-level name element in the Compose file. Note: if you pass multiple files to docker-compose via the -f option, then the value from the last file will be used.
docker-compose takes the name of the directory it is in as the default project name.
You can set a different project name by using -p or --project-name.
https://docs.docker.com/compose/reference/#use--p-to-specify-a-project-name
I had a similar problem because my projects all had the docker/docker-compose.yml structure.
To build on other answers, I create a .env file with my docker compose projects. I have a number of projects that all use the docker directory but are different projects.
To use docker-compose -p is a bit error prone, so creating .env file in the same directory as the docker-compose.yml:
-rw-rw-r-- 1 auser auser 1692 Aug 22 20:34 docker-compose.yml
-rw-rw-r-- 1 auser auser 31 Aug 22 20:44 .env
alleviates the necessary overhead of remembering -p.
In the .env file, I can now set the COMPOSE_PROJECT_NAME variable:
COMPOSE_PROJECT_NAME=myproject
On running:
docker-compose up -d
the COMPOSE_PROJECT_NAME is substituted without the use of -p.
Reference:
https://docs.docker.com/compose/env-file/
docker-compose up --remove-orphans
you can run this command to clean orphan containers. As specified in the warning
If the orphaned containers are expected and not intended to remove, you can set COMPOSE_IGNORE_ORPHANS variable to true.
Consise but just right away working source is here.
One option is to put it as a line into .env file next to docker-compose.yml like this:
COMPOSE_IGNORE_ORPHANS=True
Another option is pass or set it as an environment variable.
sh:
COMPOSE_IGNORE_ORPHANS=True docker-compose up -d
or
export COMPOSE_IGNORE_ORPHANS=True
docker-compose up -d
cmd:
SET COMPOSE_IGNORE_ORPHANS=True&& docker-compose up -d
powershell:
$env:COMPOSE_IGNORE_ORPHANS = 'True'; & docker-compose up -d
TL;DR
You can also add a unique name: myproject to each of your compose files.
My journey
In case this helps anybody else scrounging around to find help for the above issue (This is in support of the already good comments here):
I have several config files in the same directory
redis.yml
mariadb.yml
...
and I kept getting the same error about orphan containers when I ran
docker-compose -f <one of my configs>.yml up
as of now you can simply put each yml file into a separate project. This is simply done using the command like parameter "-p my_project_name" as has already been mentioned before. BUT the name must be in all lowercase!
This got me a little closer but I also kept forgetting that to bring the docker container down using docker-compose I needed to include that parameter as well.
For example to start the container:
docker-compose -p myproject-d redis.yml up -d
and to destroy the container
docker-compose -p myproject-d redis.yml down
Today I found that I can simply add the name: bit into the yml config. Here is an example for redis:
version: '3.9'
name: redis
services:
redis_0:
...
Now I can simply start the container with the following and don't have to worry about project names again:
docker-compose -f redis.yml <up/down>
This happens when your docker-compose file has got updated. I received similar error on Docker startup and found out that another team member updated the docker-compose.yml as part of cleanup.
To fix this, I deleted the docker group using the Delete button in Docker Desktop and started it again. This fixed the error for me.
As a complement for the existing answers, if you're using docker-compose with the -f option, to my surprise docker-compose will use the name of the parent folder of the first file passed via -f as the project name.
For example, assuming the following folder structure:
/
└── Users/
└── papb/
├── a.yml
└── foo/
└── b.yml
If you're in /Users and run docker-compose -f papb/a.yml -f papb/foo/b.yml:
The project name will be inferred as papb
Any relative paths you have in both files will be resolved against /Users/papb
If you're in /Users and run docker-compose -f papb/foo/b.yml -f papb/a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo
If you're in /Users/papb and run docker-compose -f foo/b.yml -f a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo