We have docker-compose-dev.yaml, docker-compose-staging.yaml, docker-compose-prod.yaml
We would like to refactor some of the service definitions inside docker-compose-common.yaml, so rest of the lifecycle specific definitions can inherit.
Is it possible to reuse service definitions across multiple docker-compose files, in the above fashion?
Yes, this is possible, please check the official documentation here.
The file containing the common definitions must be named docker-compose.yml, for the overrides you can keep the names you already use (-dev, -staging, etc.).
Just make sure you use the -f parameter when running docker compose:
docker-compose -f docker-compose.yaml -f docker-compose-prod.yaml up -d
Related
I have a simple docker-compose file which is used to launch my containers. I wish to have another yaml file which contains additional, optional containers. It can live in a separate directory. My goal is to find a way to force the namespace of the created swarms so they exist within the same network/use space so they can talk to each other.
compose1.yaml
services:
web:
build: .
compose2.yaml
services:
web1:
build: .
So if i run both of these they would be prepended with the folder they exist in, in my case: a, and b respectively.
I wanted to ensure that they flow together, despite not being in the same file hierarchy.
I have been coming over keywords in the docker-compse documents, and was not sure what the best way to do this in the yaml file would be, but noticed in the CLI, might be able up to update various names.
How does one accomplish this?
Note: I have also created a third file under the b directory, a sibling to compose2.yaml. So i can run those separately and they work just fine.
a/
compose.yaml
b/
compose2.yaml
another.yaml
So i have been able to merge them together by doing: cd /b/ && docker-compose -f compose2.yaml -f another.yaml up -d to run 2 files together, and they exist under the B namespace. Likewise, I can also run them sequentially instead of referencing them in 1 command.
So my question is how can I do something like:
docker-compose --namespace test compose.yaml up
docker-compose --namespace test compose2.yaml up
such that I could view items accordingly with docker? It seems that I would need to consider running the command from under the first shared parent folder?
so if a and b existed under test, I could just do:
cd /test
docker-compose -f a/compose.yaml up -d
docker-compose -f b/compose2.yaml up -d
then my services would be listed as: test_web, test_db-box, etc.
So I found out that one person's namespace is another person's project-name.
That being said, after understanding nuances, the project-name ( -p | --project-name ) is the prepend for the compose services.
docker-compose --project-name foo -f a/compose.yaml up
docker-compose --project-name foo -f b/compose2.yaml up
This will create the services: foo_web_1
The format for this is: %{prepend}%{servicename}%{number}
The issue then is, Can we find a way to implement this CLI property to work from within the YAML file, possibly as a config option for the file. The Docker Compose website information states that you can supply an environment variable ( _ COMPOSE_PROJECT_NAME_ ) to change change the project name from the default of the base directory, BUT not from within a Compose YAML.
If i want to then launch multiple compose files, under a particular project what I would want to do is to just encapsulate it with a BASH or SHELL script.
#/bin/bash
export COMPOSE_PROJECT_NAME=ultimate-project
docker-compose -f a/compose.yaml up -d
docker-compose -f b/compose2.yaml up -d
and that would create services :
ultimate-project_web_1
ultimate-project_web2_1
How do you name your docker-compose.yml files? Is there a convention to follow?
With Dockerfiles <purpose>.Dockerfile seems to work the best being instantly recognized by VSCode and PyCharm.
I like the idea of structuring docker/compose files into folders so that default names could be used, but as far as I know docker can't see files up the tree creating different problems.
According to the -f documentation:
If you don’t provide this flag on the command line, Compose [...] looks for a
docker-compose.yml and a docker-compose.override.yml file.
If you don't want to use the -f flag you can use the default name docker-compose.yml and override it with docker-compose.override.yml.
However, if you use -f, since you'll provide the filename, you can use whatever you want.
I think a good way to name them depending on the environment could be docker-compose.{env}.yml and place them at the top level directory:
docker-compose.prod.yml
docker-compose.dev.yml
docker-compose.test.yml
docker-compose.staging.yml
And you can use the default docker-compose.yml to define the base configuration that is common to all environments.
I have a project with components base on Docker and orchestrated with docker-compose. Some of them are optional, and can be added at runtime.
I can think about two ways to achieve that:
Create a new serviceA.yml compose file and run it as a separate project
Add serviceA to my base compose.yml and run it again
What is the preferred option to do that?
I've also seen that you can combine docker-compose files with the extend keyword, but I don't think this can fit, since I have a variable number of services that I can add at runtime.
I usually end up having multiple yml files.
Then you can add several -f flags to docker-compose command to indicate which services to run.
For instance, having:
docker-compose.yml: containing basic services
docker-compose-additional-services.yml: containing additional/optional services.
You can execute to start:
docker-compose -f docker-compose.yml -f docker-compose-additional-services.yml up
And it will start all services.
Note, that all services are merged as if they were in a single file, so you can reference (depends_on, link) services from one file to the other.
We are using docker in a team of developers. We have on project all devs work on. Since we do not want to have one docker-compose.yml for each developer we use environment variables to pass the username to docker-compose. Inside docker-compose we have something like this
services:
myservice:
image: myimage
container_name: ${user}_myservice
This used to work very well for us but has stopped working lately. Assume there are two users. The first user runs docker-compose up myservice launching ${user1}_myservice. When the second user issues the same command, the second user will kill the container running under ${user1}_myservice and start ${user2}_myservice.
Somehow it seems that docker services are now linked directly and not only through the container_name variable as before.
We recently upgraded docker to Docker version 17.09.0-ce, build afdb6d4. I attribute the change to the "new" docker version. I have tried downgrading docker-compose to previous versions and it seems this is not related to docker-compose.
UPDATE
Inspired by the answer below we found the following workaround:
We set the env variable COMPOSE_PROJECT_NAME to be the username on login of the user on the host. Then we extend the service name in our docker-compose.yml files to be <proj>_<service>, thereby avoiding any conflicts between identical service names across projects.
Rather than mucking about with variables in docker-compose.yml, it's probably easier just to make use of the --project-name (-p) option to docker-compose.
Normally, docker-compose derives the project name from the name of the directory that contains your docker-compose.yaml file. So if two people try to start an application from a directory named myapp, they will end up with a conflict because both instances will attempt to use the same name.
However, if they were to run instead:
docker-compose --project-name ${USER}_myapp ...
Then docker-compose for each user would use different project names (like alice_myapp and bob_myapp) and there would be no conflict.
If people get tired of using the -p option, they could create a .env like this:
COMPOSE_PROJECT_NAME=alice_myapp
And this would have the same effect as specifying -p alice_myapp on the command line.
I'm trying to wrap my head around Docker, but I'm having a hard time figuring it out. I tried to implement it in my small project (MERN stack), and I was thinking how do you distinct between development, (maybe staging), and production environments.
I saw one example where they used 2 Docker files, and 2 docker-compose files, (each pair for one env, so Dockerfile + docker-compose.yml for prod, Dockerfile-dev + docker-compose-dev.yml for dev).
But this just seems like a bit of an overkill for me. I would prefer to have it only in two files.
Also one of the problem is that e.g. for development I want to install nodemon globally, but not for poduction.
In perfect solution I imagine running something like that
docker-compose -e ENV=dev build
docker-compose -e ENV=dev up
Keep in mind, that I still don't fully get docker, so if you caught some of mine misconceptions about docker, you can point them out.
You could take some clues from "Using Compose in production"
You’ll almost certainly want to make changes to your app configuration that are more appropriate to a live environment. These changes may include:
Removing any volume bindings for application code, so that code stays inside the container and can’t be changed from outside
Binding to different ports on the host
Setting environment variables differently (e.g., to decrease the verbosity of logging, or to enable email sending)
Specifying a restart policy (e.g., restart: always) to avoid downtime
Adding extra services (e.g., a log aggregator)
The advice is then not quite similar to the example you mention:
For this reason, you’ll probably want to define an additional Compose file, say production.yml, which specifies production-appropriate configuration. This configuration file only needs to include the changes you’d like to make from the original Compose file.
docker-compose -f docker-compose.yml -f production.yml up -d
This overriding mechanism is better than trying to mix dev and prod logic in one compose file, with environment variable to try and select one.
Note: If you name your second dockerfile docker-compose.override.yml, a simple docker-compose up would read the overrides automatically.
But in your case, a name based on the environment is clearer.
Docker Compose will read docker-compose.yml and docker-compose.override.yml by default. Understanding-Multiple-Compose-Files
You can set a default docker-compose.yml and different overwrite compose file. For example, docker-compose.prod.yml docker-compose.test.yml. Keep them in the same place.
Then create a symbolic link named docker-compose.override.yml for each env.
Track docker-compose.{env}.yml files and add docker-compose.override.yml to .gitignore.
In prod env: ln -s ./docker-compose.prod.yml ./docker-compose.override.yml
In test env: ln -s ./docker-compose.test.yml ./docker-compose.override.yml
The project structure will then look like this:
project\
- docker-compose.yml # tracked
- docker-compose.prod.yml # tracked
- docker-compose.test.yml # tracked
- docker-compose.override.yml # ignored & linked to override composefile for current env
- src/
- ...
Then you have done. In each environment, you can use the compose-file with the same command docker-compose up
If you are not sure, use docker-compose config to check if it's been override properly.