From the docker docs:
Docker Compose’s extends keyword enables sharing of common
configurations among different files, or even different projects
entirely. Extending services is useful if you have several services
that reuse a common set of configuration options. Using extends you
can define a common set of service options in one place and refer to
it from anywhere.
For some reason this feature was removed in version 3.
Found also this thread, but it is inactive for 2 years.
I'm trying to find a replacement for this feature in the newer versions.
Would like to hear if somebody found a replacement for extends.
Thanks.
There are 2 ways to achieve what you need, you can decide to use one of them or both at the same time as they work slightly differently:
Multiple compose files
You can specify multiple compose files when running a docker compose command, you could for instance set up your project with:
docker-compose -f config1.yml -f config2.yml up
You could also use an environment variable to specify your files:
COMPOSE_FILE=config1.yml:config2.yml docker-compose up
What happens is that docker compose creates a single config merging what you defined in each of them.
Here the documentation showing how to merge multiple compose files.
You can also generate your final config file running the config command.
YAML Anchors
Since docker compose files are basically YAML files, you can take advantage of YAML Anchors to define a block of properties and reuse them in multiple parts of your config.
For example:
version: '3'
common: &common
image: "myrepo/myimage"
restart: "unless-stopped"
volumes:
- "volume:/mnt/myvolume"
services:
service1:
<<: *common
ports:
- "5000:5000"
service2:
<<: *common
environment:
- MYENV: value
Related
I need some help with the following template:
services:
nginx:
image: nginx
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.nginx-${COMPOSE_PROJECT_NAME}.rule=Host(`fuu.bar`)"
networks:
- treafik
My goal is to create a template which I can use e. g. in portainer with almost zero configuration.
I thought that the following variables are available in docker-compose config but the expression ${COMPOSE_PROJECT_NAME} results in an empty string: docker-compose config
services:
nginx:
image: nginx
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.nginx-.rule=Host(`fuu.bar`)"
networks:
- treafik
Are there any default environment variables provided by docker-compose which I can use for environment interpolation?
---- Update
I use traefik (v2) as a reverse proxy. To make the containers available through treafik, you need to define routers on every service. The router name has to be unique. Lets imagine you deploy 2 or more stacks of the above template. The router name has to be unique for all services across all stacks. Because Im a lazy guy, I tried to simply integrate the environment variable COMPOSE_PROJECT_NAME (which I know is already unique in my setup because every stack must have a unique name). But the variable is not available when deploying the stack.
Of course, I could simply define the variable COMPOSE_PROJECT_NAME by myself in a .env-file, but i hoped that there are any default environment variables provided by docker.
You can use environment variables to passing strings to your docker file.
There are many ways through docker documentation. For example:
You can set default values for any environment variables referenced in the Compose file, or used to configure Compose, in an environment file named .env. The .env file path is as follows:
Starting with +v1.28, .env file is placed at the base of the project
directory
Project directory can be explicitly defined with the --file option or
COMPOSE_FILE environment variable. Otherwise, it is the current
working directory where the docker compose command is executed
(+1.28).
For previous versions, it might have trouble resolving .env file with
--file or COMPOSE_FILE. To work around it, it is recommended to use --project-directory, which overrides the path for the .env file. This inconsistency is addressed in +v1.28 by limiting the filepath to the
project directory.
Letzt imagine i have 3 compose files (only focus on the mysql service)
docker-compose.yml
docker-compose.staging.yml
docker-compose.prod.yml
In my docker compose.yml i have my basic mysql stuff with dev als build target
version: "3.4"
services:
mysql:
build:
target: dev
...
And start it with
docker-compose up -d
In my staging environment i would like to expose port 3306, but also want another build target so i would create the docker-compose.staging.yml with the following content.
version: "3.4"
services:
mysql:
build
target: prod
ports:
- 3306:3306
And combine it with
docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d
So the build target is overwritten and the port 3306 is now exposed to the outside.
Now i want the same in the docker-compose.prod.yml, just without having the port 3306 exposed to the outside ... How can i override the ports directive to not having ports exposed?
I tried to put an empty array in the prod.yml without success (port is still exposed):
version: "3.4"
services:
mysql:
ports: []
In the end i would like to stack the up command like this:
docker-compose -f docker-compose.yml -f docker-compose.staging.yml -f docker-compose.prod.yml up -d
I also know the docs says
For the multi-value options ports, expose, external_links, dns, dns_search, and tmpfs, Compose concatenates both sets of values
But how can i reach my goal anyway without duplicating configuration?
Yes for sure, i could omit the docker-compose.staging.yml but in the staging.yml are build steps defined, which should also be used for the prod stage to not have any differences between the built container.
So duplicating things isn't really an option.
Thanks
I would actually strongly suggest just not using the "target" command in your compose files. I find it to be extremely beneficial to build a single image for local/staging/production - build once, test it, and deploy it in each environment. In this case, you change things using environment variables or mounted secrets/config files.
Further, using compose to build the images is... fragile. I would recommend building the images in a CI system, pushing them to a registry, and then using the image version tags in your compose file- it is a much more reproducible system.
You might consider using extends key in your compose files like this:
mysql:
extends:
file: docker-compose.yml
service: mysql
ports:
- 3306:3306
# other definitions
Although you'd have to change your compose version from 3.4 to < 3 ( like 2.3 ) because v3 doesn't support this feature ref as there is a open feature request hanging for a long time now.
Important note here is that you shouldn't expose any ports in your base docker-compose.yml file, only on the specific composes.
Oficial docs ref for extends
edit
target clause is not supported in v2.0 so I've adjusted the answer to match the extends and target requirement. That's compose v2.3.
edit from comments
As there is a deploy keyword requirement, then there is compose v3 requirement. And as for now, there is no possibility to extend composes. I've read in some official doc (can't find it now for ref) that they encourage us to use flat composes specific for environment so that it's always clear. Also Docker states that's hard to implement in v3 (ref in the above issue) and it's not going to be implemented anywhere soon. You have to use separate compose files per environment.
I have three services to build, A、B and C. A should be built in the very first place, because B and C depend on A (they import A as image). I thought they should be built in order but I just find out they are built in some random order?
So, how do I control build order in docker-compose.yml?
The accepted answer above does not seem correct.
As seen in the documentation
There are several things to be aware of when using depends_on:
depends_on does not wait for db and redis to be “ready” before
starting web - only until they have been started. If you need to wait
for a service to be ready, see Controlling startup order for more on
this problem and strategies for solving it.
Version 3 no longer supports the condition form of depends_on.
The depends_on option is ignored when deploying a stack in swarm mode
with a version 3 Compose file.
Update:
In recent practice, I have found my answer to only pertain to run ordering.
Refer to the answer by Quinten Scheppermans and the comment by Authur Weborg about dobi.
You can control build order using depends_on directive.
services:
...
B:
depends_on:
- A
C:
depends_on:
- A
If you need to run one service before an other you can use docker-compose run bar after setting depends_on: - foo in your docker-compose.yml file.
For example:
# docker-compose.yml
services:
foo:
. . .
bar:
. . .
depends_on:
- foo
then run,
# terminal
user#Name Docker % docker-compose run bar
However, per the docs, this will only control the order in which processes start, not the order in which they finish. This can often lead to situations in which dependent services do not properly start. Potential solutions for this include using wait-for-it, wait-for, or Dockerise. I recommend reading the docs before implementing any of the solutions listed above, as there are caveats.
Since compose v2, BuildKit is enabled by default, it will build images parallelly.
By disabling BuildKit you get to build images according to the order in your docker-compose.yml file.
DOCKER_BUILDKIT=0 docker-compose build
If you still want to build images parallelly, you can consider defining services in multiple docker-compose.yml files for building, then composing up in another docker-compose.yml file for deployment.
As stated in https://github.com/docker/compose/blob/e9220f45df07c1f884d5d496507778d2cd4a1687/compose/project.py#L182-L183
Preserves the original order of self.services where possible, reordering as needed to resolve dependencies.
So for me it worked by manually sorting the services with the depending services at first and following the services which is used by the other and the other as last.
Example
version: '3'
services:
mydb:
image: mydb
serviceA:
build:
dockerfile: DockerfileA
depends_on:
- mydb
serviceB:
build:
dockerfile: DockerfileB # which contains 'FROM serviceA'
depends_on:
- mydb
Source: https://github.com/docker/compose/issues/5228#issuecomment-565469948
I want manage environment variables in a stack, then the service can use it. For example:
I definition a evn tracker_ip=192.168.0.101, then I want use it in service create.
what should I do
There can be several answers depending on what you are trying to do and how you are deploying your stack.
Using the CLI / Rancher Compose
If you are using the command line, you can just use variable interpolation. Instructions on how to do so can be found in the official documentation:
https://docs.rancher.com/rancher/v1.5/en/cli/variable-interpolation/
Using the Rancher UI / Catalogs
If you want to do it through the Rancher UI, you can do it by creating a template in a catalog and having questions to input your environment variables. More details on how to do so here:
https://docs.rancher.com/rancher/v1.5/en/catalog/private-catalog/
You can define questions in the rancher-compose.yml file like this:
version: '2'
catalog:
name: My Application
version: v0.0.1
questions:
- variable: TRACKER_IP
label: Tracker IP address
required: true
default: 192.168.0.101
type: string
You can then push the answers to the environment section of your docker-compose.yml template for use within your image:
version: '2'
services:
web:
image: myimage
ports:
- 8000
environment:
TRACKER_IP: ${TRACKER_IP}
There is no way to do exactly what you're asking, because that would allow editing of the variables of running containers and containers are immutable. Environment variables can be defined on services but not defined once on stacks and made available to all services.
Secrets are somewhat like this and can be shared across services, but not edited.
Depending on where the tracker_ip is associated, you could also create an external service as part of the stack. The external service essentially just creates a DNS entry in Rancher. So you could then just link your service to the external_tracker service in compose and refer to tracker.
version: '2'
services:
myservice:
...
link:
- tracker_service:tracker
...
If I have (simplified), the following docker-compose.yml:
parent:
image: parent
links:
- child
child:
image: child
Can I construct a docker-compose.override.yml file that will not create or start the child image?
An undesirable (for me) solution, would be to reverse the files, such that the default yml file would create only the parent, and the override would create both.
However, I would like the master configuration file to contain the most common usage scenario.
In defense of the original poster: I totally get why you would want to do this.
The docker-compose way always seems to be "put your variation in the override files",
but in the interest of keeping things simple within a large development team, I like:
The ability to start everything with one command (e.g. "docker-compose up" or "docker-compose up main")
All of my docker definitions in one place
The only variation in override files to be which containers are disabled
Here's how I did it in my override files:
# Disable database containers
postgres:
image: alpine:latest
command: "true"
entrypoint: "true"
mysql:
image: alpine:latest
command: "true"
entrypoint: "true"
The upshot is, all containers are started by "docker-compose up", but those that I've overridden immediately die.
If you want an ever lighter weight container than alpine, try tianon/true.
I really like solution from Ryan. But it can be improved.
Create docker-compose.override.yml next to docker-compose.yml with content:
# disable services
version "3"
fluentd:
image: hello-world
command: hello
restart: "no"
elasticsearch:
image: hello-world
command: hello
restart: "no"
I believe hello-world is a smallest image (size: ~1Kb) on the other hand the size of alpine linux is ~6Mb
For versions of compose < 1.28, january 2021
If you're disabling multiple services in an override file (As in Ryan's answer ), you might find it useful for Don't-Repeat-Yourself to make use of extension fields, & yaml anchors (essentially "back references" in yaml).
As in:
# Your version
version: "2.4"
# `&anchor-name` defines the anchor (Here it is a map/dict)
# `*anchor-name` references the anchor
# `<<: *anchor-name` merges (`<<:`) the keys of `*anchor-name` with the current map
x-disabled-service: &disabled
image: "tianon/true"
command: "true"
entrypoint: "true"
services:
elasticsearch:
<<: *disabled
fluentdb:
<<: *disabled
This uses tianon/true as suggested in Ryan's comment as a very small image.
The outcome is functionally the same as Ryan, or Roman's answers, but a bit less verbose.
For compose versions >= 1.28
Also, according to mcarson's answer to a similar SO question as of compose 1.28, January 2021, there exists a new compose field, profiles, which lets you group together containers that can be switched on using a commandline option --profile.
https://docs.docker.com/compose/profiles/
You don't have to start every service when you run compose, you can just run up and pass the names of the services to start. See the official reference for up here.
So for instance: docker-compose up -d parent