If I have (simplified), the following docker-compose.yml:
parent:
image: parent
links:
- child
child:
image: child
Can I construct a docker-compose.override.yml file that will not create or start the child image?
An undesirable (for me) solution, would be to reverse the files, such that the default yml file would create only the parent, and the override would create both.
However, I would like the master configuration file to contain the most common usage scenario.
In defense of the original poster: I totally get why you would want to do this.
The docker-compose way always seems to be "put your variation in the override files",
but in the interest of keeping things simple within a large development team, I like:
The ability to start everything with one command (e.g. "docker-compose up" or "docker-compose up main")
All of my docker definitions in one place
The only variation in override files to be which containers are disabled
Here's how I did it in my override files:
# Disable database containers
postgres:
image: alpine:latest
command: "true"
entrypoint: "true"
mysql:
image: alpine:latest
command: "true"
entrypoint: "true"
The upshot is, all containers are started by "docker-compose up", but those that I've overridden immediately die.
If you want an ever lighter weight container than alpine, try tianon/true.
I really like solution from Ryan. But it can be improved.
Create docker-compose.override.yml next to docker-compose.yml with content:
# disable services
version "3"
fluentd:
image: hello-world
command: hello
restart: "no"
elasticsearch:
image: hello-world
command: hello
restart: "no"
I believe hello-world is a smallest image (size: ~1Kb) on the other hand the size of alpine linux is ~6Mb
For versions of compose < 1.28, january 2021
If you're disabling multiple services in an override file (As in Ryan's answer ), you might find it useful for Don't-Repeat-Yourself to make use of extension fields, & yaml anchors (essentially "back references" in yaml).
As in:
# Your version
version: "2.4"
# `&anchor-name` defines the anchor (Here it is a map/dict)
# `*anchor-name` references the anchor
# `<<: *anchor-name` merges (`<<:`) the keys of `*anchor-name` with the current map
x-disabled-service: &disabled
image: "tianon/true"
command: "true"
entrypoint: "true"
services:
elasticsearch:
<<: *disabled
fluentdb:
<<: *disabled
This uses tianon/true as suggested in Ryan's comment as a very small image.
The outcome is functionally the same as Ryan, or Roman's answers, but a bit less verbose.
For compose versions >= 1.28
Also, according to mcarson's answer to a similar SO question as of compose 1.28, January 2021, there exists a new compose field, profiles, which lets you group together containers that can be switched on using a commandline option --profile.
https://docs.docker.com/compose/profiles/
You don't have to start every service when you run compose, you can just run up and pass the names of the services to start. See the official reference for up here.
So for instance: docker-compose up -d parent
Related
I have a compose.yml like this one:
version: '3.7'
services:
nginx:
restart: unless-stopped
image: ghcr.io/user/frontend:latest
ports:
- 80:80
depends_on:
- backend
backend:
restart: unless-stopped
image: ghcr.io/user/backend:latest
entrypoint: /home/app/web/wsgi-entrypoint.sh
expose:
- 8000
We have 2 images stored on Github: frontend and backend.
My goal is the following: when an image has been updated on the Github Docker Registry, I'd like to automatically update the image on the server and launch the new one substituting the old one via docker-compose.
For example: I have a running compose made by frontend and backend, but I just pushed a new image: ghcr.io/user/frontend:latest.
Now, I want a single command which updates only the images that have been changed (in this case ghcr.io/user/frontend:latest) and when I reload the frontend webpage I see the changes.
My attempt is the following:
docker-compose up -d --build
But the system says:
compose-backend_1 is up-to-date
compose-nginx_1 is up-to-date
which is not true!
So, the working procedure I use is a bit manual:
docker pull ghcr.io/user/frontend:latest
I see in the console: Status: Downloaded newer image,
which is the proof that a new image has been downloaded.
Then, if I relaunch the same command the console displays: Status: Image is up to date for ghcr.io/user/frontend:latest
Finally:
docker-compose up -d --build
says: Recreating compose-nginx_1 ... done
I suppose the command docker-compose up -d --build ALONE is not looking for new images and so does not update the image that is changed.
So, is there a SINGLE specific command to fix this?
Should be achieved by running docker-compose pull, and then docker-compose up -d
Or, shorter: docker-compose up -d --pull always
You can use variable substitution in many places in a docker-compose.yml file, in particular including the image:. If you give every build a unique tag, then you can supply the tag as an environment variable, and it will work the way you describe.
Let's say the two images have the same tagging scheme (they don't necessarily need to). You could update the Compose file to say
version: '3.8'
services:
nginx:
restart: unless-stopped
image: ghcr.io/user/frontend:${TAG:-latest} # <--
ports:
- 80:80
depends_on:
- backend
backend:
restart: unless-stopped
image: ghcr.io/user/backend:${TAG:-latest} # <--
Notice the $TAG reference at the end of the image: lines. If TAG isn't set in the environment, it will use latest, but if it is, it will use that exact build.
Now, if I run:
TAG=20221020 docker-compose up -d
For both containers, Compose will notice that they're running an older build, automatically pull the updated image from GitHub, and restart both containers against the newer image.
This brings the mild complication of your continuous-deployment system needing to know the current image tag. In exchange, though, you get the ability to very easily roll back – if today's build turns out to have a critical bug you can run the exact same command with a different tag to redeploy on yesterday's build. A setup like this is also necessary if you're considering migrating to Kubernetes, since it depends on the text of the image: string changing to trigger a redeployment.
I am using an official Postgres12 image that I'm pulling inside the docker-compose.yml. Everything is working fine.
services:
db:
container_name: db
image: postgres:12
volumes:
- ...
ports:
- 5432:5432
environment:
- POSTGRES_USER=...
Now, when I run docker-compose up, I get this image
My question is: is there a way in which I can rename the image inside docker-compose.yml? I know there is a command but I require it to be everything inside the file if possible.
Thanks!
In a Compose file, there's no direct way to run docker tag or any other command that modifies some existing resource.
If you're trying to optionally point Compose at a local mirror of Docker Hub, you can take advantage of knowing the default repository is docker.io and use an optional environment variable:
image: ${REGISTRY:-docker.io}/postgres:latest
REGISTRY=docker-mirror.example.com docker-compose up
Another possible approach is to build a trivial image that doesn't actually extend the base postgres image at all:
build:
context: .
dockerfile: Dockerfile.postgres
image: local-images.example.com/my-project/postgres
# Dockerfile.postgres
FROM postgres:latest
# End of file
There's not really any benefit to doing this beyond the cosmetic appearances in the docker images output. Having it be clear that you're using a standard Docker Hub image could be slightly preferable; its behavior is better understood than something you built locally and if you have multiple projects running at once they can more obviously share the same single image.
I am trying to deploy my django app to app engine using dockerfile and for that after following a few blogs such as these, I created a docker-compose.yml file but when I run the docker compose up command or docker-compose -f docker-compose-deploy.yml run --rm gcloud sh -c "gcloud app deploy" I get an error key cannot contain a space. See below:
For example:
$ docker compose up
key cannot contain a space
$ cat docker-compose.yml
version: '3.7'
services:
app:
build:
context: .
ports: ['8000:8000']
volumes: ['./app:/app']
Can someone please help me to fix this error? I have tried yamllint to validate the yaml file for any space/indentation type of error and it doesn't show any error to me.
EDIT:
Here is the content for file in the longer command:
version: '3.7'
services:
gcloud:
image: google/cloud-sdk:338.0.0
volumes:
- gcp-creds:/creds
- .:/app
working_dir: /app
environment:
- CLOUDSDK_CONFIG=/creds
volumes:
gcp-creds:
Ok this is resolved finally! After beating my head around, I was able to finally resolve this issue by doing the following things:
Unchecked the option to use "Docker Compose v2" from my docker desktop settings. Here is the setting in Docker Desktop
Closed the docker desktop app and restarted it.
Please try these steps in case you face the issue. Thanks!
Just adding another alt answer here that I confirmed worked for me when following the steps above did not. My case is slightly different, but as Google brought me here first I thought I'd leave a note.
Check your env var values for spaces!
This may only be applicable if you are using env_var files (appreciate that OP is not in the minimal example, hence saying this is different).
Unescaped spaces in variables will cause this cryptic error message.
So, given a compose file like this:
version: '3.7'
services:
gcloud:
image: google/cloud-sdk:338.0.0
volumes:
- gcp-creds:/creds
- .:/app
working_dir: /app
env_file:
- some_env_file.env
If some_env_file.env looks like this:
MY_VAR=some string with spaces
then we get the cryptic key cannot contain a space.
If instead we change some_env_file.env to be like this:
MY_VAR="some string with spaces"
then all is well.
The issue has been reported to docker-compose.
Google brought me here first, and when your suggestion sadly didn't work for me, it then took me to this reddit thread, where I found out the above.
Docker Compose (at least since v2) automatically parses .env files before processing the docker-compose.yml file, regardless of any env_file setting within the yaml file. If any of the variables inside your .env file contains spaces, then you will get the error key cannot contain a space.
Two workarounds exist at this time:
Rename your .env file to something else, or
Create an alternate/empty .env file, e.g. .env.docker and then explicitly set the --env-file parameter, i.e. docker compose --env-file .env.docker config.
Track the related issues here:
https://github.com/docker/compose/issues/6741
https://github.com/docker/compose/issues/8736
https://github.com/docker/compose/issues/6951
https://github.com/docker/compose/issues/4642
https://github.com/docker/compose/commit/ed18cefc040f66bb7f5f5c9f7b141cbd3afbbc89
https://docs.docker.com/compose/env-file/
One more thing to be care about - since Compose V2, this error may be raise in case you have inline comments in the env file used by Compose. For example
---
version: "3.7"
services:
backend:
build:
context: .
dockerfile: Dockerfile
env_file: .app.env
and that .app.env is like this
RABBIT_USER=user # RabbitMQ user
the same error may occur. To fix just move comment to its own line
# RabbitMQ user
RABBIT_USER=user
Letzt imagine i have 3 compose files (only focus on the mysql service)
docker-compose.yml
docker-compose.staging.yml
docker-compose.prod.yml
In my docker compose.yml i have my basic mysql stuff with dev als build target
version: "3.4"
services:
mysql:
build:
target: dev
...
And start it with
docker-compose up -d
In my staging environment i would like to expose port 3306, but also want another build target so i would create the docker-compose.staging.yml with the following content.
version: "3.4"
services:
mysql:
build
target: prod
ports:
- 3306:3306
And combine it with
docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d
So the build target is overwritten and the port 3306 is now exposed to the outside.
Now i want the same in the docker-compose.prod.yml, just without having the port 3306 exposed to the outside ... How can i override the ports directive to not having ports exposed?
I tried to put an empty array in the prod.yml without success (port is still exposed):
version: "3.4"
services:
mysql:
ports: []
In the end i would like to stack the up command like this:
docker-compose -f docker-compose.yml -f docker-compose.staging.yml -f docker-compose.prod.yml up -d
I also know the docs says
For the multi-value options ports, expose, external_links, dns, dns_search, and tmpfs, Compose concatenates both sets of values
But how can i reach my goal anyway without duplicating configuration?
Yes for sure, i could omit the docker-compose.staging.yml but in the staging.yml are build steps defined, which should also be used for the prod stage to not have any differences between the built container.
So duplicating things isn't really an option.
Thanks
I would actually strongly suggest just not using the "target" command in your compose files. I find it to be extremely beneficial to build a single image for local/staging/production - build once, test it, and deploy it in each environment. In this case, you change things using environment variables or mounted secrets/config files.
Further, using compose to build the images is... fragile. I would recommend building the images in a CI system, pushing them to a registry, and then using the image version tags in your compose file- it is a much more reproducible system.
You might consider using extends key in your compose files like this:
mysql:
extends:
file: docker-compose.yml
service: mysql
ports:
- 3306:3306
# other definitions
Although you'd have to change your compose version from 3.4 to < 3 ( like 2.3 ) because v3 doesn't support this feature ref as there is a open feature request hanging for a long time now.
Important note here is that you shouldn't expose any ports in your base docker-compose.yml file, only on the specific composes.
Oficial docs ref for extends
edit
target clause is not supported in v2.0 so I've adjusted the answer to match the extends and target requirement. That's compose v2.3.
edit from comments
As there is a deploy keyword requirement, then there is compose v3 requirement. And as for now, there is no possibility to extend composes. I've read in some official doc (can't find it now for ref) that they encourage us to use flat composes specific for environment so that it's always clear. Also Docker states that's hard to implement in v3 (ref in the above issue) and it's not going to be implemented anywhere soon. You have to use separate compose files per environment.
I have three services to build, A、B and C. A should be built in the very first place, because B and C depend on A (they import A as image). I thought they should be built in order but I just find out they are built in some random order?
So, how do I control build order in docker-compose.yml?
The accepted answer above does not seem correct.
As seen in the documentation
There are several things to be aware of when using depends_on:
depends_on does not wait for db and redis to be “ready” before
starting web - only until they have been started. If you need to wait
for a service to be ready, see Controlling startup order for more on
this problem and strategies for solving it.
Version 3 no longer supports the condition form of depends_on.
The depends_on option is ignored when deploying a stack in swarm mode
with a version 3 Compose file.
Update:
In recent practice, I have found my answer to only pertain to run ordering.
Refer to the answer by Quinten Scheppermans and the comment by Authur Weborg about dobi.
You can control build order using depends_on directive.
services:
...
B:
depends_on:
- A
C:
depends_on:
- A
If you need to run one service before an other you can use docker-compose run bar after setting depends_on: - foo in your docker-compose.yml file.
For example:
# docker-compose.yml
services:
foo:
. . .
bar:
. . .
depends_on:
- foo
then run,
# terminal
user#Name Docker % docker-compose run bar
However, per the docs, this will only control the order in which processes start, not the order in which they finish. This can often lead to situations in which dependent services do not properly start. Potential solutions for this include using wait-for-it, wait-for, or Dockerise. I recommend reading the docs before implementing any of the solutions listed above, as there are caveats.
Since compose v2, BuildKit is enabled by default, it will build images parallelly.
By disabling BuildKit you get to build images according to the order in your docker-compose.yml file.
DOCKER_BUILDKIT=0 docker-compose build
If you still want to build images parallelly, you can consider defining services in multiple docker-compose.yml files for building, then composing up in another docker-compose.yml file for deployment.
As stated in https://github.com/docker/compose/blob/e9220f45df07c1f884d5d496507778d2cd4a1687/compose/project.py#L182-L183
Preserves the original order of self.services where possible, reordering as needed to resolve dependencies.
So for me it worked by manually sorting the services with the depending services at first and following the services which is used by the other and the other as last.
Example
version: '3'
services:
mydb:
image: mydb
serviceA:
build:
dockerfile: DockerfileA
depends_on:
- mydb
serviceB:
build:
dockerfile: DockerfileB # which contains 'FROM serviceA'
depends_on:
- mydb
Source: https://github.com/docker/compose/issues/5228#issuecomment-565469948