I have three services to build, A、B and C. A should be built in the very first place, because B and C depend on A (they import A as image). I thought they should be built in order but I just find out they are built in some random order?
So, how do I control build order in docker-compose.yml?
The accepted answer above does not seem correct.
As seen in the documentation
There are several things to be aware of when using depends_on:
depends_on does not wait for db and redis to be “ready” before
starting web - only until they have been started. If you need to wait
for a service to be ready, see Controlling startup order for more on
this problem and strategies for solving it.
Version 3 no longer supports the condition form of depends_on.
The depends_on option is ignored when deploying a stack in swarm mode
with a version 3 Compose file.
Update:
In recent practice, I have found my answer to only pertain to run ordering.
Refer to the answer by Quinten Scheppermans and the comment by Authur Weborg about dobi.
You can control build order using depends_on directive.
services:
...
B:
depends_on:
- A
C:
depends_on:
- A
If you need to run one service before an other you can use docker-compose run bar after setting depends_on: - foo in your docker-compose.yml file.
For example:
# docker-compose.yml
services:
foo:
. . .
bar:
. . .
depends_on:
- foo
then run,
# terminal
user#Name Docker % docker-compose run bar
However, per the docs, this will only control the order in which processes start, not the order in which they finish. This can often lead to situations in which dependent services do not properly start. Potential solutions for this include using wait-for-it, wait-for, or Dockerise. I recommend reading the docs before implementing any of the solutions listed above, as there are caveats.
Since compose v2, BuildKit is enabled by default, it will build images parallelly.
By disabling BuildKit you get to build images according to the order in your docker-compose.yml file.
DOCKER_BUILDKIT=0 docker-compose build
If you still want to build images parallelly, you can consider defining services in multiple docker-compose.yml files for building, then composing up in another docker-compose.yml file for deployment.
As stated in https://github.com/docker/compose/blob/e9220f45df07c1f884d5d496507778d2cd4a1687/compose/project.py#L182-L183
Preserves the original order of self.services where possible, reordering as needed to resolve dependencies.
So for me it worked by manually sorting the services with the depending services at first and following the services which is used by the other and the other as last.
Example
version: '3'
services:
mydb:
image: mydb
serviceA:
build:
dockerfile: DockerfileA
depends_on:
- mydb
serviceB:
build:
dockerfile: DockerfileB # which contains 'FROM serviceA'
depends_on:
- mydb
Source: https://github.com/docker/compose/issues/5228#issuecomment-565469948
Related
Is is possible to skip build step even if it's specified. For example for the following file only redis image is pulled after running docker-compose pull command.
version: '3.4'
services:
myservice:
image: ${PRIVATE_REGISTRY-}my-service
build:
context: .
dockerfile: MyService/Dockerfile
ports: [5555:80]
redis:
image: redis
restart: ${RESTART_POLICY-no}
And if I remove build: step from the file then the other image is getting pulled as well.
No, generally this is not possible as docker doesn't know then what to do. If you access the situation that your image doesn't have to be built everytime then try considering building it from cli when this is needed. In general I would advise to work with images and named versions. This version may be "lastest" for example in CI. Suggestion to take this approach:
DEV: your approach is perfect for often changing containers
STAGE: when moving from dev to stage, tag the images and give them a version. Only use namend versions in higher environments
PROD: same as stage, other version naming strategies
From the docker docs:
Docker Compose’s extends keyword enables sharing of common
configurations among different files, or even different projects
entirely. Extending services is useful if you have several services
that reuse a common set of configuration options. Using extends you
can define a common set of service options in one place and refer to
it from anywhere.
For some reason this feature was removed in version 3.
Found also this thread, but it is inactive for 2 years.
I'm trying to find a replacement for this feature in the newer versions.
Would like to hear if somebody found a replacement for extends.
Thanks.
There are 2 ways to achieve what you need, you can decide to use one of them or both at the same time as they work slightly differently:
Multiple compose files
You can specify multiple compose files when running a docker compose command, you could for instance set up your project with:
docker-compose -f config1.yml -f config2.yml up
You could also use an environment variable to specify your files:
COMPOSE_FILE=config1.yml:config2.yml docker-compose up
What happens is that docker compose creates a single config merging what you defined in each of them.
Here the documentation showing how to merge multiple compose files.
You can also generate your final config file running the config command.
YAML Anchors
Since docker compose files are basically YAML files, you can take advantage of YAML Anchors to define a block of properties and reuse them in multiple parts of your config.
For example:
version: '3'
common: &common
image: "myrepo/myimage"
restart: "unless-stopped"
volumes:
- "volume:/mnt/myvolume"
services:
service1:
<<: *common
ports:
- "5000:5000"
service2:
<<: *common
environment:
- MYENV: value
Letzt imagine i have 3 compose files (only focus on the mysql service)
docker-compose.yml
docker-compose.staging.yml
docker-compose.prod.yml
In my docker compose.yml i have my basic mysql stuff with dev als build target
version: "3.4"
services:
mysql:
build:
target: dev
...
And start it with
docker-compose up -d
In my staging environment i would like to expose port 3306, but also want another build target so i would create the docker-compose.staging.yml with the following content.
version: "3.4"
services:
mysql:
build
target: prod
ports:
- 3306:3306
And combine it with
docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d
So the build target is overwritten and the port 3306 is now exposed to the outside.
Now i want the same in the docker-compose.prod.yml, just without having the port 3306 exposed to the outside ... How can i override the ports directive to not having ports exposed?
I tried to put an empty array in the prod.yml without success (port is still exposed):
version: "3.4"
services:
mysql:
ports: []
In the end i would like to stack the up command like this:
docker-compose -f docker-compose.yml -f docker-compose.staging.yml -f docker-compose.prod.yml up -d
I also know the docs says
For the multi-value options ports, expose, external_links, dns, dns_search, and tmpfs, Compose concatenates both sets of values
But how can i reach my goal anyway without duplicating configuration?
Yes for sure, i could omit the docker-compose.staging.yml but in the staging.yml are build steps defined, which should also be used for the prod stage to not have any differences between the built container.
So duplicating things isn't really an option.
Thanks
I would actually strongly suggest just not using the "target" command in your compose files. I find it to be extremely beneficial to build a single image for local/staging/production - build once, test it, and deploy it in each environment. In this case, you change things using environment variables or mounted secrets/config files.
Further, using compose to build the images is... fragile. I would recommend building the images in a CI system, pushing them to a registry, and then using the image version tags in your compose file- it is a much more reproducible system.
You might consider using extends key in your compose files like this:
mysql:
extends:
file: docker-compose.yml
service: mysql
ports:
- 3306:3306
# other definitions
Although you'd have to change your compose version from 3.4 to < 3 ( like 2.3 ) because v3 doesn't support this feature ref as there is a open feature request hanging for a long time now.
Important note here is that you shouldn't expose any ports in your base docker-compose.yml file, only on the specific composes.
Oficial docs ref for extends
edit
target clause is not supported in v2.0 so I've adjusted the answer to match the extends and target requirement. That's compose v2.3.
edit from comments
As there is a deploy keyword requirement, then there is compose v3 requirement. And as for now, there is no possibility to extend composes. I've read in some official doc (can't find it now for ref) that they encourage us to use flat composes specific for environment so that it's always clear. Also Docker states that's hard to implement in v3 (ref in the above issue) and it's not going to be implemented anywhere soon. You have to use separate compose files per environment.
I want to use docker-compose to maintain containers, there is a cluster of API servers.
They build from the same image, I knew the docker-compose scale app=5 will start 5 containers, but they all same, including port setting.
I want to run multiple containers like this:
services:
# wx_service_cluster
wx_service_51011:
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/go/src/wx_service
ports:
- "51011:8080"
wx_service_51012:
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/go/src/wx_service
ports:
- "51012:8080"
wx_service_...:
....
THERE ARE ALMOST 100 SERVICES NEED TO BE WROTE
ANYONE CAN HELPS ME TO MAKE IT SIMPLER.
can I make it simpler?
like a shell loop:
for each_port in $( seq 51011 51040 )
{
wx_service_${each_port}:
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/go/src/wx_service
ports:
- "${each_port}:8080"
}
Simple answer to your actual questions: Use ENV variables and probably combine it with dotenv https://docs.docker.com/compose/environment-variables/
services:
foo_{$instance1}
ports:
- "${instance1}:8080"
foo_{$instance12}
ports:
- "${instance2}:8080"
but this will not help you "generating a docker-compose file with X service entries for WX" .. you seem to plan some kind of "hosting".
Alternatives:
You should step back, and rather use random-port assignment and the use docker inspect to find the port - see an example here https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/unison.rb#L199 .. so basically you either use a template system to generate you docker-compose.yml file - e.g. like https://github.com/markround/tiller .. then you generate services with a static prefix like wx_service_ .. and later you use a different script ( for you nginx / haproxy ) to configure and upstream for each of those, find the name and port (using inspect) dynamically.
If i am right and you really go for some kind of hosting scenario and you do it commercially - you might even rethink this and add consul to the game. Let every wx service register as a service in consul and then use a additional httpd passenger like nginx / haproxy to reconfigure itself and add a backend+frontend/upstream+server entry in the passender using tiller+consul watch.
The last one is just the next level stuff, but if you do that "commercially" you should not do what you asked for initially - nevertheless, if you choose to, use dotenv as outlined above
If I have (simplified), the following docker-compose.yml:
parent:
image: parent
links:
- child
child:
image: child
Can I construct a docker-compose.override.yml file that will not create or start the child image?
An undesirable (for me) solution, would be to reverse the files, such that the default yml file would create only the parent, and the override would create both.
However, I would like the master configuration file to contain the most common usage scenario.
In defense of the original poster: I totally get why you would want to do this.
The docker-compose way always seems to be "put your variation in the override files",
but in the interest of keeping things simple within a large development team, I like:
The ability to start everything with one command (e.g. "docker-compose up" or "docker-compose up main")
All of my docker definitions in one place
The only variation in override files to be which containers are disabled
Here's how I did it in my override files:
# Disable database containers
postgres:
image: alpine:latest
command: "true"
entrypoint: "true"
mysql:
image: alpine:latest
command: "true"
entrypoint: "true"
The upshot is, all containers are started by "docker-compose up", but those that I've overridden immediately die.
If you want an ever lighter weight container than alpine, try tianon/true.
I really like solution from Ryan. But it can be improved.
Create docker-compose.override.yml next to docker-compose.yml with content:
# disable services
version "3"
fluentd:
image: hello-world
command: hello
restart: "no"
elasticsearch:
image: hello-world
command: hello
restart: "no"
I believe hello-world is a smallest image (size: ~1Kb) on the other hand the size of alpine linux is ~6Mb
For versions of compose < 1.28, january 2021
If you're disabling multiple services in an override file (As in Ryan's answer ), you might find it useful for Don't-Repeat-Yourself to make use of extension fields, & yaml anchors (essentially "back references" in yaml).
As in:
# Your version
version: "2.4"
# `&anchor-name` defines the anchor (Here it is a map/dict)
# `*anchor-name` references the anchor
# `<<: *anchor-name` merges (`<<:`) the keys of `*anchor-name` with the current map
x-disabled-service: &disabled
image: "tianon/true"
command: "true"
entrypoint: "true"
services:
elasticsearch:
<<: *disabled
fluentdb:
<<: *disabled
This uses tianon/true as suggested in Ryan's comment as a very small image.
The outcome is functionally the same as Ryan, or Roman's answers, but a bit less verbose.
For compose versions >= 1.28
Also, according to mcarson's answer to a similar SO question as of compose 1.28, January 2021, there exists a new compose field, profiles, which lets you group together containers that can be switched on using a commandline option --profile.
https://docs.docker.com/compose/profiles/
You don't have to start every service when you run compose, you can just run up and pass the names of the services to start. See the official reference for up here.
So for instance: docker-compose up -d parent