Assuming I have a set of images which depend on a common base image:
base (this is only a set of common dependencies)
FROM ubuntu:16.04
ENV FOO 1
child1
FROM mybaseimage # where mybaseimage corresponds to base
CMD ["bar1_command"]
child2
FROM mybaseimage # where mybaseimage corresponds to base
CMD ["bar2_command"]
Is it possible to create docker-compose file which would build base without running it? Lets say I have following dependencies:
version: '2'
services:
child1:
build: ./path-to-child1-dockerfile
services:
child2:
build: ./path-to-child2-dockerfile
depends_on:
- child1
I would like base to be build even if it is not explicitly started. Is something like this even possible? Or should I simply use external Makefile to build dependencies?
build_base:
docker build -t mybaseimage mybaseimage
build_all: build_base
docker-compose build
It's possible. There's a kind of workaround. You're close, but you were missing explicit image tags (so you had little ability on child images to declare which image you inherited from).
version: '3.2'
services:
base:
image: mybaseimage
build: ./path-to-base-dockerfile
child1:
build: ./path-to-child1-dockerfile
depends_on:
- base
child2:
build: ./path-to-child2-dockerfile
depends_on:
- base
Let's say you have no images built. You run docker-compose up. The following things will happen:
docker-compose sees that child1 and child2 services depend on base. So it will deploy base first.
docker-compose sees that you have not yet tagged any image as mybaseimage. It knows how to build mybaseimage (you gave it a build path), so it will build it now, and tag it as mybaseimage.
docker-compose deploys the base service.
ideally you should design base so that it quits immediately, or has no entrypoint. since we don't actually want it to run this service.
docker-compose considers deploying child1 and child2
docker-compose sees that you have not yet tagged any image as child1. It knows how to build child1 (you gave it a build path), so it will build it now, and tag it as child1.
docker-compose deploys the child1 service
same sequence of steps for child2
The next docker-compose up will be simpler (we have tagged images available, so we skip all build steps).
If you already have tagged images, and want to rebuild: use docker-compose build to tell it to build all images (yes, base and children will both be rebuilt).
Use a Makefile. docker-compose is not designed to build chains of images, it's designed for running containers.
You might also be interested in dobi which is a build-automation tool (like make) designed to work with docker images and containers.
Disclaimer: I'm the author of dobi
You don't need separate images for this problem - just use an environment variable:
FROM ubuntu:16.04
ENV PROGRAM
CMD ${PROGRAM}
Then for container 1, set environment variable PROGRAM to bar1_command. Do the same for container 2, 3, ..., N.
Related
There are a few approaches to fix container startup order in docker-compose, e.g.
depends_on
docker-compose-wait
Docker Compose wait for container X before starting Y (Asked 7 years, 6 months ago, Modified 7 months ago, Viewed 483k times)
...
However, if one of the services in a docker-compose file includes a build directive, it seems docker-compose will try to build the image first (ignoring depends_on basically - or interpreting depends_on as start dependency, not build dependency).
Is it possible for a build directive to specify that it needs another service to be up, before starting the build process?
Minimal Example:
version: "3.5"
services:
web:
build: # this will run before postgres is up
context: .
dockerfile: Dockerfile.setup # needs postgres to be up
depends_on:
- postgres
...
postgres:
image: postgres:10
...
Notwithstanding the general advice that programs should be written in a way that handles the unavailability of services (at least for some time) gracefully, are there any ways to allow builds to start only when other containers are up?
Some other related questions:
multi-stage build in docker compose?
Update/Solution: Solved the underlying problem by pushing all the (database) setup required to the CMD directive of a bootstrap container:
FROM undertest-base:latest
...
CMD ./wait && ./bootstrap.sh
where wait waits for postgres and bootstrap.sh contains the code for setting up the postgres database with fixtures so the over system becomes fully testable after that script.
With that, setting up an ephemeral test environment with database setup becomes a simple docker-compose up again.
There is no option for this in Compose, and also it won't really work.
The output of an image build is a self-contained immutable image. You can do things like docker push an image to a registry, and Docker's layer cache will avoid rebuilding an image that it's already built. So in this hypothetical setup, if you could access the database in a Dockerfile, but you ran
docker-compose build
docker-compose down -v
docker-compose up -d --build
the down -v step will remove the storage the database uses. While the up --build option will cause the image to be rebuilt, the build sequence will skip all of the steps and produce the same image as originally, and whatever changes you might have made to the database won't have happened.
At a more mechanical layer, the build sequence doesn't use the Compose-provided network, so you also wouldn't be able to connect to the database container.
There are occasional use cases where a dependency in build: would be handy, in particular if you're trying to build a base image that other images in your Compose setup share. But neither the stable Compose file v3 build: block nor the less-widely-supported Compose specification build: supports any notion of an image build depending on anything else.
my docker-compose.yml file is below.
I want to build parallel those images - I am running the a command docker-compose build --parallel
BUT I want to run a command before it builds the images of service2 & service3 while building service1 - parallel.
When the command will be finished it will join to the building-parallel-process.
version: '3.4'
services:
service1:
image: "company/service1:${TAG}"
build:
context: ./folder/service1/
dockerfile: Dockerfile
service2:
image: "company/service2:${TAG}"
build:
context: ./folder/service2/
dockerfile: Dockerfile
service3:
image: "company/service3:${TAG}"
build:
context: ./folder/service3
dockerfile: Dockerfile
Compose doesn't really have any sort of workflow handling like this, especially around building images. It's assumed that building an image only depends on the local source tree and nothing else. Compose also doesn't have any ability to run non-Docker commands or launch temporary containers as part of the up workflow.
The good news is that re-running a build is very quick if nothing has changed. So with the workflow you've described, you might separately build the first image, run the command, and then rebuild everything; re-rebuilding the first image will take almost no time and you won't get a new image.
#!/bin/sh
# Build the one image that needs special handling
docker-compose build service1
# Run the command
the_command
# Rebuild everything in parallel (service1 will be a no-op)
docker-compose build --parallel
If you can run the preparatory step in a Dockerfile RUN command that might be easier to manage. If that needs software that isn't ordinarily part of your image, you could use a multi-stage build to do it in effectively a temporary image.
I have setup a docker-compose project which are creating multiple images:
cache_server:
image: current_timezone/full-supervisord-cache-server:1.00
container_name: renamed-varnish-cache
networks:
- network_frontend
build:
context: "./all-services/"
dockerfile: "./cache-server/Dockerfile.cacheserver.varnish"
args:
- DOCKER_CONTAINER_USERNAME=username
ports:
- "6081:6081"
- "6082:6082"
When I use docker-compose up -f file1.yml file2.override.yml I will then get the containers: in the case of above one it will be named : renamed-varnish-cache
In the corresponding Dockerfile (./nginx-proxy/Dockerfile.proxy.nginx) I want to be able use the container_name property defined in the docker-compose.yml shown above.
When the containers are created I want to update the Varnish configurations inline inside Dockerfile : RUN sed -i "s|webserver_container_name|renamed-varnish-cache|g" /etc/varnish/default.vcl"
For instance:
backend webserver_container_name{
.host = "webserver_container_name";
.port = "8080";
}
To: I anticipate I will have to replace the - with _ for the backend:
backend renamed_varnish_cache{
.host = "renamed-varnish-cache";
.port = "8080";
}
Is there a way to receive the docker-compose named items as variables inside Dockerfile?
In core Docker, there are two separate concepts. An image is a built version of some piece of software packaged together with its dependencies; a container is a running instance of an image. There are separate docker build and docker run commands to build images and launch containers, and you can launch multiple containers from a single image.
Docker Compose wraps these concepts. In particular, the build: block corresponds to the image-build step, and that is what invokes the Dockerfile. None of the other Compose options are available or visible inside the Dockerfile. You cannot access the container_name: or environment: variables or volumes: because those don't exist at this point in the build lifecycle; you also cannot contact other Compose services from inside the Dockerfile.
It's pretty common to have multiple containers run off the same image if they have largely the same code base but need a different top-level command. One example is a Python Django application that needs Celery background workers; you'd have the same project structure but a different command for the Celery worker.
version: '3.8'
services:
web:
build: .
image: my/django-app
worker:
image: my/django-app
command: celery worker ...
Now with this stack you can docker-compose build to build the one image, and then run docker-compose up to launch both containers from that image. (During the build you can't know what the container names will be, and there will be two container names so you can't just use one in the Dockerfile.)
At a design level, this means that you often can't include configuration-type settings in the image itself (other containers' hostnames, user IDs for host-shared filesystems). If your application lets you specify these things as environment variables, that's the easiest option. You can use bind mounts (volumes:) to inject whole config files. If neither of these things work for you, you can use an entrypoint script to rewrite the config file.
Let's say that we have docker-compose.yml which build few services with Dockerfile's in different locations like this:
version: '3.4'
services:
service1:
build: ./service1-dir
service2:
build: ./service2-dir
Let's say that in Dockerfile of service1 we have some folder that we already copy.
Can I pass this folder to docker file of service2?
With another words - can I use multi-stage build technique to pass layers between different Dockerfiles or multi-stage build should be only in one Dockerfile?
You can split things up the way you describe; it'll probably be more robust to do things in two stages in a single Dockerfile, or to do the primary part of your build using host tools.
There are two essential parts to this. The first is that, in your docker-compose.yml file, if you specify both a build: description and an image: name, then Docker Compose will tag the built image for you. The second is that the Dockerfile COPY directive can copy content --from a preceding build stage or from an arbitrary other image. So if your docker-compose.yml says
version: '3'
services:
service1:
build: ./service1-dir
image: me/service1
service2:
build: ./service2-dir
and service2-dir/Dockerfile says
COPY --from=me/service1 /app/somefile .
it will copy content from one image to the other.
The one challenge here is that docker-compose build doesn't specify the order the images are built in. If it builds service2 first, it will get old content from the previous build of service1 (or fail if it's the initial build). To do this reliably, you need to do something like
docker-compose build service1
docker-compose build
docker-compose up -d
If the build sequence isn't too complex, just including it in both Dockerfiles could make sense. It can also work to built whatever artifacts you need on the host, and have your Dockerfiles copy that in as-is instead of building it themselves; this works well especially if that content is platform-neutral (Java .jar files, HTML/Javascript/CSS files from a Webpack build for a browser application).
What is the difference between docker-compose build and docker build?
Suppose in a dockerized project path there is a docker-compose.yml file:
docker-compose build
And
docker build
docker-compose can be considered a wrapper around the docker CLI (in fact it is another implementation in python as said in the comments) in order to gain time and avoid 500 characters-long lines (and also start multiple containers at the same time). It uses a file called docker-compose.yml in order to retrieve parameters.
You can find the reference for the docker-compose file format here.
So basically docker-compose build will read your docker-compose.yml, look for all services containing the build: statement and run a docker build for each one.
Each build: can specify a Dockerfile, a context and args to pass to docker.
To conclude with an example docker-compose.yml file :
version: '3.2'
services:
database:
image: mariadb
restart: always
volumes:
- ./.data/sql:/var/lib/mysql
web:
build:
dockerfile: Dockerfile-alpine
context: ./web
ports:
- 8099:80
depends_on:
- database
When calling docker-compose build, only the web target will need an image to be built. The docker build command would look like :
docker build -t web_myproject -f Dockerfile-alpine ./web
docker-compose build will build the services in the docker-compose.yml file.
https://docs.docker.com/compose/reference/build/
docker build will build the image defined by Dockerfile.
https://docs.docker.com/engine/reference/commandline/build/
Basically, docker-compose is a better way to use docker than just a docker command.
If the question here is if docker-compose build command, will build a zip kind of thing containing multiple images, which otherwise would have been built separately with usual Dockerfile, then the thinking is wrong.
Docker-compose build, will build individual images, by going into individual service entry in docker-compose.yml.
With docker images, command, we can see all the individual images being saved as well.
The real magic is docker-compose up.
This one will basically create a network of interconnected containers, that can talk to each other with name of container similar to a hostname.
Adding to the first answer...
You can give the image name and container name under the service definition.
e.g. for the service called 'web' in the below docker-compose example, you can give the image name and container name explicitly, so that docker does not have to use the defaults.
Otherwise the image name that docker will use will be the concatenation of the folder (Directory) and the service name. e.g. myprojectdir_web
So it is better to explicitly put the desired image name that will be generated when docker build command is executed.
e.g.
image: mywebserviceImage
container_name: my-webServiceImage-Container
example docker-compose.yml file :
version: '3.2'
services:
web:
build:
dockerfile: Dockerfile-alpine
context: ./web
ports:
- 8099:80
image: mywebserviceImage
container_name: my-webServiceImage-Container
depends_on:
- database
Few additional words about the difference between docker build and docker-compose build.
Both have an option for building images using an existing image as a cache of layers.
with docker build, the option is --cache-from <image>
with docker-composer, there is a tag cache_from in the build section.
Unfortunately, up until now, at this level, images made by one are not compatible with the other as a cache of layers (Ids are not compatible).
However, docker-compose v1.25.0 (2019-11-18), introduces an experimental feature COMPOSE_DOCKER_CLI_BUILD so that docker-compose uses native docker builder (therefore, images made by docker build can be used as a cache of layers for docker-compose build)