my docker-compose.yml file is below.
I want to build parallel those images - I am running the a command docker-compose build --parallel
BUT I want to run a command before it builds the images of service2 & service3 while building service1 - parallel.
When the command will be finished it will join to the building-parallel-process.
version: '3.4'
services:
service1:
image: "company/service1:${TAG}"
build:
context: ./folder/service1/
dockerfile: Dockerfile
service2:
image: "company/service2:${TAG}"
build:
context: ./folder/service2/
dockerfile: Dockerfile
service3:
image: "company/service3:${TAG}"
build:
context: ./folder/service3
dockerfile: Dockerfile
Compose doesn't really have any sort of workflow handling like this, especially around building images. It's assumed that building an image only depends on the local source tree and nothing else. Compose also doesn't have any ability to run non-Docker commands or launch temporary containers as part of the up workflow.
The good news is that re-running a build is very quick if nothing has changed. So with the workflow you've described, you might separately build the first image, run the command, and then rebuild everything; re-rebuilding the first image will take almost no time and you won't get a new image.
#!/bin/sh
# Build the one image that needs special handling
docker-compose build service1
# Run the command
the_command
# Rebuild everything in parallel (service1 will be a no-op)
docker-compose build --parallel
If you can run the preparatory step in a Dockerfile RUN command that might be easier to manage. If that needs software that isn't ordinarily part of your image, you could use a multi-stage build to do it in effectively a temporary image.
Related
I am trying to build an image with docker compose and it fails, however it works with just docker. I have read some SO posts saying that the error thrown when failing happens when a file/folder cannot be found in the Dockerfile. The build works when building with docker so I dont know why it wouldn't work with docker-compose. Why is this happening?
The structure for this project is this:
parent_proj
|_proj
|_Dockerfile
|_docker-compose.yml
Here is my docker-compose file:
version: '3.4'
services:
integrations:
build:
context: .
dockerfile: proj/Dockerfile
network: host
image: int
ports:
- "5000:5000"
Here is the Dockerfile inside proj/
FROM openjdk:11
USER root
#RUN apt-get bash
ARG JAR_FILE=target/proj-0.0.1-SNAPSHOT.jar
COPY ${JAR_FILE} /app2.jar
ENTRYPOINT ["java","-jar", "/app2.jar"]
When I'm inside the proj folder. I can run
docker build . -t proj
The above succeeds and I can subsequently run the container. However when I am in parent_proj and run docker compose build it fails with the error message
failed to compute cache key: failed to walk
/var/lib/docker/tmp/buildkit-mount316454722/target: lstat
/var/lib/docker/tmp/buildkit-mount316454722/target: no such file or
directory
Why does this happen? How can I build successfully with docker-compose without restructuring the project?
thanks
Your Compose build options and the docker build options you show are different. The successful command is (where -f Dockerfile is the default):
docker build ./proj -t proj # -f Dockerfile
# context: image: dockerfile:
But your Compose setup is running
docker build . -t img -f proj/Dockerfile
# context: image: dockerfile:
Which one is right? In the Dockerfile, you
COPY target/proj-0.0.1-SNAPSHOT.jar /some/container/path
That target/... source path is always relative to the build-context directory (Compose context: option, the directory parameter to docker build), even if it looks like an absolute path and even if the Dockerfile is in a different directory. If that target directory is a subdirectory of proj then you need the first form.
There's a shorthand Compose build: syntax if the only thing you need to specify is the context directory, and I'd use that here. If you don't specifically care what the image name is (you're not pushing it to a registry) then Compose can pick a reasonable name on its own; you don't need to specify image:.
version: '3.8'
services:
integrations:
build: ./proj
ports:
- "5000:5000"
I have 2 services: nginx and web
When I build web image I build the frontend via the command npm install && npm run build
But I need prepared files in both containers: in the web and in the nginx.
How to share files between containers (images)? I can't simply use volumes, because they will be mounted only in runtime.
The Dockerfile COPY directive can copy files from an arbitrary image. While it's most commonly used in multi-stage builds, you can use it with any image, even one you built yourself.
Say your docker-compose.yml file looks like:
version: '3.8'
services:
web:
build: .
image: my/web
nginx:
build:
context: .
dockerfile: Dockerfile.nginx
ports: [8000:80]
Note that we've explicitly given the web image a name; also notice that there are no volumes: in this setup.
In the proxy image, we can then copy files out of that image:
# Dockerfile.nginx
FROM nginx
COPY --from=my/web /app/static /usr/share/nginx/html
The only complication here is that Compose doesn't know that one image is built off of the other. You'll probably have to manually tell it to rebuild the application image so that it gets built before the proxy image.
docker-compose build web
docker-compose build
docker-compose up -d
You can use this in a more production-oriented setup to deploy this application without having the code directly available. You can create a base docker-compose.yml that names an image: for both containers, and then add a separate docker-compose.override.yml file that has the build: blocks. After running docker-compose build twice as above, you can docker-compose push the built images, and then run this container stack on your production system getting the images from the registry; without a local copy of the source tree and without volumes.
Let's say that we have docker-compose.yml which build few services with Dockerfile's in different locations like this:
version: '3.4'
services:
service1:
build: ./service1-dir
service2:
build: ./service2-dir
Let's say that in Dockerfile of service1 we have some folder that we already copy.
Can I pass this folder to docker file of service2?
With another words - can I use multi-stage build technique to pass layers between different Dockerfiles or multi-stage build should be only in one Dockerfile?
You can split things up the way you describe; it'll probably be more robust to do things in two stages in a single Dockerfile, or to do the primary part of your build using host tools.
There are two essential parts to this. The first is that, in your docker-compose.yml file, if you specify both a build: description and an image: name, then Docker Compose will tag the built image for you. The second is that the Dockerfile COPY directive can copy content --from a preceding build stage or from an arbitrary other image. So if your docker-compose.yml says
version: '3'
services:
service1:
build: ./service1-dir
image: me/service1
service2:
build: ./service2-dir
and service2-dir/Dockerfile says
COPY --from=me/service1 /app/somefile .
it will copy content from one image to the other.
The one challenge here is that docker-compose build doesn't specify the order the images are built in. If it builds service2 first, it will get old content from the previous build of service1 (or fail if it's the initial build). To do this reliably, you need to do something like
docker-compose build service1
docker-compose build
docker-compose up -d
If the build sequence isn't too complex, just including it in both Dockerfiles could make sense. It can also work to built whatever artifacts you need on the host, and have your Dockerfiles copy that in as-is instead of building it themselves; this works well especially if that content is platform-neutral (Java .jar files, HTML/Javascript/CSS files from a Webpack build for a browser application).
What is the difference between docker-compose build and docker build?
Suppose in a dockerized project path there is a docker-compose.yml file:
docker-compose build
And
docker build
docker-compose can be considered a wrapper around the docker CLI (in fact it is another implementation in python as said in the comments) in order to gain time and avoid 500 characters-long lines (and also start multiple containers at the same time). It uses a file called docker-compose.yml in order to retrieve parameters.
You can find the reference for the docker-compose file format here.
So basically docker-compose build will read your docker-compose.yml, look for all services containing the build: statement and run a docker build for each one.
Each build: can specify a Dockerfile, a context and args to pass to docker.
To conclude with an example docker-compose.yml file :
version: '3.2'
services:
database:
image: mariadb
restart: always
volumes:
- ./.data/sql:/var/lib/mysql
web:
build:
dockerfile: Dockerfile-alpine
context: ./web
ports:
- 8099:80
depends_on:
- database
When calling docker-compose build, only the web target will need an image to be built. The docker build command would look like :
docker build -t web_myproject -f Dockerfile-alpine ./web
docker-compose build will build the services in the docker-compose.yml file.
https://docs.docker.com/compose/reference/build/
docker build will build the image defined by Dockerfile.
https://docs.docker.com/engine/reference/commandline/build/
Basically, docker-compose is a better way to use docker than just a docker command.
If the question here is if docker-compose build command, will build a zip kind of thing containing multiple images, which otherwise would have been built separately with usual Dockerfile, then the thinking is wrong.
Docker-compose build, will build individual images, by going into individual service entry in docker-compose.yml.
With docker images, command, we can see all the individual images being saved as well.
The real magic is docker-compose up.
This one will basically create a network of interconnected containers, that can talk to each other with name of container similar to a hostname.
Adding to the first answer...
You can give the image name and container name under the service definition.
e.g. for the service called 'web' in the below docker-compose example, you can give the image name and container name explicitly, so that docker does not have to use the defaults.
Otherwise the image name that docker will use will be the concatenation of the folder (Directory) and the service name. e.g. myprojectdir_web
So it is better to explicitly put the desired image name that will be generated when docker build command is executed.
e.g.
image: mywebserviceImage
container_name: my-webServiceImage-Container
example docker-compose.yml file :
version: '3.2'
services:
web:
build:
dockerfile: Dockerfile-alpine
context: ./web
ports:
- 8099:80
image: mywebserviceImage
container_name: my-webServiceImage-Container
depends_on:
- database
Few additional words about the difference between docker build and docker-compose build.
Both have an option for building images using an existing image as a cache of layers.
with docker build, the option is --cache-from <image>
with docker-composer, there is a tag cache_from in the build section.
Unfortunately, up until now, at this level, images made by one are not compatible with the other as a cache of layers (Ids are not compatible).
However, docker-compose v1.25.0 (2019-11-18), introduces an experimental feature COMPOSE_DOCKER_CLI_BUILD so that docker-compose uses native docker builder (therefore, images made by docker build can be used as a cache of layers for docker-compose build)
Here's a typical docker-compose file. I use is it both for building image (docker-compose build) and to run my tests (docker-compose run test ).
version: '2'
services:
test :
links:
- web
cmd : "mvn clean verify"
web:
image: my_repo/my_image:tag
build: .
When I use the run command docker-compose try to build the image before running the test.
Is there anyway to force it to pull existing image instead of trying to build new one ?
You can use "pull" command before run. There is pull all new images from registry
docker-compose pull
docker-compose run
Both of your solutions work fine.
I was just expecting to have to have some thing like
'docker run test --pull' or 'docker dun test --build' to force the pull/build.
Thanks !
It's normal that it's build the web image before creating the test container, because there's link between (web depends on test). If you want to not do the build each time you run docker-compose up, start by creating your web image:
docker build -t web .
then update your Dockerfile with the new image:
version: '2'
services:
test :
links:
- web
cmd : "mvn clean verify"
web:
image: web