Makefile run same target multiple time concurrently with different option - docker

I wish to build multiple docker images through my makefile. I have a make target looking like this:
docker:
docker build -t service1:latest -f ./service1/Dockerfile .
docker build -t service2:latest -f ./service2/Dockerfile .
...
To gain time, I want to run them in parallel, so I wanted to update my makefile like this:
docker:
docker build -t $(SERVICE):latest -f ./$(SERVICE)/Dockerfile .
And calling it with something which would look like this:
make -j=2 SERVICE=service1 docker SERVICE=service2 docker
But obviously it does not work since there is multiple issues with this.
I was thinking to use the pattern %, but I am not quite sure how to achieve this cleanly.
What would be the right way to achieve this?

You could write something like this:
IMAGES = \
service1 \
service2
all: $(IMAGES)
.PHONY: $(IMAGES)
$(IMAGES):
docker build -t $#:latest -f $#/Dockerfile .
The .PHONY directive is necessary because otherwise it will find the directory named service1 or service2 and decide that the target does not need updating. .PHONY tells it to ignore this and build the target in any case.
Using this Makefile, if I run make -j it spawns to build processes in parallel.
While this works, I'm not sure that make is really the right tool for the job. The idea behind make is that it will only rebuild those things that need to be rebuilt, saving time if only a few things have been modified.
In this situation, make doesn't really have any way to make that sort of decisions.
Since you want to rebuild everything every time, you might be better off with a simple shell script and xargs:
#!/bin/bash
seq 2 |
xargs -iSERVICENUM -P0 docker build -t serviceSERVICENUM -f serviceSERVICENUM/Dockerfile .
Or if your services aren't actually named in a numeric sequence:
#!/bin/bash
SERVICES='
foo
bar
'
xargs -iSERVICE -P0 docker build -t SERVICE -f SERVICE/Dockerfile . <<<$SERVICES

Related

Why is docker build not showing any output from commands?

Snippet from my Dockerfile:
FROM node:12.18.0
RUN echo "hello world"
RUN psql --version
When I run docker build . I don't see any output from these two commands even if they are not cached. The documentation says that docker build is verbose by default. Why am I not seeing the output from commands? I used to see them before.
The output while building:
=> [7/18] RUN echo "hello world" 0.9s
The output I am seeing after building finishes:
=> CACHED [6/18] RUN apt-get install postgresql -y 0.0s
=> [7/18] RUN echo "hello world" 6.4s
=> [8/18] RUN psql --version 17.1s
The Dockerfile is created from node:12.18.0 which is based on Debian 9.
Docker version 19.03.13, build 4484c46d9d.
The output you are showing is from buildkit, which is a replacement for the classic build engine that docker ships with. You can adjust output from this with the --progress option:
--progress string Set type of progress output (auto, plain, tty). Use plain to show container output
(default "auto")
Adding --progress=plain will show the output of the run commands that were not loaded from the cache. This can also be done by setting the BUILDKIT_PROGRESS variable:
export BUILDKIT_PROGRESS=plain
If you are debugging a build, and the steps have already been cached, add --no-cache to your build to rerun the steps and redisplay the output:
docker build --progress=plain --no-cache ...
If you don't want to use buildkit, you can revert to the older build engine by exporting DOCKER_BUILDKIT=0 in your shell, e.g.:
DOCKER_BUILDKIT=0 docker build ...
or
export DOCKER_BUILDKIT=0
docker build ...
Just use this flag --progress=plain after build.
For example:
docker-compose build --progress=plain <container_name>
OR
docker build --progress=plain .
If you don't want to use this flag every time, then permanently tell docker to use this flag by doing:
export BUILDKIT_PROGRESS=plain
Here is the official documentation when you type docker build --help.
--progress string Set type of progress output (auto, plain, tty). Use plain to show container output (default "auto")
In Docker 20.10 i had to use the --no-cache flag, too. Otherwise cached output is not shown.
docker build --progress=plain --no-cache .
As an alternative to specifying the --progress=plain option, you can also permanently disable the "pretty" output by setting this env variable in your shell config:
export BUILDKIT_PROGRESS=plain
Do 2 things
Instead of docker build . use this
docker build . --progress=plain
Add random junk to your RUN command every build (this tricks docker into thinking it hasn't seen the command before, so it doesn't use the cached version)
Example. If your command is RUN ls use this instead RUN ls && echo sdfjskdflsjdf (change the sdfjskdflsjdf to something else each time you build).
Why this works
I tried other answers and they all presented problems and imperfections. It's highly frustrating that Docker doesn't have some simple functionality like --verbose=true.
Here's what I ended up using (it's ludicrous but it works).
Suppose you want to see the output of ls command, this won't work docker build .
RUN ls
but this will print the output docker build --progress=plain:
RUN ls
now try again, it won't print! - that's because docker caches the unchanged layer, so the trick is to alter the command each time by adding some nonsense to it && echo sdfljsdfljksdfljk, and changing the nonsense each time docker build --progress=plain:
# This prints
RUN ls && echo sdfljsdfljksdfljk
# Next time you run it use a different token
RUN ls && echo sdlfkjsldfkjlskj
So each and every time, I mash the keyboard and come up with a new token. Stupifying. (note that I tried something like && openssl rand -base64 12 to generate a random string, but docker realises the code hasn't changed that doesn't work).
This solution is highly inferior to genuine docker support for printing output to console.
If your error looks something like this:
#7 0.584 /bin/sh: 1: /install.sh: not found
it's telling you the error is in line number 1. you are running into windows line endings
I was using VS code and I solved it pretty easily by converting the file from CRLF to LF using VS code.
just click on the CRLF button in the bottom right corner of the editor and save the file.
everything should work fine when you build the image now.

Make: dependency on a docker image

I'm using a docker workflow to generate some files, based on a given spec file, with the Makefile being (it's generating a client according to an OpenAPI spec):
SWAGGER ?= ${PWD}/swagger.yaml
GENERATOR ?= openapitools/openapi-generator-cli\:latest
generated: Makefile ${SWAGGER}
docker run --rm --user $$(id -u):$$(id -g) \
-v ${PWD}:/output -v ${SWAGGER}:/input/swagger.yaml \
${GENERATOR} \
generate -g python -i /input/swagger.yaml -o /output/generated \
this works fine, and will rebuild if I modify the input SPEC file.
But it doesn't rebuild when the docker image is changed.
Let's say I docker build the image with the same name:tag again, but with different code inside, or I use a different tagged version of the upstream image, whatever. This is kind of expected because the Makefile has no knowledge of the docker image's content or modification date. How can I make the Makefile understand the dependency on the docker image ?
I've tried to docker inspect the image to fetch the creation date, but I don't know how to make make understand that as a dependency (if the creation date is newer than the output dir, then rebuild)
I can't just add a dependency on the code inside the docker image, because the docker image might not even have been built from locally available files.
make might not be the tool for that kind of thing, maybe there is something else that I could use that understands the docker image dependency.
Whether the artifact you depend on is a Docker image or some service/data somewhere else, you need to represent what you know about it, in the form of a regular file. Because make only deals with files. :-)
I'd recommend you define some macro for "invoke docker build and save evidence of its success to a marker file with predictable name" so that you can use that to replace all calls to docker build, ensure consistent file handling. Full-blown example, assuming a/Dockerfile and b/Dockerfile exist.
# For consistency, the src dirs are named like the images they produce
IMAGES = a b
# Keep "stamps" around, recording that images were built.
# You could keep them in e.g. a `.docker-buildstamps/*` dir,
# but this example uses `*/docker-buildstamp`.
BUILDSTAMP_FILE = docker-buildstamp
BUILDSTAMPS = $(addsuffix /$(BUILDSTAMP_FILE),$(IMAGES))
.PHONY: all
all: $(BUILDSTAMPS)
# Pattern rule: let e.g. `a/docker-buildstamp` depend on changes to `a/*` (-but avoid circular dep)
%/$(BUILDSTAMP_FILE): % %/[!$(BUILDSTAMP_FILE)]*
$(docker_build)
clean:
docker image rm -f $(IMAGES)
rm -f $(BUILDSTAMPS)
# Turn `a/docker-buildstamp` back into `a`
define from_buildstamp
$(#:%/$(BUILDSTAMP_FILE)=%)
endef
# Self-explanatory
define docker_build
docker build -t $(from_buildstamp) $(from_buildstamp)
touch $#
endef

Dockerfile capture output of a command

I have the following line in my Dockerfile which is supposed to capture the display number of the host:
RUN DISPLAY_NUMBER="$(echo $DISPLAY | cut -d. -f1 | cut -d: -f2)" && echo $DISPLAY_NUMBER
When I tried to build the Dockerfile, the DISPLAY_NUMBER is empty. But however when I run the same command directly in the terminal I get the see the result. Is there anything that I'm doing wrong here?
Commands specified with RUN are executed when the image is built. There is no display during build hence the output is empty.
You can exchange RUN with ENTRYPOINT then the command is executed when the docker starts.
But how to forward the hosts display to the container is another matter entirely.
Host environment variables cannot be passed during build, only at run-time.
Only build args can be specified by:
first "declaring the arg"
ARG DISPLAY_NUMBER
and then running
docker build . --no-cache -t disp --build-arg DISPLAY_NUMBER=$DISPLAY_NUMBER
You can work around this issue using the envsubst trick
RUN echo $DISPLAY_NUMBER
And on the command line:
envsubst < Dockerfile | docker build . -f -
Which will rewrite the Dockerfile in memory and pass it to Docker with the environment variable changed.
Edit: Note that this solution is pretty useless though, because you probably
want to do this during run-time anyways, because this value should depend on not on where the image is built, but rather where it is run.
I would personally move that logic into your ENTRYPOINT or CMD script.

Conditionally set ENV var based on hostname in Dockerfile

How can I set an ENV var in my Dockerfile based on the hostname? I tried this:
RUN if [ hostname = "foo" ]; then ENV BAR "BAZ"; else ENV BAR "BIFF"; fi
But that failed with
ENV: not found
RUN if [ hostname = "foo" ]; then ENV BAR "BAZ"; else ENV BAR "BIFF"; fi
You can't nest docker build instructions, everything after the RUN instruction gets executed in the image context, docker build commands don't
exist there. So that explains the error you are seeing.
Even you if you translated that to proper shell code BAR would only be active for that single RUN instruction during the build.
Either orchestrate on the host and pass BAR via run -e to your container or add a startup script to the image that sets BAR as needed on container start:
FROM foo
COPY my-start.sh /
CMD ["/my-start.sh"]
First of all, you can't embed Docker build command into shell of RUN, the shell will run inside the intermediate container during build process, and Docker build commands will be ran by Docker build engine, they're different things. And besides, Docker does not support conditional commands like IF or something like that. Docker is about immutable infrastructure, Dockerfile is the definition of your image and it's supposed to be able to generate the same image no matter what build context it is in. And from the delivery perspective of view, the image is your deliverable build artifacts, if you want to deliver different stuff, then use different Dockerfile to build different images, otherwise if the differences is about the runtime, I think you could really consider postpone the env definition to the runtime with -e option of docker run.
The reason why your build is failing has been explained by #shizhz & #Erik Dannenberk.
However, if you do really need that behavior I suggest you make a little script to do that:
export BAR=`[[ hostname = "foo" ]] && echo "BAZ" || echo "BIFF"`
docker build -t hello/hi - <<EOF
FROM alpine
ENV BAR $BAR
CMD echo $BAR
EOF

Getting Docker Container Id in Makefile to use in another command

I have a rule in my Makefile. Within this rule I need to manipulate some docker specific things so I need to get the id of the container in a portable way. In addition, I am using Docker Compose. Here is what I have that doesn't work.
a-rule: some deps
$(shell uuid="$(docker-compose ps -q myService)" docker cp "$$uuid":/a/b/c .)
I receive no errors or output, but I do not get a successful execution.
My goal is to get the uuid of the container that myService is running in and then use that uuid to copy a file from the container to my docker host.
edit:
the following works, but I'm still wondering if its possible to do inline variable settings
uuid=$(shell docker-compose ps -q myService)
a-rule: some deps
docker cp "$(uuid)":/a/b/c .
I ran into the same problem and realised that makefiles take output from shell variables with the use of $$. So I tried that and this should work for you:
a-rule: some deps
uuid=$$(docker-compose ps -q myService);\
docker cp "$$uuid":/a/b/c .
Bit late to the party but I hope this helps someone.

Resources