Use console output in Dockerfile - docker

I want to use some console output as the name of my base docker image.
Specifically, I have a chain of dependent docker build files so I am trying to automate this process. So for instance, the Dockerfile of one image derived1 depends on the base image base_image_namein the following scenario:
base_image_name/
Dockerfile
derived1/
Dockerfile
derived2/
Dockerfile
When the base image builds, it grabs its name from its current folder by using ${PWD##*/}. In this case, the base image's folder is called base_image_name, and so the base image is called company:base_image_name.
Then when the derived images build, they should just be able to figure out the base image's name by moving up a directory and looking at that directories name. So for instance, when build the company:derived1 image builds, it should look up one directory, see that it is called base_image_name, and from that infer that it should use the base image company:base_image_name.
I would like to have this structure several layers deep, so I want to automate it. To do that, I have tried several permutations of the syntax
FROM company:$(cd $PWD/../; echo ${PWD##*/})
but I can't seem to get it right. To understand what the command $(cd $PWD/../; echo ${PWD##*/}) is doing, just type it into your terminal..
echo $(cd $PWD/../; echo ${PWD##*/})
simply returns the name of the directory one level up. However, when I try to use this in a Dockerfile, I get the error
Error response from daemon: Dockerfile parse error line 1: FROM requires either one or three arguments
Could somebody please provide me with the correct syntax?
EDIT:
I also tried building the derived images with a build-arg, but that doesn't seem to work either:
build.sh:
BASE=$(cd $PWD/../../; echo ${PWD##*/})
echo "BASE="$BASE
docker build --build-arg BASE=${BASE} -t company:"${PWD##*/}" .
where the Dockerfile looks like
FROM company:$BASE
Specifically, this yields the build error:
BASE=base_image_name
Sending build context to Docker daemon 5.12kB
Step 1/3 : FROM company:$BASE
invalid reference format
So it seems that docker is not interpretting that build arg correctly.

Dockerfiles don't support shell syntax in general, except for some very limited environment variable expansion.
They do support ARGs that can be passed in from the command line, and an ARG can be used to define the image FROM. So you could start your Dockerfile with
ARG tag
FROM company:${tag:-latest}
and then build the image with
docker build --build-arg tag=$(cd $PWD/../; echo ${PWD##*/}) .
(which is involved enough that you might want to write it into a shell script).
At a very low level, it's also worth remembering that docker build works by making a tar file of the current directory, sending it across an HTTP connection to the Docker daemon, and running the build there. Once that process has happened, any notion of the host directory name is lost. In other words, even if the syntax worked, docker build also doesn't have the capability to know which host directory the Dockerfile is in.

Aha. Found it.
As Jonathon points out, it seems as though you can't easily pull stuff in from your environment into the build system. It seems that you must use Docker build-args.
The solution was to evaluate the variable in the terminal and pass that as a build-arg:
build.sh:
BASE=$(cd $PWD/../; echo ${PWD##*/})
echo "BASE="$BASE
docker build --build-arg BASE=${BASE} -t company:"${PWD##*/}" .
Then inside the Dockerfile of the derived image:
ARG BASE
FROM company:$BASE

You're trying to use bash command substitution in something that isn't consumed by bash.
The [Dockerfile reference[(https://docs.docker.com/engine/reference/builder/) indicates that environment variable substitution is supported by the FROM instruction.
You'll need to instead simply use an environment variable in FROM that you compute outside of the Dockerfile and pass to docker build.

Related

How to run a tensorflow/tfx container?

I am new to docker, and have downloaded the tfx image using
docker pull tensorflow/tfx
However, I am unable to find anywhere how to successfully launch a container for the same.
here's a naive attempt
You can use docker image ls to get a list of locally-built Docker images. Note that an "image" is a template for a VM.
To instantiate the VM and shell into it, you would use a command like docker run -t --entrypoint bash tensorflow/tfx. This spins up a temporary VM based on the tensorflow/tfx image.
By default, Docker assumes you want the latest version of that image stored on your local machine, i.e. tensorflow/tfx:latest in the list. If you want to change it, you can reference a specific image version by name or hash, e.g. docker run -t --entrypoint bash tensorflow/tfx:1.0.0 or docker run -t --entrypoint bash fe507176d0e6. I typically use the docker image ls command first and cut & paste the hash, so my notes can be specific about which build I'm referencing even if I later edit the relevant Dockerfile.
Also note that changes you make inside that VM will not be saved. Once you exit the bash shell, it goes away. The shell is useful for checking the state & file structure of a constructed image. If you want to edit the image itself, use a Dockerfile. Each line of a Dockerfile creates a new image when the Dockerfile is compiled. If you know that something went wrong between lines 5 and 10 of the Dockerfile, you can potentially shell into each of those images in turn (with the docker run command I gave above) to see what went wrong. Kinda tedious, but it works.
Also note that docker run is not equivalent to running a TFX pipeline. For the latter, you want to look into the TFX CLI commands or otherwise compile the pipeline - and probably upload it to an external Kubeflow server.
Also note that the Docker image is just a starting point for one piece of your TFX pipeline. A full pipeline will require you to specify the components you want, a more-complete Dockerfile, and more. That's a huge topic, and IMO, the existing documentation leaves a lot to be desired. The Dockerfile you create describes the image which will be distributed to each of the workers which process the full pipeline. It's the place to specify dependencies, necessary files, and other custom setup for the machine. Most ML-relevant concerns are handled in other files.

Copy file into Dockerfile from different directory

Is it possible for Dockerfile to copy over some file from the host filesystem and not from the context it's being build from ?
# Inside Dockerfile
FROM gradle:latest
COPY ~/.super/secrets.yaml /opt
# I think you can work around it with but doesn't look nice
COPY ../../../../../../.super/secrets.yaml /opt
when I ran the command on the /home/user/some/path/to/project/ path ?
docker build .
The usual way to get "external" files into your docker container is by copying them into your build directory before starting the docker build. It is strongly recommended to create a script for this to ensure that your preparation step is reproducible.
No this is not possible to go up the directory. Here is why.
When runnig docker build . have you ever considered what this dot stand for at the end? Well, here is part of docker documentation
The docker build command builds Docker images from a Dockerfile and a
“context”. A build’s context is the set of files located in the
specified PATH or URL. The build process can refer to any of the files
in the context. For example, your build can use a COPY instruction to
reference a file in the context.
As you see, this dot is referencing context path (here it means "this directory"). All files under context path get sent to docker daemon and you can reference only these files in your Dockerfile. Of course, you can think that you are clever and reference / (root path) so you have access to all files on your machine. (I highly encourage you to try this and see what happens). What you should see that happens is that docker client is freezing. Or, does it really? Well, it's not really freezing, its sending all / directory to docker daemon and it can take ages or (what's more probable) you may run out of memory.
So now when you understand this limitation you see that the only way to make it work is to copy the file you are interested in to the location of context path and then run docker build command.
When you do docker build ., that last argument is the build context directory: you can only access files from it.
You could do docker build ../../../ but then every single file in that root directory will get packaged up and send to the Docker daemon, which will be slow.
So instead, do something like:
cp ../../../secret.yaml .
docker build .
rm secret.yaml
However, keep in mind that will result in the secret being embedded in the image forever, which might be a security risk. If it's a secret you need for runtime, better to pass it in via environment variable at runtime. If you only need the secret for building the image, there are other alternatives, e.g. https://pythonspeed.com/articles/docker-build-secrets/.

Can a variable be used in docker FROM?

I am wondering if a env variable can be used in a docker from? Reason for this is to control the tagging. For example, say I have this line in my Dockerfile:
FROM myApp
What I want is this:
FROM myApp:${VERSION}
This way I can say docker build . myApp --build-arg VERSION=9
The process to build docker images for this app is the same. I don't want to have Dockerfiles that are almost identical just to use a different base image.If I want to build version 9, it should use version 9 of the base image.
Quoting this link
:
This is now possible if anyone comes here looking for answers: https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
FROM instructions support variables that are declared by any ARG instructions that occur before the first FROM.
ARG CODE_VERSION=latest
FROM base:${CODE_VERSION}
CMD /code/run-app
FROM extras:${CODE_VERSION}
CMD /code/run-extras
For at least this docker version this is feasible
docker --version
docker version 18.09.8, build bfed4f5
It requires a preset variable in Dockerfile e.g.
ARG TAG=latest
FROM traefik:${TAG}
Then you can override this preset with the following
docker build --build-arg "TAG=2.2.8" -t my-app:$TAG
The version number will not show up during build. if you want to test if it works, reference a non-existing version - it will fail with: manifest my-app:version not found.
You could simply generate your Dockerfile from a template. Put
something like this in a Makefile:
MYTAG=latest
.PHONY: Dockerfile
Dockerfile: Dockerfile.in
sed 's/MYTAG/$(MYTAG)/' $< > $# || rm -f $#
Then you can run:
make MYTAG=8; docker build -t my-app-8 .
This would only make sense if you are frequently building images that
require a different tag in the FROM line.
It is not possible.
Although, you can use a variable tag like from myApp:latest and overwrite the latest tag when you're creating a new version.
Build your container programatically using buildah (It can take Dockerfile too).
So for your use-case:
VERSION=v0.1.0
myCon=$(buildah from myApp:${VERSION})
buildah config --cmd "sleep 1d" $myCon
buildah commit $myCon $USER/sleeping1d
You can obviously script it, save and invoke it, and one more advantage is
buildah doesn't need docker daemon running, which is great for CI. Also it's an open-source project, check out the project page.
BTW I saw this issue lately which is exactly what you want - https://github.com/projectatomic/buildah/issues/581
Unfortunately it's not possible to do that. The first line of your Dockerfile must be a FROM directive, and so that precludes the use of the ARG directive. There is a good answer there from larsks about generating a Dockerfile, but I'd also like to suggest merely creating different Dockerfiles and then specifying a particular one in your docker build command using the -f switch:
docker build -t codemiester/app:latest -f ./Dockerfile.apache2.ubuntu

After building a docker image how can I run the image without pushing to docker hub?

The docs are not very clear to me. I run docker build -f . in the Dockerfile directory, it seems to build successfully, great. I am not sure what to do next, I take a look at the docs https://docs.docker.com/engine/reference/builder/ it tells me When you’re done with your build, you’re ready to look into Pushing a repository to its registry.
Which takes me here https://docs.docker.com/engine/userguide/containers/dockerrepos/#contributing-to-docker-hub I have no interest in publishing it or creating an account.
My other option is to name the build I guess, right? If I run docker build -t <nodebb> . then I get file exists: .. If I run docker build -f <nodebb> . then I get no such file or directory: nodebb.
So I am kind of lost, I wish I could understand the Docs better but I don't and would appreciate the guidance. Thanks!
Your issue with tagging: You cannot use <> in tags
The error that you saw was because '<' and '>' are interpreted by bash. docker build -t <nodebb> . tries to do the following:
It reads the file nodebb and pipes it into docker build -t.
It takes the output of docker build -t < nodebb and writes it to the file .
This fails for several reasons:
-t expects an argument, the tag name
the file . already exists (that's the error you saw)
Generally, you can avoid this by escaping the argument:
docker build -t "<nodebb>" .
However, this will result in another error:
Error parsing reference: "<nodebb>" is not a valid repository/tag
For good reasons, <> are not allowed in tag names. Instead, use a valid tag:
docker build -t nodebb .
Running an image without a tag
You can also run a built image without a tag; if you build an image, the last line will always be something like this:
Successfully built 028edf7c13d1
You can run that image with docker run 028edf7c13d1.

Labelling images in docker

I've got a jenkins server monitoring a git repo and building a docker image on code change. The .git directory is ignored as part of the build, but I want to associate the git commit hash with the image so that I know exactly what version of the code was used to make it and check whether the image is up to date.
The obvious solution is to tag the image with something like "application-name-branch-name:commit-hash", but for many develop branches I only want to keep the last good build, and adding more tags will make cleaning up old builds harder (rather than using the jenkins build number as the image is built, then retagging to :latest and untagging the build number)
The other possibility is labels, but while this looked promising initially, they proved more complicated in practice..
The only way I can see to apply a label directly to an image is in the Dockerfile, which cannot use the build environment variables, so I'd need to use some kind of templating to produce a custom Dockerfile.
The other way to apply a label is to start up a container from the image with some simple command (e.g. bash) and passing in the labels as docker run arguments. The container can then be committed as the new image. This has the unfortunate side effect of making the image's default command whatever was used with the labelling container (so bash in this case) rather than whatever was in the original Dockerfile. For my application I cannot use the actual command, as it will start changing the application state.
None of these seem particularly ideal - has anyone else found a better way of doing this?
Support for this was added in docker v1.9.0, so updating your docker installation to that version would fix your problem if that is OK with you.
Usage is described in the pull-request below:
https://github.com/docker/docker/pull/15182
As an example, take the following Dockerfile file:
FROM busybox
ARG GIT_COMMIT=unknown
LABEL git-commit=$GIT_COMMIT
and build it into an image named test as anyone would do naïvely:
docker build -t test .
Then inspect the test image to check what value ended up for the git-commit label:
docker inspect -f '{{index .ContainerConfig.Labels "git-commit"}}' test
unkown
Now, build the image again, but this time using the --build-arg option:
docker build -t test --build-arg GIT_COMMIT=0123456789abcdef .
Then inspect the test image to check what value ended up for the git-commit label:
docker inspect -f '{{index .ContainerConfig.Labels "git-commit"}}' test
0123456789abcdef
References:
Docker build command documentation for the --build-arg option
Dockerfile reference for the ARG directive
Dockerfile reference for the LABEL directive
You can specify a label on the command line when creating your image. So you would write something like
docker build -t myproject --label "myproject.version=githash" .
instead of hard-coding the version you can also get it directly from git:
docker build -t myproject --label "myproject.version=`git describe`" .
To read out the label from your images you can use docker inspect with a format string:
docker inspect -f '{{index .Config.Labels "myproject.version"}}' myproject
If you are using docker-compose, you could add the following to the build section:
labels:
git-commit-hash: ${COMMIT_HASH}
where COMMIT_HASH is your environment variable, which holds commit hash.

Resources