This question is a little like this question. How to pass GitLab CI file variable to Dockerfile and docker container?
I used the most voted answer however I failed.
The below is what I want to do.
I set up the gitlab ci variable called PIP_CONFIG with file type.
and the value is like
[global]
timeout = 60
in .gitlab-ci.yml
...
- docker build -t $IMAGe
--build-arg PIP_CONFIG=$PIP_CONFIG
src/.
...
in Dockerfile
FROM python:3
ARG PIP_CONFIG
RUN mkdir ~/.pip
RUN echo $PIP_CONFIG > ~/.pip/pip.conf
...
...
RUN pip install -r requirements.txt
And then I got an error
Configuration file could not be loaded.
File contains no section headers.
file: '/root/.pip/pip.conf', line: 1
'/builds/xxx/xxx/project_name.tmp/PIPCONFIG\n' <<<< this line
...
It seems like it only wrote the path of temp file rather than the content of file.
I also try use COPY $PIP_CONFIG ~/.pip/pip.conf but it said the path is not exsist in /builds/xxx/xxx/project_name.tmp/PIPCONFIG.
Could someone tell me what should I do correctly? Thanks.
PS: The reason why I do not write the config directly in repository and jus use COPY from repo is that there is some sensitive token in pip config.
After some try, I understand that just use type 'variable' in the gitlab ci setting.
And then pass the value with quote("$VARABLE") for maybe you have the space or break line in your value.
like this
in .gitlab-ci.yml
...
- docker build -t $IMAGe
--build-arg PIP_CONFIG="$PIP_CONFIG"
src/.
...
Remember add quote for Dockerfile, too.
FROM python:3
ARG PIP_CONFIG
RUN mkdir ~/.pip
RUN echo "$PIP_CONFIG" > ~/.pip/pip.conf
...
...
RUN pip install -r requirements.txt
And then, I can do what I want, write a pip config to image from gitlab ci variable.
Hope can help others.
Related
We have a couple of DockerFiles that runs the same command
COPY --from=/source /dest
Since we will change the /source from time to time, is it possible to extract this into some common util file and use it everywhere?
You could try to pass it as an ARG to docker build.
docker build --build-arg mysourcedir=/source Dockerfile
and in the Dockerfile
# use /source as default if no arg is supplied
ARG mysourcedir=/source
...
COPY --from=$mysourcedir /dest
...
Or if you use docker compose then you could put this as an environment variable into a .env file and pass it with the --env-file flag. Details can be found in the documentation.
I use this docker build - < Dockerfile -t deepface to build a docker image.
When I runin command it show Error:
> ERROR [3/4] COPY ./requirements.txt /requirements.txt
> 0.0s
> ------
> > [3/4] COPY ./requirements.txt /requirements.txt:
> ------ failed to compute cache key: "/requirements.txt" not found: not found
My Director File is
>Deepface
|->Dockerfile
|->requirements.txt
My requirements.txt is
numpy==1.19.5
pandas==1.2.4
gdown==3.13.0
tqdm==4.60.0
Pillow==8.2.0
opencv-python==4.5.2.52
tensorflow==2.5.0
keras==2.4.3
Flask==2.0.1
matplotlib==3.4.2
deepface==0.0.53
and my Dockerfile is
FROM python:3.9
WORKDIR /code
COPY ./requirements.txt /requirements.txt
RUN pip install -r ./requirements.txt
How can I solve this problem?
This could be related to this BuildKit docker issue.
In order to see if this is indeed the problem, try building with BuildKit disabled:
$ DOCKER_BUILDKIT=0 docker build ...
If this helps, then you can try one of these for a permanent fix:
Update docker (or follow the above linked GitHub issue to see if it is fixed)
Disable BuildKit globally by adding
{ "features": { "buildkit": true } }
to /etc/docker/daemon.json
(or c:\Users\CURRENT_USER\.docker\daemon.json on Windows).
As a side note, I would recommend avoiding copying requirements.txt to the root folder of the container. Use a subdirectory, such as /app and use WORKDIR in your Dockerfile to make it the base directory.
As a secondary side note - instead of running
docker build - < Dockerfile ... you can just run
docker build ...
The particular docker build syntax you use
docker build - <Dockerfile
has only the Dockerfile; nothing else is in the Docker build context, and so you can't COPY anything into the image. Even though this syntax is in the docker build documentation I wouldn't use it.
A more typical invocation is to just specify the current directory as the build context:
docker build -t deepface .
(Don't forget to also COPY your application code into the image, and set the standard CMD the container should run.)
1. WHY?
Whenever you piped through STDIN Dockerfile you can't use ADD and COPY instructions within local paths.
There is a trick. Docker can use paths only in scope of so called context.
Context is a directory you specify for docker build. E.g. docker build my-docker-context-dir. But as long as you use STDIN instead of directory there is no directory.
In this case docker is absolutely blind to everything but the contents of Dockerfile. Read this official Build with -
Perhaps its also worth reading whole docker build section. Frankly at first I also skipped it, and got some pitfalls just like you.
2. What to do?
Whenever you want to put some files into docker image, you have to create a context directory.
So your directory structure should be like this:
>Deepface
|->Dockerfile
|->context-dir
|->requirements.txt
Now you can call docker build as follows (note, there is a -t option as proposed by David Maze):
cd Deepface
docker build -t deepface -f Dockerfile context-dir
For some reason the file may be in the .dockerignore file, please check if the file is not there.
I was trying to run a container with kvm, using the code I found here: https://github.com/jessfraz/dockerfiles/tree/master/kvm
I created a new directory, cd'd into it and created the dockerfile and start.sh files.
When I gave order to build, it outputted the following error message:
Sending build context to Docker daemon 3.584kB
Error response from daemon: No build stage in current context
I have no idea what this means and I couldn't Google an answer.
Any help?
Does your dockerfile have a: FROM repo/image
As the first line? I got this error when I forgot to specify the docker image that I was building from.
Even if you're building a "source image" you still need to specify FROM scratch as the first line of the dockerfile.
This usually happens because of the text that is written before the FROM command. Try removing the comments in your dockerfile and build again.
For reference https://github.com/moby/buildkit/issues/164
This message appears when you declare an environment variable (ENV) before declaring FROM.
For example:
# Define variables.
ARG PORT
ENV SERVER_PORT=$PORT
# Install minimal Python 3.
FROM python:3.7-alpine
# Install Python requirements.
COPY requirements.txt /
RUN pip install -r /requirements.txt
# Copy app source code.
COPY src/ /app
...
To resolve this, swap the declarations so that any environment variables are set after FROM.
# Install minimal Python 3.
FROM python:3.7-alpine
# Define variables.
ARG PORT
ENV SERVER_PORT=${PORT}
# Install Python requirements.
COPY requirements.txt /
RUN pip install -r /requirements.txt
# Copy app source code.
COPY src/ /app
...
According to the documentation on docs.docker.com, the first non-comment line of your Dockerfile must be the FROM line. To quote the docs:
The FROM instruction initializes a new build stage and sets the Base Image for subsequent instructions. As such, a valid Dockerfile must start with a FROM instruction.
If you are still using the deprecated MAINTAINER keyword, you must have the FROM command as the first command in the Dockerfile:
BAD:
MAINTAINER your name "your#email.com"
FROM dockerimagename
GOOD:
FROM dockerimagename
MAINTAINER your name "your#email.com"
The problem is resolved.
When I went to dockerfile to edit the code I noticed that I accidentally uncommented the first line. Stupid mistake, I know.
Thank you both for the help.
It was my case because I had ENV specified before FROM and as already mentioned, the FROM should be the first expression in your Dockerfile.
BUT
Since this PR https://github.com/moby/moby/pull/31352 you can specify ARG before FROM which might be suitable alternative for you.
So I've changed
ENV MY_VAR 1
FROM ...
to
ARG MY_VAR=1
FROM ...
BTW You can read about ARG vs ENV difference here https://vsupalov.com/docker-arg-vs-env/
I do not think that this is your case, but it might help someone else with that error.
My Dockerfile ran without that error.
I added "FROM vcatechnology/linux-mint:18.2" to the start of the Dockerfile, no error either.
After deleting that FROM statement again, it was still searching for it, causing this error.
I could only get rid of the error by adding the FROM statement again.
Thus, this error can simply appear if you have used a Dockerfile starting with a FROM statement and if you then drop that FROM statement again.
In my case, I changed RUN to FROM.
Old Dockerfile:
RUN php:8-apache
COPY /src var/www/html/
ENV APACHE_DOCUMENT_ROOT ./src/public/
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
EXPOSE 80
New Dockerfile:
FROM php:8-apache
COPY /src var/www/html/
ENV APACHE_DOCUMENT_ROOT ./src/public/
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
EXPOSE 80
I use docker for development and in production for laravel project. I have slightly different dockerfile for development and production. For example I am mounting local directory to docker container in development environment so that I don't need to do docker build for every change in code.
As mounted directory will only be available when running the docker container I can't put commands like "composer install" or "npm install" in dockerfile for development.
Currently I am managing two docker files, is there any way that I can do this with single docker file and decide which commands to run when doing docker build by sending parameters.
What I am trying to achieve is
In docker file
...
IF PROD THEN RUN composer install
...
During docker build
docker build [PROD] -t mytag .
As a best practice you should try to aim to use one Dockerfile to avoid unexpected errors between different environments. However, you may have a usecase where you cannot do that.
The Dockerfile syntax is not rich enough to support such a scenario, however you can use shell scripts to achieve that.
Create a shell script, called install.sh that does something like:
if [ ${ENV} = "DEV" ]; then
composer install
else
npm install
fi
In your Dockerfile add this script and then execute it when building
...
COPY install.sh install.sh
RUN chmod u+x install.sh && ./install.sh
...
When building pass a build arg to specify the environment, example:
docker build --build-arg "ENV=PROD" ...
UPDATE (2020):
Since this was written 3 years ago, many things have changed (including my opinion about this topic). My suggested way of doing this, is using one dockerfile and using scripts. Please see #yamenk's answer.
ORIGINAL:
You can use two different Dockerfiles.
# ./Dockerfile (non production)
FROM foo/bar
MAINTAINER ...
# ....
And a second one:
# ./Dockerfile.production
FROM foo/bar
MAINTAINER ...
RUN composer install
While calling the build command, you can tell which file it should use:
$> docker build -t mytag .
$> docker build -t mytag-production -f Dockerfile.production .
You can use build args directly without providing additional sh script. Might look a little messy, though. But it works.
Dockerfile must be like this:
FROM alpine
ARG mode
RUN if [ "x$mode" = "xdev" ] ; then echo "Development" ; else echo "Production" ; fi
And commands to check are:
docker build -t app --build-arg mode=dev .
docker build -t app --build-arg mode=prod .
I have tried several approaches to this, including using docker-compose, a multi-stage build, passing an argument through a file and the approaches used in other answers. My company needed a good way to do this and after trying these, here is my opinion.
The best method is to pass the arg through the cmd. You can pass it through vscode while right clicking and choosing build image
Image of visual studio code while clicking image build
using this code:
ARG BuildMode
RUN echo $BuildMode
RUN if [ "$BuildMode" = "debug" ] ; then apt-get update \
&& apt-get install -y --no-install-recommends \
unzip \
&& rm -rf /var/lib/apt/lists/* \
&& curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l /vsdbg ; fi
and in the build section of dockerfile:
ARG BuildMode
ENV Environment=${BuildMode:-debug}
RUN dotnet build "debugging.csproj" -c $Environment -o /app
FROM build AS publish
RUN dotnet publish "debugging.csproj" -c $Environment -o /app
The best way to do it is with .env file in your project.
You can define two variables CONTEXTDIRECTORY and DOCKERFILENAME
And create Dockerfile-dev and Dockerfile-prod
This is example of using it:
docker compose file:
services:
serviceA:
build:
context: ${CONTEXTDIRECTORY:-./prod_context}
dockerfile: ${DOCKERFILENAME:-./nginx/Dockerfile-prod}
.env file in the root of project:
CONTEXTDIRECTORY=./
DOCKERFILENAME=Dockerfile-dev
Be careful with the context. Its path starts from the directory with the dockerfile that you specified, not from docker-compose directory.
In default values i using prod, because if you forget to specify env variables, you won't be able to accidentally build a dev version in production
Solution with diffrent dockerfiles is more convinient, then scripts. It's easier to change and maintain
I have a Dockerfile as shown below:
FROM centos:centos6
MAINTAINER tapash
###### Helpful utils
RUN yum -y install sudo
RUN yum -y install curl
RUN yum -y install unzip
#########Copy hibernate.cfg.xml to Client
ADD ${hibernate_path}/hibernate.cfg.xml /usr/share/tomcat7/webapps/roc_client/WEB-INF/classes/
I need a command line argument to be passed during docker build to be specified for the $hibernate_path.
How do I do this?
If this is purely a build-time variable, you can use the --build-arg option of docker build.
This flag allows you to pass the build-time variables that are accessed like regular environment variables in the RUN instruction of the Dockerfile. Also, these values don’t persist in the intermediate or final images like ENV values do.
docker build --build-arg hibernate_path=/a/path/to/hibernate -t tag .
In 1.7, only the static ENV Dockerfile directive is available.
So one solution is to generate the Dockerfile you need from a template Dockerfile.tpl.
Dockerfile.tpl:
...
ENV hibernate_path=xxx
ADD xxx/hibernate.cfg.xml /usr/share/tomcat7/webapps/roc_client/WEB-INF/classes/
...
Whenever you want to build the image, you generate first the Dockerfile:
sed "s,xxx,${hibernate_path},g" Dockerfile.tpl > Dockerfile
Then you build normally: docker build -t myimage .
You then benefit from (in docker 1.7):
build-time environment substitution
run-time environment variable.
You create a script that puts in your Dockerfile the required value and launches docker build -t mytag .