I have an npm package where we are keeping our common code and publishing it to an internal repository. The package name is docker-images. Inside that I have a dockerfile with the following
FROM <Our internal base image>
# Setting src variable.
ARG src
# Set working directory
WORKDIR /home/default
USER root
# Copy the src code
COPY $src /home/default
# Install all the dependencies
RUN npm install
# Change permissions to default user and ensure we enter at the right spot
RUN chown -R default:default /home/default
USER default
Also in this package I have a shell script with that does the building
OPTIND=1 # Reset getopts in case it was changed in a previous run
while getopts "h::f::s::" opt; do
case "$opt" in
h)
exit 1
;;
f)
dockerfile=$OPTARG
;;
s)
src=$OPTARG
;;
*)
exit 1
;;
esac
done
docker build --pull=true --build-arg "src=${src}" --tag="latest" --file=${dockerfile} ${src}
From another npm package I have a script which calls this script to build it that script does
npm install docker-images
PKG_ROOT=$(cd "$(dirname "$BASH_SOURCE")" && cd ../ && pwd)
./node_modules/docker-images/scripts/publish.sh -f "$PKG_ROOT/node_modules/docker-images/dockerfiles/dockerfile" -s "$PKG_ROOT"
However when builds on our jenkins box it gives me the error
Step 3 : ARG src
---> Using cache
---> 09e6987081e7
Step 4 : WORKDIR /home/default
---> Using cache
---> d4f1edf337ca
Step 5 : USER root
---> Using cache
---> f5e52439f60f
Step 6 : COPY $src /home/default
lstat home/jenkins-slave/workspace/dockerbuild: no such file or directory
I also printed out the command that my shell script is calling which is
docker build --pull=true --build-arg src=/home/jenkins-slave/workspace/dockerbuild --file=/home/jenkins-slave/workspace/dockerbuild/node_modules/docker-images/dockerfiles/dockerfile /home/jenkins-slave/workspace/dockerbuild
Obviously the path /home/jenkins-slave/workspace/dockerbuild exists since it can find the dockerfile but I don't know why it won't copy the src
The paths in docker are all relative , so just as an experiment can you try the following where you are copying the source:
WORKDIR $src
COPY . /home/default
Rambler is right, the paths need to be relative to the Docker build context, but you don't need to change your Dockerfile, just use a relative path in the build-arg value.
With this simple Dockerfile:
FROM alpine
ARG src
COPY $src .
You will get the failure if you use a full path in the argument:
> docker build -t temp --build-arg src=/home/scrapbook/tutorial/src .
...
Step 3 : COPY $src .
lstat home/scrapbook/tutorial/src: no such file or directory
But if you use a relative path from the build context, the same Dockerfile is fine:
> docker build -t temp --build-arg src=./src .
...
Successfully built d4899d51a284
Related
The Dockerfile uses the COPY --from command from the other build Node layer, but the generated directory is not found.
Note 1: This Dockerfile works locally on my machine doing builds normally.
Note 2: In the execution log it mentions the removal of an intermediate container, is that it? Would it be possible to preserve this container so that the copy works?
FROM node:16.16 as build
# USER node
WORKDIR /app
COPY package.json /app
RUN npm install --location=global npm#latest && npm install --silent
COPY . .
ARG SCRIPT
ENV SCRIPT=$SCRIPT
ARG API_URL
ENV API_URL=$API_URL
ARG API_SERVER
ENV API_SERVER=$API_SERVER
CMD ["/bin/sh", "-c", "envsubst < src/proxy.conf.template.js > src/proxy.conf.js"]
RUN npm run ${SCRIPT}
FROM nginx:1.23
VOLUME /var/cache/nginx
EXPOSE 80
COPY --from=build /app/dist/siai-spa /usr/share/nginx/html
COPY ./config/nginx-template.conf /etc/nginx/nginx-template.conf
b9ed43dcc388: Pull complete
Digest: sha256:db345982a2f2a4257c6f699a499feb1d79451a1305e8022f16456ddc3ad6b94c
Status: Downloaded newer image for nginx:1.23
---> 41b0e86104ba
Step 15/24 : VOLUME /var/cache/nginx
---> Running in dc0e24ae6e51
Removing intermediate container dc0e24ae6e51
---> 3b2799dad197
Step 16/24 : EXPOSE 80
---> Running in f30edd617285
Removing intermediate container f30edd617285
---> 21985745ce49
Step 17/24 : COPY --from=build /app/dist/siai-spa /usr/share/nginx/html
COPY failed: stat app/dist/siai-spa: file does not exist
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1
I guess, you should use CMD instead of RUN while npm run ${SCRIPT} as this needs to be executed during container running time rather than image build time.
Solved problem!
The difference was that locally I used docker-compose which captures the build arguments from the .env file. The npm run command did not override the ${SCRIPT} variable as the docker command does not use the env file, required to pass through the --build-arg parameters.
I am trying to build my Dockerfile from a child folder context.
this is my build file in build.sh
#!/bin/bash -ex
docker build -t app:latest -f ../Dockerfile .
This is my Dockerfile
FROM app:latest
WORKDIR /app
# This path must exist as it is used as a mount point for testing
# Ensure your app is loading files from this location
FROM ubuntu:20.04
RUN apt update
RUN apt install -y python3-pip
# Install Dependencies
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt
This is my requirements.txt
Flask
pandas
requests
gunicorn
pytest
When I attempt to run build.sh within the scripts folder I get this error
#8 ERROR: "/requirements.txt" not found: not found
------
> [stage-1 4/7] COPY requirements.txt .:
------
failed to compute cache key: "/requirements.txt" not found: not found
When I just do the docker build command in the command line in the app directory I will do this:
docker build -t app:latest -f Dockerfile .
This will work, however going into the child directory and attempting to build it using the bash script will fail with the requirements.txt caching issue.
How do I successfully build my docker container from the child folder?
The docker build command takes a path to a context directory
docker build [... options ...] .
# ^ this path
When the Dockerfile COPY requirements.txt . it is always relative to the path at the end of the docker build command. It doesn't matter where the Dockerfile itself is physically located.
If you want to build an image from a parent directory, where the Dockerfile is located in that parent directory, you need to specify the path. If the Dockerfile is named Dockerfile and is in the root of the context directory (the standard recommended location) then you do not need a docker build -f option.
cd $HOME/testing-docker
docker build -t app .
cd $HOME/testing-docker/scripts
docker build -t app ..
# ^^ build the parent directory
# but no -f option
# the Dockerfile is in the default place
# Any way to specify the path will work
cd
docker build -t app $HOME/testing-docker
I am having difficulties with ARG & ENV in docker after I have upgraded to Docker version 20.10.7, build f0df350 on windows 10.
I have made dockerfile to show issue:
FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
ARG node_build=production
ENV node_build_env=${node_build}
FROM node:12.18.3 AS node-build
WORKDIR /root
RUN echo $node_build_env > test.txt
FROM base AS final
WORKDIR /app
COPY --from=node-build /root/test.txt ./
My goal here is that an ARG can be set and it will be then set as environment variable inside the container and if none is set it has a default value.
In this Dockerfile I am attempting to write the environment variable node_build_env to a text file then copy it to the final layer. The problem though is that the file is empty.
To re-create these are commands I am using:
docker build -t testargs:test .
docker run -it --rm testargs:test /bin/bash
cat test.txt
The file is empty. However if I run:
docker build -t testargs:test . --target node-build
and then manually run the command:
echo $node_build_env > test.txt
It works and the value production is written into the file.
Why does it work when I do it manually but not as part of the RUN command?
You are using multi-stage builds.
Your ARG & ENV belongs to base stage. And you're not using your base stage in your node-build build stage.
That means there is no node_build_env value in node-build. Hence the following line creates an empty test.txt file.
RUN echo $node_build_env > test.txt
However your final stage uses base stage. Which means it has access to node_build_env variable. So after building your image using docker build -t testargs:test . and then open up an interactive session with that container and try to execute the following command,
echo $node_build_env
You will see production will be printed out in the terminal.
I believe this will help you solve the problem. Cheers 🍻 !!!
edit:
this is working version:
ARG node_build=production
FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
FROM node:12.18.3 AS node-build
ARG node_build
ENV node_build_env=$node_build
WORKDIR /root
RUN echo $node_build_env > test.txt
FROM base AS final
WORKDIR /app
COPY --from=node-build /root/test.txt ./
I want to copy the file to a folder location that is outside the working directory. I used the following lines in my docker file, but the files are not there when I look in the container.
WORKDIR /app
RUN cd ../opt/venv/lib/python3.7/site-packages/xxx/
COPY ./resources/abc.py .
When look a that /opt/venv/lib/python3.7/site-packages/xxx/ location the abc.py is not there
What is the issue with my approach? Appreciate your inputs.
You can't COPY a file from outside the build context. So if you are trying to COPY /opt/venv/lib/python3.7/site-packages/xxx/resources/abc.py into your docker image, and that is not in your build context, it will fail. Full stop.
Here's some annotated code.
# change to the /app directory in the container
WORKDIR /app
# run the command cd in the container. cd is a shell builtin, and after
# this command finishes you will still be inside the /app directory in
# your container.
RUN cd ../opt/venv/lib/python3.7/site-packages/xxx/
# Attempt to copy ./resources/abc.py from your host's build context
# (NOT /opt/venv/lib/python3.7/site-packages/xxx/) into the container.
COPY ./resources/abc.py .
The basic fix for this is to first copy abc.py into your build directory. Then you will be able to copy it into your docker container during your build like so:
WORKDIR /app
COPY abc.py .
# /app/abc.py now exists in your container
Note on cd
cd is a shell builtin that changes the working directory of the shell. When you execute it inside a script (or in this case a docker RUN) it only changes the working directory for that process, which ends when the script ends. After which your working directory will be the one you started in. So you cannot use it in the way you were intending. Much better explanation here.
Take this Dockerfile for example:
FROM alpine:latest
RUN cd /opt # cd to /opt
RUN pwd # check current directory, you're STILL in '/'
RUN cd /opt && \
pwd # works as expected because you're still in the same process that cd ran in.
# But once you exit this RUN command you will be back in '/'
# Use WORKDIR to set the working directory in a dockerfile
Here's the output of building that Dockerfile (removed noisy docker output):
$ docker build --no-cache .
Sending build context to Docker daemon 3.584kB
Step 1/4 : FROM alpine:latest
Step 2/4 : RUN cd /opt
Step 3/4 : RUN pwd
/
Step 4/4 : RUN cd /opt && pwd
/opt
From what I understand, you're trying to copy a file into a specific location (/opt/venv/lib/python3.7/site-packages/xxx/) in your Docker image that is outside the WORKDIR you defined in the Dockerfile for your image.
You can easily do this by specifying the absolute destination path in the COPY command:
WORKDIR /app
COPY ./resources/abc.py /opt/venv/lib/python3.7/site-packages/xxx/abc.py
Hi I am new to Docker and trying to wrap around my head on how to clone a private repo from github and found some interesting link issues/6396
I followed one of the post and my dockerfile looks like
FROM python:2.7 as builder
# Deploy app's code
#RUN set -x
RUN mkdir /code
RUN mkdir /root/.ssh/
RUN ls -l /root/.ssh/
# The GITHUB_SSH_KEY Build Argument must be a path or URL
# If it's a path, it MUST be in the docker build dir, and NOT in .dockerignore!
ARG SSH_PRIVATE_KEY=C:\\Users\\MyUser\\.ssh\\id_rsa
RUN echo "${SSH_PRIVATE_KEY}"
# Set up root user SSH access for GitHub
ADD ${SSH_PRIVATE_KEY} /root/.ssh/id_rsa
RUN ssh -o StrictHostKeyChecking=no -vT git#github.com 2>&1 | grep -i auth
# Test SSH access (this returns false even when successful, but prints results)
RUN git clone git#github.com:***********.git
COPY . /code
WORKDIR /code
ENV PYTHONPATH /datawarehouse_process
# Setup app's virtualenv
RUN set -x \
&& pip install tox \
&& tox -e luigi
WORKDIR /datawarehouse_process
# Finally, remove the $GITHUB_SSH_KEY if it was a file, so it's not in /app!
# It can also be removed from /root/.ssh/id_rsa, but you're probably not
going
# to COPY that directory into the runtime image.
RUN rm -vf ${GITHUB_SSH_KEY} /root/.ssh/id*
#FROM python:2.7 as runtime
#COPY --from=builder /code /code
When I run docker build . from the correct location I get this error below. Any clue will be appreciated.
c:\Domain\Project\Docker-Images\datawarehouse_process>docker build .
Sending build context to Docker daemon 281.7MB
Step 1/15 : FROM python:2.7 as builder
---> 43c5f3ee0928
Step 2/15 : RUN mkdir /code
---> Running in 841fadc29641
Removing intermediate container 841fadc29641
---> 69fdbcd34f12
Step 3/15 : RUN mkdir /root/.ssh/
---> Running in 50199b0eb002
Removing intermediate container 50199b0eb002
---> 6dac8b120438
Step 4/15 : RUN ls -l /root/.ssh/
---> Running in e15040402b79
total 0
Removing intermediate container e15040402b79
---> 65519edac99a
Step 5/15 : ARG SSH_PRIVATE_KEY=C:\\Users\\MyUser\\.ssh\\id_rsa
---> Running in 10e0c92eed4f
Removing intermediate container 10e0c92eed4f
---> 707279c92614
Step 6/15 : RUN echo "${SSH_PRIVATE_KEY}"
---> Running in a9f75c224994
C:\Users\MyUser\.ssh\id_rsa
Removing intermediate container a9f75c224994
---> 96e0605d38a9
Step 7/15 : ADD ${SSH_PRIVATE_KEY} /root/.ssh/id_rsa
ADD failed: stat /var/lib/docker/tmp/docker-
builder142890167/C:\Users\MyUser\.ssh\id_rsa: no such file or
directory
From the Documentation:
ADD obeys the following rules:
The path must be inside the context of the build; you cannot ADD
../something /something, because the first step of a docker build is
to send the context directory (and subdirectories) to the docker
daemon.
You are passing an absolute path to ADD, but you can see from the error:
/var/lib/docker/tmp/docker-builder142890167/C:\Users\MyUser\.ssh\id_rsa:
no such file or directory
It is being looked for within the build context. Again from the documentation:
Traditionally, the Dockerfile is called Dockerfile and located in the
root of the context.
So, you need to place the RSA key somewhere in the directory tree which has it's root at the path that you specify in your Docker build command, so if you are entering docker build . your ARG statement would change to something like:
ARG SSH_PRIVATE_KEY = .\.ssh\id_rsa