I'm trying to run a docker build which performs a maven build.
This is my Dockerfile:
FROM maven:3.5.4-jdk-8-slim
COPY * /myapp/src/
WORKDIR /myapp/src/
RUN mvn clean package
RUN ls -la target
RUN ls -la ./target
COPY ./target/myapp.jar ../
COPY config.yml ../
WORKDIR /myapp/
CMD ["java", "-jar", "myapp.jar"]
The issue I'm receiving is
COPY failed: stat /var/snap/docker/common/var-lib-docker/tmp/docker-builder185918822/target/myapp.jar: no such file or directory
I added the two ls statement for debugging purposes, this is the output:
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:27 min
[INFO] Finished at: 2018-10-30T13:59:42Z
[INFO] ------------------------------------------------------------------------
Removing intermediate container 5c0f369c5f9d
---> d2b36b5070e3
Step 5/10 : RUN ls -la target
---> Running in 4de6062b1573
total 19104
drwxr-xr-x 4 root root 4096 Oct 30 13:59 .
drwxr-xr-x 5 root root 4096 Oct 30 13:59 ..
drwxr-xr-x 2 root root 4096 Oct 30 13:59 classes
drwxr-xr-x 2 root root 4096 Oct 30 13:59 maven-archiver
-rw-r--r-- 1 root root 19538983 Oct 30 13:59 myapp.jar
-rw-r--r-- 1 root root 3294 Oct 30 13:59 original-myapp.jar
Removing intermediate container 4de6062b1573
---> 47f4a7534ffb
Step 6/10 : RUN ls -la ./target
---> Running in 55c68b1fbbe9
total 19104
drwxr-xr-x 4 root root 4096 Oct 30 13:59 .
drwxr-xr-x 5 root root 4096 Oct 30 13:59 ..
drwxr-xr-x 2 root root 4096 Oct 30 13:59 classes
drwxr-xr-x 2 root root 4096 Oct 30 13:59 maven-archiver
-rw-r--r-- 1 root root 19538983 Oct 30 13:59 myapp.jar
-rw-r--r-- 1 root root 3294 Oct 30 13:59 original-myapp.jar
Removing intermediate container 55c68b1fbbe9
---> bbc8a395474c
Step 7/10 : COPY ./target/myapp.jar ../
COPY failed: stat /var/snap/docker/common/var-lib-docker/tmp/docker-builder185918822/target/myapp.jar: no such file or directory
As you can see from the output of both ls commands, the jar file is there. Why is docker failing to find it?
Related
I have a cloudbuild.yaml file that looks like this:
steps:
- name: 'gcr.io/cloud-builders/gsutil'
args: [ "-m", "rsync", "-r", "gs://${_BUCKET}/maven-repository", "/cache/.m2" ]
volumes:
- path: '/cache/.m2'
name: 'm2_cache'
- name: docker/compose:debian-1.29.2
entrypoint: bash
args:
- -c
- |
./test.sh
volumes:
- path: '/cache/.m2'
name: 'm2_cache'
timeout: 2700s
substitutions:
_BUCKET: 'my-bucket'
In the first step we download our maven settings.xml file from GCS. This file is crucial for subsequent build steps since it contain the username/password to our Artifact Registry Maven repository (I've simplified this example as we don't actually store the credential in the settings.xml as plain text). Without these credentials, our Maven build won't run. Normally the script that we call in the second step starts several docker containers and then run our maven tests. But I've replaced it with test.sh to easier show what the problem is. The test.sh file is shown below:
#!/bin/bash
echo "### [Host] Contents in /cache/.m2"
ls -la /cache/.m2
mkdir ~/test
echo "Johan" > ~/test/ikk.txt
echo "### [Host] Contents in ~/test"
ls -la ~/test
docker run --rm -v /cache/.m2:/cache/.m2 -v ~/test:/root/test -w /usr/src/somewhere ubuntu bash -c 'echo "### [Docker] Contents in /cache/.m2" && ls -la /cache/.m2 && echo "### [Docker] Contents in /root/test" && ls -la /root/test'
I.e. we try to mount two volumes to the ubuntu container that we start in the test.sh file. I list the contents in two directors both outside (### [Host]) and inside (### [Docker]) the ubuntu container. Here's the relevant output of running this in cloud build:
### [Host] Contents in /cache/.m2
total 16
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwxr-xr-x 3 root root 4096 Sep 15 08:55 ..
-rw-r--r-- 1 root root 8063 Sep 13 11:03 settings.xml
### [Host] Contents in ~/test
total 12
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwxr-xr-x 6 root root 4096 Sep 15 08:55 ..
-rw-r--r-- 1 root root 6 Sep 15 08:55 ikk.txt
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
Digest: sha256:20fa2d7bb4de7723f542be5923b06c4d704370f0390e4ae9e1c833c8785644c1
Status: Downloaded newer image for ubuntu:latest
### [Docker] Contents in /cache/.m2
total 8
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwxr-xr-x 3 root root 4096 Sep 15 08:55 ..
### [Docker] Contents in /root/test
total 8
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwx------ 1 root root 4096 Sep 15 08:55 ..
As you can see, the volume mounts doesn't seem to work when I run the ubuntu container from the test.sh file in cloud build (since the contents of /root/test and /cache/.m2 are empty).
Running the test.sh locally on my machine yields the expected outcome:
### [Host] Contents in /cache/.m2
total 40
drwxr-xr-x 7 johan staff 224 Mar 15 2022 .
drwxr-x---+ 87 johan staff 2784 Sep 15 10:58 ..
-rw-r--r-- 1 johan staff 2344 Sep 14 11:37 copy_reference_file.log
drwxr-xr-x 221 johan staff 7072 Sep 14 10:52 repository
-rw-r--r-- 1 johan staff 327 Nov 24 2021 settings-docker.xml
-rw-r--r--# 1 johan staff 9842 Mar 15 2022 settings.xml
drwxr-xr-x 3 johan staff 96 Nov 19 2021 wrapper
### [Host] Contents in ~/test
total 8
drwxr-xr-x# 3 johan staff 96 Sep 15 10:53 .
drwxr-xr-x# 135 johan staff 4320 Sep 15 10:49 ..
-rw-r--r-- 1 johan staff 6 Sep 15 10:58 ikk.txt
### [Docker] Contents in /cache/.m2
total 24
drwxr-xr-x 7 root root 224 Mar 15 2022 .
drwxr-xr-x 3 root root 4096 Sep 15 08:58 ..
-rw-r--r-- 1 root root 2344 Sep 14 09:37 copy_reference_file.log
drwxr-xr-x 221 root root 7072 Sep 14 08:52 repository
-rw-r--r-- 1 root root 327 Nov 24 2021 settings-docker.xml
-rw-r--r-- 1 root root 9842 Mar 15 2022 settings.xml
drwxr-xr-x 3 root root 96 Nov 19 2021 wrapper
### [Docker] Contents in /root/test
total 8
drwxr-xr-x 3 root root 96 Sep 15 08:53 .
drwx------ 1 root root 4096 Sep 15 08:58 ..
-rw-r--r-- 1 root root 6 Sep 15 08:58 ikk.txt
Here you can see that the volumes are mounted correctly and I can access the files inside the ubuntu container.
How can I mount volumes inside a container in cloud build?
In a Dockerfile, I have existing folders from my derived image /home and /root. I cannot create a folder in the /home directory:
RUN mkdir -p /home/TEST_3
No error messages are display but I can see that the folder was not created.
The same command is working fine in the /root directory
Bother folder seems to be similar in term of permission:
drwxr-xr-x 2 root root 4096 Apr 24 2018 home
drwx------ 1 root root 4096 Jun 20 08:38 root
The hard link count is different. The problematic folder /homehas more permission
Another strange thing is that I can create a folder in /home using:
WORKDIR /home/TEST_2
but then any file that I create in this folder doesn't seems to exist.
Here a small Dockerfile to recreate this issue/feature
# Specify the base image and tag
FROM gcr.io/deeplearning-platform-release/pytorch-cpu:m27
RUN ls -la
RUN ls -la /home/
RUN ls -la /root/
## creation ofthe folder is working and the folder exist (and I can create a file in the new folder)
RUN mkdir -p /root/TEST_1
RUN ls -la /root/TEST_1
RUN touch /root/TEST_1/new_file.txt
RUN ls -la /root/TEST_1
## creation of the folder is working and the folder exist but the file created doesn't exist
WORKDIR /home/TEST_2
WORKDIR /
RUN ls -la /home/TEST_2
RUN touch /home/TEST_2/new_file.txt
RUN ls -la /home/TEST_2
## creation of the folder succeed (no error messages), but the folder doesn't exist
RUN mkdir -p /home/TEST_3
RUN ls -la /home/TEST_3
RUN touch /home/TEST_3/new_file.txt
RUN ls -la /home/TEST_3
and here the output:
docker build -f Dockerfile_test .
Sending build context to Docker daemon 37.89kB
Step 1/17 : FROM gcr.io/deeplearning-platform-release/pytorch-cpu:m27
---> 7eb6565a8cc6
Step 2/17 : RUN ls -la
---> Running in 1ced0ed2bf49
total 80
drwxr-xr-x 1 root root 4096 Oct 26 12:52 .
drwxr-xr-x 1 root root 4096 Oct 26 12:52 ..
-rwxr-xr-x 1 root root 0 Oct 26 12:52 .dockerenv
drwxr-xr-x 1 root root 4096 Jun 20 08:20 bin
drwxr-xr-x 2 root root 4096 Apr 24 2018 boot
drwxr-xr-x 5 root root 340 Oct 26 12:52 dev
-rwxr-x--x 1 root root 94 Jun 19 21:51 entrypoint.sh
-rwxr-xr-x 1 root root 0 Jun 20 08:38 env.sh
drwxr-xr-x 1 root root 4096 Oct 26 12:52 etc
drwxr-xr-x 2 root root 4096 Apr 24 2018 home
drwxr-xr-x 1 root root 4096 Jun 20 08:22 lib
drwxr-xr-x 2 root root 4096 Jun 12 16:55 lib64
drwxr-xr-x 2 root root 4096 Jun 12 16:54 media
drwxr-xr-x 2 root root 4096 Jun 12 16:54 mnt
drwxr-xr-x 1 root root 4096 Jun 20 08:23 opt
dr-xr-xr-x 226 root root 0 Oct 26 12:52 proc
drwx------ 1 root root 4096 Jun 20 08:38 root
drwxr-xr-x 1 root root 4096 Jun 20 08:33 run
-rwxr-x--x 1 root root 109 Jun 19 21:51 run_jupyter.sh
drwxr-xr-x 1 root root 4096 Jun 20 08:20 sbin
drwxr-xr-x 2 root root 4096 Jun 12 16:54 srv
dr-xr-xr-x 13 root root 0 Oct 26 12:52 sys
drwxrwxrwt 1 root root 4096 Jun 20 10:09 tmp
drwxr-xr-x 1 root root 4096 Jun 20 08:33 usr
drwxr-xr-x 1 root root 4096 Jun 12 16:55 var
Removing intermediate container 1ced0ed2bf49
---> 87dbdffa7315
Step 3/17 : RUN ls -la /home/
---> Running in 756f013caf7f
total 8
drwxr-xr-x 2 root root 4096 Apr 24 2018 .
drwxr-xr-x 1 root root 4096 Oct 26 12:52 ..
Removing intermediate container 756f013caf7f
---> 24419a25ac99
Step 4/17 : RUN ls -la /root/
---> Running in bde249ae0452
total 52
drwx------ 1 root root 4096 Jun 20 08:38 .
drwxr-xr-x 1 root root 4096 Oct 26 12:52 ..
-rw-r--r-- 1 root root 3106 Apr 9 2018 .bashrc
drwxr-xr-x 3 root root 4096 Jun 20 08:31 .cache
drwxrwsr-x 1 root root 4096 Jun 20 08:26 .conda
drwx------ 4 root root 4096 Jun 20 08:33 .config
-rw-r--r-- 1 root root 401 Jun 20 08:33 .gitconfig
drwxr-xr-x 3 root root 4096 Jun 20 08:38 .ipython
drwxr-xr-x 1 root root 4096 Jun 20 10:10 .jupyter
drwxr-xr-x 3 root root 4096 Jun 20 08:31 .npm
-rw-r--r-- 1 root root 148 Aug 17 2015 .profile
-rw-r--r-- 1 root root 405 Jun 20 10:10 .wget-hsts
drwxrwxrwx 1 root root 4096 Jun 20 10:09 miniconda3
Removing intermediate container bde249ae0452
---> f6e6b3780145
Step 5/17 : RUN mkdir -p /root/TEST_1
---> Running in 95ffe0bc574f
Removing intermediate container 95ffe0bc574f
---> a87cac09ec17
Step 6/17 : RUN ls -la /root/TEST_1
---> Running in b8fd10afa3ee
total 8
drwxr-xr-x 2 root root 4096 Oct 26 12:52 .
drwx------ 1 root root 4096 Oct 26 12:52 ..
Removing intermediate container b8fd10afa3ee
---> 87b173385d4b
Step 7/17 : RUN touch /root/TEST_1/new_file.txt
---> Running in cbdd9ac79955
Removing intermediate container cbdd9ac79955
---> 0f57f2c15855
Step 8/17 : RUN ls -la /root/TEST_1
---> Running in 0a7d46a44e9e
total 8
drwxr-xr-x 1 root root 4096 Oct 26 12:53 .
drwx------ 1 root root 4096 Oct 26 12:52 ..
-rw-r--r-- 1 root root 0 Oct 26 12:53 new_file.txt
Removing intermediate container 0a7d46a44e9e
---> 5413892c3b96
Step 9/17 : WORKDIR /home/TEST_2
---> Running in d0322094ae3b
Removing intermediate container d0322094ae3b
---> 9815f6870830
Step 10/17 : WORKDIR /
---> Running in f84feebf3db9
Removing intermediate container f84feebf3db9
---> 5c85f2fadd04
Step 11/17 : RUN ls -la /home/TEST_2
---> Running in 4644cbcd5c3c
total 8
drwxr-xr-x 2 root root 4096 Oct 26 12:53 .
drwxr-xr-x 3 root root 4096 Oct 26 12:53 ..
Removing intermediate container 4644cbcd5c3c
---> c82625e638d9
Step 12/17 : RUN touch /home/TEST_2/new_file.txt
---> Running in ac89ce336a51
Removing intermediate container ac89ce336a51
---> d9569d5ca42d
Step 13/17 : RUN ls -la /home/TEST_2
---> Running in 999f75b4ece1
total 8
drwxr-xr-x 2 root root 4096 Oct 26 12:53 .
drwxr-xr-x 3 root root 4096 Oct 26 12:53 ..
Removing intermediate container 999f75b4ece1
---> 57062c0ccf0a
Step 14/17 : RUN mkdir -p /home/TEST_3
---> Running in f9788ac58dc7
Removing intermediate container f9788ac58dc7
---> e88a503ae524
Step 15/17 : RUN ls -la /home/TEST_3
---> Running in 16ea32c637c6
ls: cannot access '/home/TEST_3': No such file or directory
The command '/bin/sh -c ls -la /home/TEST_3' returned a non-zero code: 2
I am using Docker version 19.03.4, build 9013bf5 on MacOS 10.14.6
My questions are the following:
where is comming the difference (be able to create a folder in/root but not in /home
how to fix this issue to be able to create a folder in /home and
be able to create files as well in the new folder.
TL;DR
what is the difference between /home and /root
/home has a volume associated with it
how to get around the issue
create the file on entry-point (after the container is built), OR create the file inside the volume (I use the first option in my long answer)
Long answer:
Running a docker inspect containerId after building the Image & container show theirs a volumes assocated with the home directory:
"Source": "/var/lib/docker/volumes/1447dbc568742221a757a650427f38fb485e42fc7c6e959f33966922fe862d13/_data",
"Destination": "/home",
The volume is RW (read write), so you can write to the file once inside the container. (I tested using docker exec -it containerId /bin/sh then touch /home/TEST_2/new_file.txt)
Try using an Entrypoint script to create the folders/files.
Edit:
I wanted to see for myself, but ran into a couple hurdles. The parent container (gcr.io/deeplearning-platform-release/pytorch-cpu:m27) already has an entrypoint.sh file.
Also, adding a CMD or ENTRYPOINT command will override the parents entrypoint container (which you probably want).
Working Solution:
From the host, COPY the entrypoint.sh file (Assuming you want to modify it on your host). docker cp containerId:/entrypoint.sh .
Modify the entrypoint, adding your TOUCH commands:
#!/bin/bash
touch /home/TEST_2/new_file.txt # <-- I added this
. /env.sh
. /root/miniconda3/etc/profile.d/conda.sh
conda activate base
exec "$#"
Change your dockerfile to copy the new entrypoint.sh, which will replace the existing one:
COPY entrypoint.sh /entrypoint.sh
Your listing commands on startup will be the same, however if you go into the container you will see your file:
# ls home/TEST_2
new_file.txt
#
I have a docker image https://github.com/carnellj/spmia-chapter1 which does not find its CMD ./run.sh executable although it is there in the file system.
I was able to run /bin/sh in the container, and I can ls -l:
D:\Dokumente\ws\spring-microservices\spmia-chapter1 (master)
λ docker run -i -t johncarnell/tmx-simple-service:chapter1 /bin/sh
/ # ls -l
total 56
drwxr-xr-x 2 root root 4096 Mar 3 11:20 bin
drwxr-xr-x 5 root root 360 Apr 22 07:10 dev
drwxr-xr-x 1 root root 4096 Apr 22 07:10 etc
drwxr-xr-x 2 root root 4096 Mar 3 11:20 home
drwxr-xr-x 1 root root 4096 Apr 22 06:01 lib
drwxr-xr-x 5 root root 4096 Mar 3 11:20 media
drwxr-xr-x 2 root root 4096 Mar 3 11:20 mnt
dr-xr-xr-x 123 root root 0 Apr 22 07:10 proc
drwx------ 1 root root 4096 Apr 22 07:10 root
drwxr-xr-x 2 root root 4096 Mar 3 11:20 run
-rwxr-xr-x 1 root root 245 Apr 22 06:50 run.sh
drwxr-xr-x 2 root root 4096 Mar 3 11:20 sbin
drwxr-xr-x 2 root root 4096 Mar 3 11:20 srv
dr-xr-xr-x 13 root root 0 Apr 22 07:10 sys
drwxrwxrwt 2 root root 4096 Mar 3 11:20 tmp
drwxr-xr-x 1 root root 4096 Mar 7 01:04 usr
drwxr-xr-x 1 root root 4096 Mar 7 01:04 var
/ # ./run.sh
/bin/sh: ./run.sh: not found
/ # ls run.sh
run.sh
/bin/sh does not find ./run.sh although it is there in the file system, as proven by ls run.sh. Also, cat shows the content of run.sh:
/ # cat run.sh
#!/bin/sh
echo "********************************************************"
echo "Starting simple-service "
echo "********************************************************"
java -jar /usr/local/simple-service/simple-service-0.0.1-SNAPSHOT.jar
When I run vi from sh and copy the content of run.sh into a new file myrun.sh and make myrun.sh executable, I can execute ./myrun.sh and the spring service starts.
What is going on here? Why would sh not see an executable which is there in the filesystem? Executables from PATH or executables which I add manually run fine.
I am running Docker on Windows 10.
OK the reason is, run.sh is created with Windows line endings in the docker image if you check out with automatic lf->crlf conversion. One possible solution is to tell git not to convert line endings.
I'm using a .dockerignore file that looks like this:
*
*/node_modules
*/bower_components
!www
!app
!inc
so here what I'm saying is: ignore all files in the current directory (this post is related to: How to include local libraries in build?) except for www, app and inc and ignore all directories named node_modules and bower_components in those subdirectories
when I run a build:
build --no-cache -t test -f app/Dockerfile .
(here's my Dockerfile):
FROM node:latest
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ADD . /usr/src/app/
and then run a container:
run -i --entrypoint=/bin/bash test
if I look in the directory I see node_modules folders!
# ls -alF
total 20
drwxr-xr-x 5 root root 4096 Mar 19 03:39 ./
drwxr-xr-x 4 root root 4096 Mar 19 03:39 ../
drwxr-xr-x 4 root root 4096 Mar 18 03:26 inc/
drwxr-xr-x 3 root root 4096 Mar 19 00:57 app/
drwxr-xr-x 12 root root 4096 Mar 18 08:55 www/
# ls -alF app
total 36
drwxr-xr-x 3 root root 4096 Mar 19 00:57 ./
drwxr-xr-x 5 root root 4096 Mar 19 03:39 ../
-rw-r--r-- 1 root root 38 Mar 19 00:57 .dockerignore
-rw-r--r-- 1 root root 359 Mar 19 03:37 Dockerfile
-rw-r--r-- 1 root root 700 Mar 18 04:50 config.json
-rw-r--r-- 1 root root 3786 Mar 18 03:43 index.js
drwxr-xr-x 119 root root 4096 Mar 17 18:57 node_modules/
-rw-r--r-- 1 root root 2910 Mar 17 20:51 npm-debug.log
-rw-r--r-- 1 root root 348 Mar 19 03:24 package.json
so the first line of the .dockerignore worked fine, as did the 4th, 5th and 6th lines, but the subdirectories failed. for the sake of completeness I originally used **/node_modules but that failed too.
what is the correct way of expressing this?
ok, apparently the problem is that when I create exceptions to the ignore rules, I have to reapply the exceptions, so if I do this instead:
*
!www
!app
!inc
*/node_modules
*/bower_components
it works because */node_modules applies to "all of the above", which includes app. in my original declaration I was excluding the directories I do want to include from the rules
I use the Jenkins Docker plugin to build my image (this works), I can run the container just fine with a command like
docker run test-env:latest /etc/bootstrap.sh -jenkins
However, when I run
docker run -u root -v $WORKSPACE:/test test-env:latest /etc/bootstrap.sh -jenkins
I run in to some issues. The volume /test exists in the container, however it contains none of the files that $WORKSPACE has. Before running my docker container I do a
cd $WORKSPACE && ls -al
which shows the following contents:
total 52
drwxr-xr-x 10 root root 4096 Sep 3 15:49 .
drwxr-xr-x 3 root root 4096 Sep 3 15:49 ..
drwxr-xr-x 8 root root 4096 Sep 3 15:49 .git
-rw-r--r-- 1 root root 361 Sep 3 15:49 .gitignore
-rw-r--r-- 1 root root 71 Sep 3 15:49 README.md
drwxr-xr-x 3 root root 4096 Sep 3 15:49 bin
-rw-r--r-- 1 root root 1900 Sep 3 15:49 build.sbt
drwxr-xr-x 3 root root 4096 Sep 3 15:49 conf
drwxr-xr-x 2 root root 4096 Sep 3 15:49 docs
drwxr-xr-x 2 root root 4096 Sep 3 15:49 lib
drwxr-xr-x 2 root root 4096 Sep 3 15:49 project
drwxr-xr-x 3 root root 4096 Sep 3 15:49 scripts
drwxr-xr-x 4 root root 4096 Sep 3 15:49 src
and see the git repository (test_repo) I want to share with the container is there with all the correct files, but inside the container when I do
cd /test && ls -al
all I see is
total 16
drwxr-xr-x 4 root root 4096 Sep 2 14:30 .
drwxr-xr-x 1 root root 4096 Sep 3 10:27 ..
drwxr-xr-x 3 root root 4096 Sep 2 13:26 test_repo
drwxr-xr-x 5 root root 4096 Sep 2 14:30 target
both of which do not contain my code, which I verified with further testing by using $WORKSPACE/test_repo and $WORKSPACE/target as the volumes I mount.
Why are none of my $WORKSPACE files visible in the container? How can I get it to work properly?
Edit: value of $WORKSPACE = /var/tmp/workspace/test-jenkins where test-jenkins is the name of my Jenkins job