How to save data from a docker container on a local host? - docker

I run R-Studio in a container on GitLab. R-Studio build a lot of csv and pdf files. When I run
docker run --rm -it registry.gitlab.com/user/paperboy /bin/bash
I can find in the folder /home/output/csv and /home/output/pdf the files. I will save all this files in a /output/csv and /output/pdf files on a host, in my case on GitLab. The question is how to save data outside the docker Container?
Here is my Dockerfile.
FROM rocker/r-base:latest
RUN apt-get update \
&& apt-get install -yq --no-install-recommends groff \
&& rm -rf /var/lib/apt/lists/*
# Create directories
RUN mkdir -p /home/output/ /home/output/csv/ /home/output/pdf/ /home/script/
WORKDIR /home/script
# Install R-packages
COPY /src/install_packages.R /home/script/install_packages.R
RUN Rscript /home/script/install_packages.R
# Copy data
COPY /src/pairs.csv /home/script/pairs.csv
COPY /src/master.R /home/script/master.R
COPY /src/paperboy.ms /home/script/paperboy.ms
# Run the script
RUN ["Rscript", "master.R"]
$ docker run -d
-v $(pwd)/output/:/home/output
-v $(pwd)/output/csv/:/home/output/csv
-v $(pwd)/output/pdf/:/home/output/pdf
$CONTAINER_IMAGE/$DOCKER_IMAGE
5d11eb7e3d93e8b98b6381f1970c25be426ff67abef5e378b715263f174849c9
This is a part from the .gitlab-ci.yml
run:
stage: run
script:
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS#gitlab.com/$CI_PROJECT_PATH.git
- git config --global user.name ""
- git config --global user.email ""
- git checkout
- docker login registry.gitlab.com --username gitlab+deploy-token-aaaa --password bbbb
- docker pull $CONTAINER_IMAGE/$DOCKER_IMAGE
- docker image ls
- docker run -t -d
-v $(pwd)/output/:/home/output
-v $(pwd)/user/paperboy/output/csv/:/home/output/csv
-v $(pwd)/user/paperboy/output/pdf/:/home/output/pdf
$CONTAINER_IMAGE/$DOCKER_IMAGE
- rm -rf "%CACHE_PATH%/%CI_PIPELINE_ID%"
- pwd
- ls -la
- ls -laR output
- git status
only:
- master
The csv and pdf folder are empty.
$ ls -laR output
output:
total 32
drwxr-xr-x 4 root root 4096 Oct 8 11:37 .
drwxrwxrwx 5 root root 4096 Oct 8 11:37 ..
drwxr-xr-x 2 root root 4096 Oct 8 11:37 csv
drwxr-xr-x 2 root root 4096 Oct 8 11:37 pdf
output/csv:
total 16
drwxr-xr-x 2 root root 4096 Oct 8 11:37 .
drwxr-xr-x 4 root root 4096 Oct 8 11:37 ..
output/pdf:
total 16
drwxr-xr-x 2 root root 4096 Oct 8 11:37 .
drwxr-xr-x 4 root root 4096 Oct 8 11:37 ..

Related

Bug or misuse : docker layer content not available in final image

I am having a hard time trying to build a docker image for some python project, but at some point a RUN command resulting content is not available in the next RUN.
I mean I have a RUN command that generate a directory with some content and that directory is not available for the next RUN command.
Here is the Dockerfile snippet :
RUN poetry config virtualenvs.create true && \
poetry config virtualenvs.in-project true && \
poetry install -n --no-root --no-dev && \
ls -la
RUN ls -la
The first ls -la shows a .venv directory (see below) while it is not present anymore in the second one.
And here is the build output:
Step 9/14 : RUN poetry config virtualenvs.create true && poetry config virtualenvs.in-project true && poetry install -n --no-root --no-dev && ls -la
---> Running in 3a93a4994133
...
drwxrwxr-x 4 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
drwxr-xr-x 4 odoo odoo 4096 Sep 29 20:08 .venv
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 3a93a4994133
---> 694c4e54863c
Step 10/14 : RUN ls -la
---> Running in 5d0ebcf8f1e9
total 32
drwxrwxr-x 3 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 5d0ebcf8f1e9
---> 750255eac6bb
If someone can provide any lead, it would be really appreciated.
Thanks

How to automatically deploy to VPS in Docker from Gitlab CI

I'm trying to deploy to Docker in my VPS everytime a new commit is made in my project in Gitlab. But I'm having an issue doing that.
I tried to install sshpass and then scp folder and files. But it's saying:
sshpass: Failed to run command: No such file or directory.
The folder and files I'm trying to get from build stage, so I dont have to build again my app.
Here's my .gitlab-ci.yml file:
image: node:9.6.1
cache:
paths:
- node_modules/
- build/
- docker-compose.yml
- Dockerfile
- nginx.conf
stages:
- build
- test
- dockerize
build-stage:
stage: build
script:
- npm install
- CI=false npm run build
artifacts:
paths:
- build/
- docker-compose.yml
- nginx.conf
test-stage:
stage: test
script:
- npm install
- CI=false npm run test
dockerize-stage:
stage: dockerize
image: tmaier/docker-compose:latest
services:
- docker:dind
dependencies:
- build-stage
tags:
- docker
script:
- apk update && apk add sshpass
- sshpass -V
- export SSHPASS=$USER_PASS
- ls -la
- sshpass -e ssh -o stricthostkeychecking=no root#ip:/home mkdir new-super-viva
- sshpass -e scp -o stricthostkeychecking=no -r build user#ip:/home/new-folder
- sshpass -e scp -o stricthostkeychecking=no -r docker-compose.yml user#ip:/home/new-folder
- sshpass -e scp -o stricthostkeychecking=no -r nginx.conf user#ip:/home/new-folder
- sshpass -e ssh -o stricthostkeychecking=no user#ip:/home/new-folder docker-compose up --build
Here's the actual output from Gitlab CI:
$ sshpass -V
sshpass 1.06
(C) 2006-2011 Lingnu Open Source Consulting Ltd.
(C) 2015-2016 Shachar Shemesh
This program is free software, and can be distributed under the terms of the GPL
See the COPYING file for more information.
Using "assword" as the default password prompt indicator.
$ export SSHPASS=$USER_PASS
$ ls -la
total 912
drwxrwxrwx 7 root root 4096 Apr 2 13:24 .
drwxrwxrwx 4 root root 4096 Apr 2 13:24 ..
-rw-rw-rw- 1 root root 327 Apr 2 13:24 .env
drwxrwxrwx 6 root root 4096 Apr 2 13:24 .git
-rw-rw-rw- 1 root root 329 Apr 2 13:24 .gitignore
-rw-rw-rw- 1 root root 1251 Apr 2 13:24 .gitlab-ci.yml
-rw-rw-rw- 1 root root 311 Apr 2 11:57 Dockerfile
-rw-rw-rw- 1 root root 2881 Apr 2 13:24 README.md
drwxrwxrwx 5 root root 4096 Apr 2 13:20 build
-rw-rw-rw- 1 root root 340 Apr 2 13:24 build.sh
-rw-rw-rw- 1 root root 282 Apr 2 11:57 docker-compose.yml
-rw-rw-rw- 1 root root 1385 Apr 2 11:57 nginx.conf
drwxr-xr-x 1191 root root 36864 Apr 2 13:22 node_modules
-rw-rw-rw- 1 root root 765929 Apr 2 13:24 package-lock.json
-rw-rw-rw- 1 root root 1738 Apr 2 13:24 package.json
drwxrwxrwx 4 root root 4096 Apr 2 13:24 public
drwxrwxrwx 10 root root 4096 Apr 2 13:24 src
$ sshpass -e ssh -o stringhostkeychecking=no user#ip:/home mkdir new-folder
sshpass: Failed to run command: No such file or directory
ERROR: Job failed: exit code 3
Is there any way that I can copy the build folder, docker-compose.yml and nginx.conf files from build-stage to dockerize-stage and then pass it with sshpassto VPS? It doesnt even working to create folder with mkdir. I also tried to create folder manually and then remove this command from .gitlab-ci.yml but still the same output.
Just so you know I added that USER_PASS in:
https://gitlab.com/user/project/settings/ci_cd in environment variables and let it non-protected

Docker: created files disappear between layers

Running Docker version 17.06.0-ce, build 02c1d87, I have a dockerfile that looks like this:
FROM maven:3.5.2-jdk-8-alpine as builder
RUN chmod -R 777 /root/.m2 &&\
mkdir -p /root/.m2/repository/com/foo/bar &&\
echo "Text" > /root/.m2/repository/com/foo/bar/baz.txt &&\
ls -R -a -l /root/.m2/repository/com/foo
RUN ls -R -a -l /root/.m2/repository/com/foo
The first RUN command successfully creates a file, but the second command can't find it:
Step 1/46 : FROM maven:3.5.2-jdk-8-alpine as builder
---> 293423a981a7
Step 2/46 : RUN chmod -R 777 /root/.m2 && mkdir -p /root/.m2/repository/com/foo/bar && echo "Text" > /root/.m2/repository/com/foo/bar/baz.txt && ls -R -a -l /root/.m2/repository/com/foo
---> Running in a1c0fd142856
/root/.m2/repository/com/foo:
total 12
drwxr-xr-x 3 root root 4096 Nov 30 13:32 .
drwxr-xr-x 3 root root 4096 Nov 30 13:32 ..
drwxr-xr-x 2 root root 4096 Nov 30 13:32 bar
/root/.m2/repository/com/foo/bar:
total 12
drwxr-xr-x 2 root root 4096 Nov 30 13:32 .
drwxr-xr-x 3 root root 4096 Nov 30 13:32 ..
-rw-r--r-- 1 root root 5 Nov 30 13:32 baz.txt
---> b997ccbfd5b0
Step 3/46 : RUN ls -R -a -l /root/.m2/repository/com/foo
---> Running in 603671c87ecc
ls: /root/.m2/repository/com/foo: No such file or directory
The command '/bin/sh -c ls -R -a -l /root/.m2/repository/com/foo' returned a non-zero code: 1
What's going on? (NB. this is a toy example, but there is a real issue in that JARs installed into the Maven repository seem to disappear between layers.)
The upstream maven image defines this directory as a volume. Once an image does this, you cannot reliably make changes to that directory in the image.
From their Dockerfile:
ARG USER_HOME_DIR="/root"
...
VOLUME "$USER_HOME_DIR/.m2"
The Dockerfile documentation describes this behavior:
Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.
Your options are to:
Use another directory for your build
Request that the upstream image removes this VOLUME definition
Build your own image without this definition (it's fairly easy to fork their repo and do your own build)
For more details, you can see an old blog post by me about this behavior and the problems it creates.

Sh : file not found on alpine Docker image

I installed Docker on my Raspberry Pi3.
I want to install the "registry" image. As I do not find it in ARM, I have built an AMR version.
I recompiled the binary from the git https://github.com/docker/distribution/
I get a functional binary:
# registry --version
src_docker/go/bin/registry github.com/docker/distribution v2.6.0+unknown
Then I left the git https://github.com/docker/distribution-library-image to customize the Dockerfile.
FROM arm32v6/alpine:3.5
RUN set -ex \
&& apk add --no-cache ca-certificates apache2-utils
COPY ./registry/registry /bin/registry
COPY ./registry/config-example.yml /etc/docker/registry/config.yml
VOLUME ["/var/lib/registry"]
EXPOSE 5000
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/etc/docker/registry/config.yml"]
I have of course to replace the "registry / registry" by the binary that I compiled.
I build image :
docker build . -t ikise_arm/registry
But when I execute the image it does not find the binary
# docker run -ti ikise_arm/registry:latest
/entrypoint.sh: exec: line 10: registry: not found
If I disable the ENTRYPOINT and explore the files I find the biary but impossible to execute
# docker run -ti ikise_arm/registry:latest
/ # ls -al /bin/r*
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/reformime -> /bin/busybox
-rwxr-xr-x 1 root root 19940588 Jun 14 12:27 /bin/registry
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/rev -> /bin/busybox
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/rm -> /bin/busybox
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/rmdir -> /bin/busybox
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/run-parts -> /bin/busybox
/ # /bin/registry
sh: /bin/registry: not found
Do you have any idea where the problem may come from?
So I adapted the image https://github.com/docker-library/golang/tree/master/1.8/alpine3.6
To compile the binary registry
(https://github.com/gliderlabs/docker-alpine/issues/78)

Can't access mounted volume in docker

UPDATE: This wasn't an issue when I setup my project on a Mac. I was using Fedora 24 in this problem.
I am trying to access my app in my docker instance. When I try and ls the mounted directory, I get a permission error:
root#591d02d0d6d2:/app#
ls: cannot open directory .: Permission denied
This is what my docker file looks like:
FROM pvlltvk/ubuntu-trusty-php-fpm-5.6
RUN apt-get install -y\
php5-curl \
php5-sybase \
freetds-dev \
libxml2-dev
ADD freetds.conf /etc/freetds/freetds.conf
USER root
RUN echo 'alias sf="php /app/app/console"' >> ~/.bashrc
WORKDIR /app
I know it's a permissions issue but I couldn't get any solutions to work. If I run ls -lh I can see that the owner id is 1000 and not root.
root#591d02d0d6d2:/# ls -lh
total 24K
drwxrwxr-x. 11 1000 1000 4.0K Dec 12 17:42 app //my project
drwxr-xr-x. 2 root root 4.0K Dec 13 08:25 bin
drwxr-xr-x. 2 root root 6 Apr 10 2014 boot
drwxr-xr-x. 5 root root 360 Dec 15 02:58 dev

Resources