How to automatically deploy to VPS in Docker from Gitlab CI - docker

I'm trying to deploy to Docker in my VPS everytime a new commit is made in my project in Gitlab. But I'm having an issue doing that.
I tried to install sshpass and then scp folder and files. But it's saying:
sshpass: Failed to run command: No such file or directory.
The folder and files I'm trying to get from build stage, so I dont have to build again my app.
Here's my .gitlab-ci.yml file:
image: node:9.6.1
cache:
paths:
- node_modules/
- build/
- docker-compose.yml
- Dockerfile
- nginx.conf
stages:
- build
- test
- dockerize
build-stage:
stage: build
script:
- npm install
- CI=false npm run build
artifacts:
paths:
- build/
- docker-compose.yml
- nginx.conf
test-stage:
stage: test
script:
- npm install
- CI=false npm run test
dockerize-stage:
stage: dockerize
image: tmaier/docker-compose:latest
services:
- docker:dind
dependencies:
- build-stage
tags:
- docker
script:
- apk update && apk add sshpass
- sshpass -V
- export SSHPASS=$USER_PASS
- ls -la
- sshpass -e ssh -o stricthostkeychecking=no root#ip:/home mkdir new-super-viva
- sshpass -e scp -o stricthostkeychecking=no -r build user#ip:/home/new-folder
- sshpass -e scp -o stricthostkeychecking=no -r docker-compose.yml user#ip:/home/new-folder
- sshpass -e scp -o stricthostkeychecking=no -r nginx.conf user#ip:/home/new-folder
- sshpass -e ssh -o stricthostkeychecking=no user#ip:/home/new-folder docker-compose up --build
Here's the actual output from Gitlab CI:
$ sshpass -V
sshpass 1.06
(C) 2006-2011 Lingnu Open Source Consulting Ltd.
(C) 2015-2016 Shachar Shemesh
This program is free software, and can be distributed under the terms of the GPL
See the COPYING file for more information.
Using "assword" as the default password prompt indicator.
$ export SSHPASS=$USER_PASS
$ ls -la
total 912
drwxrwxrwx 7 root root 4096 Apr 2 13:24 .
drwxrwxrwx 4 root root 4096 Apr 2 13:24 ..
-rw-rw-rw- 1 root root 327 Apr 2 13:24 .env
drwxrwxrwx 6 root root 4096 Apr 2 13:24 .git
-rw-rw-rw- 1 root root 329 Apr 2 13:24 .gitignore
-rw-rw-rw- 1 root root 1251 Apr 2 13:24 .gitlab-ci.yml
-rw-rw-rw- 1 root root 311 Apr 2 11:57 Dockerfile
-rw-rw-rw- 1 root root 2881 Apr 2 13:24 README.md
drwxrwxrwx 5 root root 4096 Apr 2 13:20 build
-rw-rw-rw- 1 root root 340 Apr 2 13:24 build.sh
-rw-rw-rw- 1 root root 282 Apr 2 11:57 docker-compose.yml
-rw-rw-rw- 1 root root 1385 Apr 2 11:57 nginx.conf
drwxr-xr-x 1191 root root 36864 Apr 2 13:22 node_modules
-rw-rw-rw- 1 root root 765929 Apr 2 13:24 package-lock.json
-rw-rw-rw- 1 root root 1738 Apr 2 13:24 package.json
drwxrwxrwx 4 root root 4096 Apr 2 13:24 public
drwxrwxrwx 10 root root 4096 Apr 2 13:24 src
$ sshpass -e ssh -o stringhostkeychecking=no user#ip:/home mkdir new-folder
sshpass: Failed to run command: No such file or directory
ERROR: Job failed: exit code 3
Is there any way that I can copy the build folder, docker-compose.yml and nginx.conf files from build-stage to dockerize-stage and then pass it with sshpassto VPS? It doesnt even working to create folder with mkdir. I also tried to create folder manually and then remove this command from .gitlab-ci.yml but still the same output.
Just so you know I added that USER_PASS in:
https://gitlab.com/user/project/settings/ci_cd in environment variables and let it non-protected

Related

Why does docker image content differ from the container created from it?

Following is the Dockerfile for the image,
FROM jenkins/jenkins:lts-jdk11
USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean:1.25.2 http_request" && ls -la /var/jenkins_home
When this is built using docker build -t ireshmm/jenkins:lts-jdk11 ., following is the output,
Sending build context to Docker daemon 3.072kB
Step 1/3 : FROM jenkins/jenkins:lts-jdk11
---> 9aee0d53624f
Step 2/3 : USER jenkins
---> Using cache
---> 49d657d24299
Step 3/3 : RUN jenkins-plugin-cli --plugins "blueocean:1.25.2 http_request" && ls -la /var/jenkins_home
---> Running in b459c4c48e3e
Done
total 20
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:49 .
drwxr-xr-x 1 root root 4096 Jan 12 15:46 ..
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:49 .cache
-rw-rw-r-- 1 jenkins root 7152 Jan 12 15:42 tini_pub.gpg
Removing intermediate container b459c4c48e3e
---> 5fd5ba428f1a
Successfully built 5fd5ba428f1a
Successfully tagged ireshmm/jenkins:lts-jdk11
When create a container and list files docker run -it --rm ireshmm/jenkins:lts-jdk11 ls -la /var/jenkins_home, following is the output:
total 40
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:51 .
drwxr-xr-x 1 root root 4096 Jan 12 15:46 ..
-rw-r--r-- 1 jenkins jenkins 4683 Jan 22 16:51 copy_reference_file.log
drwxr-xr-x 2 jenkins jenkins 16384 Jan 22 16:51 plugins
-rw-rw-r-- 1 jenkins root 7152 Jan 12 15:42 tini_pub.gpg
Question: Why do the contents of /var/jenkins_home differ while building the image and the inside the container created from it given that no command is run after listing the files while building image? How can that happen?
The jenkins/jenkins:lts-jdk11 has an ENTRYPOINT that runs /usr/local/bin/jenkins.sh, which among other things creates the copy_reference_file.log file:
$ grep -i copy_reference /usr/local/bin/jenkins.sh
: "${COPY_REFERENCE_FILE_LOG:="${JENKINS_HOME}/copy_reference_file.log"}"
touch "${COPY_REFERENCE_FILE_LOG}" || { echo "Can not write to ${COPY_REFERENCE_FILE_LOG}. Wrong volume permissions?"; exit 1; }
echo "--- Copying files at $(date)" >> "$COPY_REFERENCE_FILE_LOG"
find "${REF}" \( -type f -o -type l \) -exec bash -c '. /usr/local/bin/jenkins-support; for arg; do copy_reference_file "$arg"; done' _ {} +
The ENTRYPOINT scripts runs whenever you start a container from that image (before any command you've provided on the command line).

docker-compose build fail with file not found

I would like to ask a question regarding the docker-compose.yml and Dockerfile.
Following is the situation I have met.
I first use docker image build (docker image build -t test .) and docker run (docker run -p 8888:8080 -p 8889:8009 8883:8443 -v tomcat8:/usr/local/tomcat8 --name test test) to test whether my Dockerfile is ready to go, and it is working.
Dockerfile
FROM ubuntu:latest
ENV HOME /usr/local/tomcat8
RUN apt-get -y update && apt-get -y install unzip wget openjdk-8-jdk
COPY ./installERDDAP.sh $HOME/installERDDAP.sh
COPY . $HOME/
RUN chmod +x $HOME/installERDDAP.sh
WORKDIR $HOME
RUN ./installERDDAP.sh
EXPOSE 8080 8443 8009
CMD ["catalina.sh","run"]
So that I script the docker-compose.yml, like below
docker-compose.yml
version: '3'
services:
erddap:
build:
context: ./
dockerfile: Dockerfile
image: erddap_dev:2.02
container_name: erddap
restart: always
ports:
- "8888:8080"
- "8883:8443"
- "8889:8009"
volumes:
- ./tomcat8:/usr/local/tomcat8
However, the error message popup and said that the file catalina.sh does not exist. When I get into the container, well, that is true. There are no files existed. But if I put RUN ls -la in the Dockerfile, I do see the files are listed, like below.
Step 12/14 : RUN ls -la
---> Running in fed4870bd617
total 176
drwxr-xr-x. 1 root root 4096 Sep 11 20:08 .
drwxr-xr-x. 1 root root 21 Sep 11 20:08 ..
drwxrwxr-x. 8 root root 166 Sep 11 19:31 .git
-rw-rw-r--. 1 root root 12 Sep 11 07:50 .gitignore
-rw-r--r--. 1 root root 165 Sep 11 20:08 .wget-hsts
-rw-r-----. 1 root root 19318 Jun 30 21:53 BUILDING.txt
-rw-r-----. 1 root root 5408 Jun 30 21:53 CONTRIBUTING.md
-rw-rw-r--. 1 root root 640 Sep 11 20:03 Dockerfile
-rw-r-----. 1 root root 57011 Jun 30 21:53 LICENSE
-rw-r-----. 1 root root 1726 Jun 30 21:53 NOTICE
-rw-r-----. 1 root root 3255 Jun 30 21:53 README.md
-rw-r-----. 1 root root 7136 Jun 30 21:53 RELEASE-NOTES
-rw-r-----. 1 root root 16262 Jun 30 21:53 RUNNING.txt
drwxr-x---. 2 root root 4096 Sep 11 20:08 bin
-rwxr-x---. 2 root root 25245 Jun 30 21:50 catalina.sh
drwx------. 2 root root 238 Jun 30 21:53 conf
drwxr-xr-x. 3 root root 20 Sep 11 20:08 content
drwxr-xr-x. 2 root root 6 Sep 11 20:08 data
drwxrwxr-x. 5 root root 44 Sep 11 03:46 doc
-rw-rw-r--. 1 root root 320 Sep 11 19:49 docker-compose.yml
-rwxrwxr-x. 1 root root 1030 Sep 11 19:41 installERDDAP.sh
drwxr-x---. 2 root root 4096 Sep 11 20:08 lib
drwxr-x---. 2 root root 6 Jun 30 21:49 logs
drwxr-xr-x. 2 root root 66 Sep 11 20:08 tarz
drwxr-x---. 2 root root 30 Sep 11 20:08 temp
drwxrwxr-x. 4 root root 29 Sep 11 19:41 tomcat8
drwxr-x---. 7 root root 99 Sep 11 20:08 webapps
drwxr-x---. 2 root root 6 Jun 30 21:49 work
Removing intermediate container fed4870bd617
Interestingly, these files are not also showing up in my volume directory. Could please someone tells me where am I wrong?
Thanks
1st Updates:
According to the suggestion below, if I commended out the volumes within docker-compose.yml or if I added WORKDIR / in front of the CMD command, none of them are working.
If I modified CMD commend in Dockfile from CMD ["catalina.sh","run"] to CMD ["./catalina.sh","run"] is not working, too. But if I changed to CMD ["/usr/local/tomcat8/catalina.sh","run"], the build process is completed. But the errors than become
Attaching to erddap
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
erddap | Cannot find /usr/local/bin/setclasspath.sh
erddap | This file is needed to run this program
2nd Updates:
installERDDAP.sh
#!/bin/bash
mkdir tarz
wget -q https://ftp.wayne.edu/apache/tomcat/tomcat-8/v8.5.57/bin/apache-tomcat-8.5.57.tar.gz -O ./tarz/apache-tomcat-8.5.57.tar.gz
tar -xf ./tarz/apache-tomcat-8.5.57.tar.gz -C /usr/local/tomcat8 --strip 1
mkdir data
ln ./bin/catalina.sh catalina.sh
From what I can tell your bind mount may be "overriding/hiding" your container files. If your "./tomcat8" directory is not empty it will basically "override/hide" the files in your container. I would remove the volume declaration to validate if this is the case, if so then you would need to clear out your "./tomcat8" directory and when you run the container again it should populate it with the files in the container, from then forward your local directory will again "override/hide" your container files.
In your docker file before the line CMD ["catalina.sh","run"] put WORKDIR / And change "catalina.sh" to "./catalina.sh"

How to save data from a docker container on a local host?

I run R-Studio in a container on GitLab. R-Studio build a lot of csv and pdf files. When I run
docker run --rm -it registry.gitlab.com/user/paperboy /bin/bash
I can find in the folder /home/output/csv and /home/output/pdf the files. I will save all this files in a /output/csv and /output/pdf files on a host, in my case on GitLab. The question is how to save data outside the docker Container?
Here is my Dockerfile.
FROM rocker/r-base:latest
RUN apt-get update \
&& apt-get install -yq --no-install-recommends groff \
&& rm -rf /var/lib/apt/lists/*
# Create directories
RUN mkdir -p /home/output/ /home/output/csv/ /home/output/pdf/ /home/script/
WORKDIR /home/script
# Install R-packages
COPY /src/install_packages.R /home/script/install_packages.R
RUN Rscript /home/script/install_packages.R
# Copy data
COPY /src/pairs.csv /home/script/pairs.csv
COPY /src/master.R /home/script/master.R
COPY /src/paperboy.ms /home/script/paperboy.ms
# Run the script
RUN ["Rscript", "master.R"]
$ docker run -d
-v $(pwd)/output/:/home/output
-v $(pwd)/output/csv/:/home/output/csv
-v $(pwd)/output/pdf/:/home/output/pdf
$CONTAINER_IMAGE/$DOCKER_IMAGE
5d11eb7e3d93e8b98b6381f1970c25be426ff67abef5e378b715263f174849c9
This is a part from the .gitlab-ci.yml
run:
stage: run
script:
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS#gitlab.com/$CI_PROJECT_PATH.git
- git config --global user.name ""
- git config --global user.email ""
- git checkout
- docker login registry.gitlab.com --username gitlab+deploy-token-aaaa --password bbbb
- docker pull $CONTAINER_IMAGE/$DOCKER_IMAGE
- docker image ls
- docker run -t -d
-v $(pwd)/output/:/home/output
-v $(pwd)/user/paperboy/output/csv/:/home/output/csv
-v $(pwd)/user/paperboy/output/pdf/:/home/output/pdf
$CONTAINER_IMAGE/$DOCKER_IMAGE
- rm -rf "%CACHE_PATH%/%CI_PIPELINE_ID%"
- pwd
- ls -la
- ls -laR output
- git status
only:
- master
The csv and pdf folder are empty.
$ ls -laR output
output:
total 32
drwxr-xr-x 4 root root 4096 Oct 8 11:37 .
drwxrwxrwx 5 root root 4096 Oct 8 11:37 ..
drwxr-xr-x 2 root root 4096 Oct 8 11:37 csv
drwxr-xr-x 2 root root 4096 Oct 8 11:37 pdf
output/csv:
total 16
drwxr-xr-x 2 root root 4096 Oct 8 11:37 .
drwxr-xr-x 4 root root 4096 Oct 8 11:37 ..
output/pdf:
total 16
drwxr-xr-x 2 root root 4096 Oct 8 11:37 .
drwxr-xr-x 4 root root 4096 Oct 8 11:37 ..

Sh : file not found on alpine Docker image

I installed Docker on my Raspberry Pi3.
I want to install the "registry" image. As I do not find it in ARM, I have built an AMR version.
I recompiled the binary from the git https://github.com/docker/distribution/
I get a functional binary:
# registry --version
src_docker/go/bin/registry github.com/docker/distribution v2.6.0+unknown
Then I left the git https://github.com/docker/distribution-library-image to customize the Dockerfile.
FROM arm32v6/alpine:3.5
RUN set -ex \
&& apk add --no-cache ca-certificates apache2-utils
COPY ./registry/registry /bin/registry
COPY ./registry/config-example.yml /etc/docker/registry/config.yml
VOLUME ["/var/lib/registry"]
EXPOSE 5000
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/etc/docker/registry/config.yml"]
I have of course to replace the "registry / registry" by the binary that I compiled.
I build image :
docker build . -t ikise_arm/registry
But when I execute the image it does not find the binary
# docker run -ti ikise_arm/registry:latest
/entrypoint.sh: exec: line 10: registry: not found
If I disable the ENTRYPOINT and explore the files I find the biary but impossible to execute
# docker run -ti ikise_arm/registry:latest
/ # ls -al /bin/r*
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/reformime -> /bin/busybox
-rwxr-xr-x 1 root root 19940588 Jun 14 12:27 /bin/registry
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/rev -> /bin/busybox
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/rm -> /bin/busybox
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/rmdir -> /bin/busybox
lrwxrwxrwx 1 root root 12 Mar 2 20:19 /bin/run-parts -> /bin/busybox
/ # /bin/registry
sh: /bin/registry: not found
Do you have any idea where the problem may come from?
So I adapted the image https://github.com/docker-library/golang/tree/master/1.8/alpine3.6
To compile the binary registry
(https://github.com/gliderlabs/docker-alpine/issues/78)

Edit apache configuration in docker

First time docker user here, I'm using this image: https://github.com/dgraziotin/osx-docker-lamp
I want to make the apache in that container to use a configuration file from the host system. How do I do that?
I know I can use nsenter, but I think my changes will get deleted when the container is turned off.
Thank you
The best solution is using VOLUME.
docker pull dgraziotin/lamp
You need to copy /etc/apache2/ from container to current directory in host computer. Then you can do this:
cd ~
mkdir conf
docker run -i -t --rm -v ~/conf:/tmp/conf dgraziotin/lamp:latest /bin/bash
On container do:
ls /tmp/conf
cd /etc/apache2/
tar -cf /tmp/conf/apache-conf.tar *
exit
On host computer:
cd conf
tar -xf apache-conf.tar
cd ..
# alter your configuration in this file and save
vi conf/apache2.conf
# run your container : daemon mode
docker run -d -p 9180:80 --name web-01 -v ~/conf:/etc/apache2 dgraziotin/lamp:latest
docker ps
To list conf content on Container use:
docker exec web-01 ls -lAt /etc/apache2/
total 72
-rw-r--r-- 1 root root 1779 Jul 17 20:24 envvars
drwxr-xr-x 2 root root 4096 Apr 10 11:46 mods-enabled
drwxr-xr-x 2 root root 4096 Apr 10 11:45 sites-available
-rw-r--r-- 1 root root 7136 Apr 10 11:45 apache2.conf
drwxr-xr-x 2 root root 4096 Apr 10 11:45 mods-available
drwxr-xr-x 2 root root 4096 Apr 10 11:44 conf-enabled
drwxr-xr-x 2 root root 4096 Apr 10 11:44 sites-enabled
drwxr-xr-x 2 root root 4096 Apr 10 11:44 conf-available
-rw-r--r-- 1 root root 320 Jan 7 2014 ports.conf
-rw-r--r-- 1 root root 31063 Jan 3 2014 magic
Use docker exec web-01 cat /etc/apache2/apache2.conf to list content inside Container.
One the WEB page to test your environment.
I hope this help you.
You should use a Dockerfile to generate a new image containing your desired configuration. For example:
FROM dgraziotin/lamp
COPY my-config-file /some/configuration/file
This assumes that there is a file my-config-file located in the same directory as the Dockerfile. Then run:
docker build -t myimage
And once the build completes you will have an image named myimage available locally.

Resources