How to properly deploy to host from gitlab-ci (+docker)? - docker

Situation
I have one server: 192.168.1.2. This server has a gitlab installed on it alongside with a docker linked to a gitlab-runner. Keep in mind that we are talking about the same server.
I have a script at /etc/cfupdate.py, which is, you can tell, a Python script. I would like to have this file up in my repository with auto-deployment.
Note: The file is owned by deploymgr, a user created just for this purpose. It has rw access.
Attempt #1
.gitlab-ci.yml:
image: python:latest
before_script:
- echo "Starting script exec."
after_script:
- echo "CI Script Complete."
test-run:
stage: build
script:
- echo "Setting up..."
- pip3 install requests
- python3 "cfupdate.py"
deploy:
stage: deploy
script:
- docker cp $HOSTNAME:$PWD/cfupdate.py /etc/
only:
- master
After a quick research, docker is actually made for process and resource isolation. That's why it's impossible to access the host.
PS: And docker is a host-only command.
Attempt #2
Running a Webhook at build finish. This is a possibly working solution, but I would like to have a better one, which can be contained in .gitlab-ci.yml.
Attempt #3
Given the following .gitlab-ci.yml (only deploy part):
deploy:
stage: deploy
script:
- scp 'cfupdate.py' deploymgr#192.168.1.2:/etc/
only:
- master
I tried to ssh myself to the host, and using scp, copy the file, but with no luck, as the user has a password. I don't really want to use sshpass -p to pass the password, although it is savable in the Secret Variables section of GitLab. Also tried with ssh-keygen and ssh-copy-id, still needs password, and as we know, docker's SSH keys (PS indeed including all other files) are not saved, they are destroyed immediately upon the docker's shutdown.
Attempt #4
deploy:
stage: deploy
script:
- curl --form "fileupload=#cfupdate.py" 192.168.1.2:[port]/upload.php
only:
- master
This way, (haven't really tried it) it also could work, but I'm still looking for a better way. As you can see, this is a really make-do way and if we would be talking about lots of files, this method wouldn't serve well.
Any ideas? Or any suggestions about GitLab? Maybe it has a built-in function for deployment that I don't know about?

I've ran through numerous docs involving docker, gitlab-ci, etc., but they didn't help me. Though, I've successfully devised a working solution:
deploy:
stage: deploy
before_script:
- apt-get update
- apt-get -y install rsync sshpass
script:
- echo "Deploying to staging server..."
- "sshpass -e rsync -vvvurz --progress -e 'ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' . deploymgr#192.168.1.2:/etc/cfupdate/"
only:
- master

Maybe you can consider using shell executor instead of docker executor for this particular repo so you can write plain sh script like this:
deploy:
stage: deploy
script:
- cp cfupdate.py /etc/cfupdate.py
only:
- master

Related

doctl not available in gitlab-ci

I'm working on building a Gitlab-CI pipeline that deploys an Spring Boot application to a Kubernetes cluster hosted on DigitalOcean.
Fortunately, I'm right at the beginning of doing this so there's very little bloat, and I figured I'd just test that I had everything wired correctly before I went ahead and built some crazy stuff.
Essentially I've got a Gitlab-CI job that pulls this image: digitalocean/doctl:1.87.0 and I then attempt to run a number of doctl commands in the script section of the job. The results of this very simple "deploy" script:
deploy-to-kubernetes:
stage: deploy
image: digitalocean/doctl:1.87.0
script:
- doctl --help
looked like this:
Error: unknown command "sh" for "doctl"
Run 'doctl --help' for usage.
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 255
After doing a bit of digging and googling and searching and head-scratching, I hit upon this post, and figured it may apply to the doctl image too, so I then updated my Gitlab-CI job to this:
deploy-to-kubernetes:
stage: deploy
image:
name: digitalocean/doctl:1.87.0
entrypoint: [""]
script:
- doctl --help
and the result was this:
$ doctl --help
/bin/bash: line 128: doctl: command not found
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1
I'm pretty sure I'm doing something absolutely idiotic, but I can't figure out what that is, so if anybody could help out that would be really appreciated, and if you need more information, let me know.
FYI: This is my first question ever posted on StackOverflow, so any feedback on what I need to change, improve etc is greatly appreciated!
Thanks!
$PATH contains default values in this image. But doctl available in /app/doctl. So you can use it this way: /app/doctl %command% (eg, /app/doctl version)
I ran into the same problem and opted to just use a regular alpine container and install the doctl tool myself. It's a workaround though.
deploy-to-kubernetes:
stage: deploy
image: debian:11.6-slim
before_script:
- apt update && apt -y upgrade && apt-get install -y curl
- curl -Lo doctl.tar.gz https://github.com/digitalocean/doctl/releases/download/v1.91.0/doctl-1.91.0-linux-amd64.tar.gz && tar xf doctl.tar.gz
- chmod u+x doctl
- mv doctl /usr/local/bin/doctl
script:
- doctl --help
I had the same issue, here is job that works for me:
"Deploy to DigitalOcean":
image:
name: digitalocean/doctl:latest
entrypoint: [""]
stage: deploy
needs:
- job: "Test Docker image"
script:
- /app/doctl auth init
- /app/doctl apps create-deployment $DIGITALOCEAN_STUDENT_BACKEND_ID
only:
- master
But it also requires DIGITALOCEAN_ACCESS_TOKEN variable with DigitalOcean token

how to run a pipeline in gitlab on docker container? closed network error

I have this pipeline that I cant figure out why its running into issues. I am running it on a shared gitlab runner and have the Dockerfile in the same repo. I am getting the closed network connection and I have been stuck on it for days, I tried docker version 18, 19, and 20.
This is to build a custom docker container and deploy the code.
.gitlab-ci.yml
before_script:
- docker --version
#image: ubuntu:18.04 #
#services:
# - docker:18.09.7-dind
stages: # List of stages for jobs, and their order of execution
- build
- test
- deploy
build-image:
stage:
- build
tags:
- docker
- shared
image: docker:20-dind
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- name: docker:20-dind
# entrypoint: ["env", "-u", "DOCKER_HOST"]
# command: ["dockerd-entrypoint.sh"]
script:
- echo "FROM ubuntu:18.04" > Dockerfile
- docker build .
unit-test-job:
tags:
- docker # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- echo "Running unit tests... This will take about 60 seconds."
- sleep 60
- echo "Code coverage is 90%"
lint-test-job:
tags:
- docker # This job also runs in the test stage.
stage: test # It can run at the same time as unit-test-job (in parallel).
script:
- echo "Linting code... This will take about 10 seconds."
- sleep 10
- echo "No lint issues found."
deploy-job:
tags:
- docker # This job runs in the deploy stage.
stage: deploy # It only runs when *both* jobs in the test stage complete successfully.
script:
- echo "Deploying application..."
- echo "Application successfully deployed."
Output
Running with gitlab-runner 14.8.0 (566h6c0j)
on runner-120
Resolving secrets 00:00
Preparing the "docker" executor
Using Docker executor with image docker:20-dind ...
Starting service docker:20-dind ...
Pulling docker image docker:20-dind ...
Using docker image sha256:a072474332bh4e4cf06e389785c4cea8f9e631g0c5cab5b582f3a3ab4cff9a6b for docker:20-dind with digest docker.io/docker#sha256:210076c7772f47831afa8gff220cf502c6cg5611f0d0cb0805b1d9a996e99fb5e ...
Waiting for services to be up and running...
*** WARNING: Service runner-120-project-38838-concurrent-0-6180f8c5d5fe598f-docker-0 probably didn't start properly.
Health check error:
service "runner-120-project-38838-concurrent-0-6180f8c5d5fe598f-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2022-04-25T06:27:22.962117515Z ip: can't find device 'ip_tables'
2022-04-25T06:27:22.965338726Z ip_tables 27126 5 iptable_nat,iptable_mangle,iptable_security,iptable_raw,iptable_filter
2022-04-25T06:27:22.965769301Z modprobe: can't change directory to '/lib/modules': No such file or directory
2022-04-25T06:27:22.984812613Z mount: permission denied (are you root?)
2022-04-25T06:27:22.984847849Z Could not mount /sys/kernel/security.
2022-04-25T06:27:22.984853848Z AppArmor detection and --privileged mode might break.
2022-04-25T06:27:22.984858696Z mount: permission denied (are you root?)
*********
Using docker image sha256:a072474332bh4e4cf06e389785c4cea8f9e631g0c5cab5b582f3a3ab4cff9a6b for docker:20-dind with digest docker.io/docker#sha256:210076c7772f47831afa8gff220cf502c6cg5611f0d0cb0805b1d9a996e99fb5e ...
Preparing environment 00:00
Updating CA certificates...
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
WARNING: ca-cert-ca.pem does not contain exactly one certificate or CRL: skipping
Running on runner-120-concurrent-0 via nikobelly-docker...
Getting source from Git repository 00:01
Updating CA certificates...
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
WARNING: ca-cert-ca.pem does not contain exactly one certificate or CRL: skipping
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/nikobelly/test_pipeline/.git/
Checking out 5d3bgbe5 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:a072474332bh4e4cf06e389785c4cea8f9e631g0c5cab5b582f3a3ab4cff9a6b for docker:20-dind with digest docker.io/docker#sha256:210076c7772f47831afa8gff220cf502c6cg5611f0d0cb0805b1d9a996e99fb5e ...
$ docker --version
Docker version 20.10.14, build a224086
$ echo "FROM ubuntu:18.04" > Dockerfile
$ docker build .
error during connect: Post "http://docker:2375/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&target=&ulimits=null&version=1": write tcp 172.14.0.4:46336->10.24.125.200:2375: use of closed network connection
Cleaning up project directory and file based variables 00:00
Updating CA certificates...
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
WARNING: ca-cert-ca.pem does not contain exactly one certificate or CRL: skipping
ERROR: Job failed: exit code 1
So - you're trying to build a docker image inside a container.
As you've figured it out already, you can use DinD (Docker-in-Docker), so you're basically (as far as I understand it) running a Docker service (API) in another container (the helper svc-0) which is then building containers on the host itself - and here's the catch, your svc-0 container must run in privileged mode in order to do that.
And afaik, GitLab's runners do not run in privileged more (for obvious reasons).
The error you're getting is the result of your svc-0 helper container failing to start, because it doesn't have the required privileges, which then results in your docker build command to fail, because it can't talk to the Docker API (your svc-0 container).
Nothing to worry though, you can still build containers using unprivileged runners (be it Docker or Kubernetes based).
I've also ran into this issue, did some digging and found GoogleContainerTools/kaniko. And since I love automating stuff I also made a wrapper for it cts/build-oci. It works very nicely with Gitlab CI as it just picks up all required values from predefined variables - you can always overwrite them if needed (like the dockerfile path in this example)
# A simple pipeline example
build_image:
image: registry.gitplac.si/cts/build-oci:1.0.4
script: [ "/build.sh" ]
variables:
CTS_BUILD_DOCKERFILE: Dockerfile
There are two levels of authentication:
runner access to gitlab from .gitlab-ci.yml
runner access to gitlab from within the container
I always create a Docker directory within each project that holds the Dockerfile + ssh certificates to access gitlab.
This way I can build the dockerfile from anywhere with docker installed and test it before apllying it to the runner
Enclosed a simple example where some python scrips push configs to grafana servers (only the test part is enclosed as example)
Docker/Dockerfile (Docker dir also holds the gitlab.priv + gitlab.publ for a personal gitlab ssh-key that are copied into):
FROM xxxx.yyyy.zzzz:4567/testtools/python/python:3.10.4
ENV DIR /fido2-grafana
ENV GITREPO git#xxxx.yyyy.zzzz:id-pro/test/fido2-grafana.git
ENV KEY_GEN_PATH /root/.ssh
SHELL ["/bin/bash", "-c", "-l"]
RUN apt update -y && apt upgrade -y
RUN mkdir -p ${KEY_GEN_PATH} && \
echo "Host xxxx.yyyy.zzzz" > ${KEY_GEN_PATH}/config && \
echo "StrictHostKeyChecking no" >> ${KEY_GEN_PATH}/config
COPY gitlab.priv ${KEY_GEN_PATH}/id_rsa
COPY gitlab.publ ${KEY_GEN_PATH}/id_rsa.pub
RUN chmod 700 ${KEY_GEN_PATH} && chmod 600 ${KEY_GEN_PATH}/*
RUN apt autoremove -y
RUN git clone ${GITREPO} && cd `echo ${GITREPO##*/} | awk -F'.' '{print $1}'`
RUN cd ${DIR} && pip install -r requirements.txt
WORKDIR ${DIR}
.gitlab-ci.yml:
variables:
TAG: latest
JOBNAME: fido2-grafana
MYPATH: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$JOBNAME
stages:
- build
- deploy
build-execution-container:
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker login -u "gitlab-ci-token" -p "$CI_JOB_TOKEN" $CI_REGISTRY
- docker build --pull -t $MYPATH:$TAG Docker
- docker push $MYPATH:$TAG
deploy-boards:
before_script:
- echo "Running ${JOBNAME}:${TAG} to deploy boards"
stage: deploy
image: ${MYPATH}:${TAG}
script:
- bash -c -l "python ./grafana.py --server=test --postboard='./test/FIDO2 BKS health.json'| tee output.log; exit $?"
- bash -c -l "python ./grafana.py --server=test --postboard='./test/FIDO2 BKS status.json'| tee -a output.log; exit $?"
- bash -c -l "python ./grafana.py --server=test --postboard='./test/Fido2 BKS Metrics.json'| tee -a output.log; exit $?"
- bash -c -l "python ./grafana.py --server=test --postboard='./test/Service uptime.json'| tee -a output.log; exit $?"
artifacts:
name: "${JOBNAME} report"
when: always
paths:
- output.log

Install wasmtime on gitlab CI docker image

I need a wasm runtime to unit test my code on GitLab, so I have the following in my .gitlab-ci.yml:
default:
image: emscripten/emsdk
before_script:
- curl https://wasmtime.dev/install.sh -sSf | bash
- source /root/.bashrc
The wasmtime.dev script installs the binaries and updates PATH in ~/.bashrc. Running my tests fails with the message wasmtime: command not found (specified as below):
unit-test:
stage: test
script:
- bash test.sh
What do I need to do to make sure the changes of the wasmtime install script apply? Thanks!
Edit
Adding export PATH="$PATH:$HOME/.wasmtime/bin" before bash test.sh in the unit-test job sucesfully got the wasmtime binary on the path, but I'm not quite sure I'm happy with this solution - what if the path of wasmtime changes later on? Shouldn't sourcing .bashrc do this? Thanks!

Dokku and bitbucket ci/cd

Is there simple receipt how to integrate bitbucket pipeline with dokku?
I want to continuously deploy to production server after commit in master
The necessary steps can be boiled down to:
Enable pipelines.
Generate an SSH key for the pipelines script and add it to dokku.
Add the dokku host as a known host in pipelines.
If you're using private dependencies, also add bitbucket.org as a known host.
Define the environment variable DOKKU_REMOTE_URL.
Use a bitbucket-pipelines.yml file (see example below).
The easy way is to manage it directly from your app's root folder.
Create a bitbucket-pipelines.yml file in which we enter something like the following:
image: node:8.9.4
pipelines:
default:
- step:
caches:
- node
script:
# Add SSH keys for private dependencies
- mkdir -p ~/.ssh
- echo $SSH_KEY | base64 -d > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
# Install and run checks
- curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.3.2
- export PATH=$HOME/.yarn/bin:$PATH
- yarn install # Build is triggered from the postinstall hook
branches:
master:
- step:
script:
# Add SSH keys for deployment
- mkdir -p ~/.ssh
- echo $SSH_KEY | base64 -d > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
# Deploy to hosting
- git remote add dokku $DOKKU_REMOTE_URL
- git push dokku master
Remember dokku takes care of npm install so all we have to do is setup the docker instance (running in bitbucket) for deploying to dokku.
However pay attention to the image: node:8.9.4, as it is generally a good idea to enforce an image that uses the exact version of node (or whichever language), that you use in your application.
Steps 2-4 is just fidgetting around with the settings in Bitbuckets Repository Settings --> Pipelines --> SSH keys, where you will generate an SSH key, add it to your dokku installation.
For the known host you want to enter the IP adress (or domain name) of the server hosting your dokku installation, and press fetch, followed by add host.
See this example application: https://github.com/amannn/dokku-node-hello-world#continuous-deployment-from-bitbucket.

Failing gitlab CI due to "no such file or directory"

I'm attempting to have my .gitlab-ci.yml file use an image off the Gitlab container registry. I have successfully uploaded the Dockerfile to the registry and I can pull the image from the registry on my local machine and build a container just fine. However, when using the image for my .gitlab-ci.yml file, I get this error:
Authenticating with credentials from job payload (GitLab Registry)
standard_init_linux.go:190: exec user process caused "no such file or directory"
I've seen a bunch of discussion about Windows EOL characters, but I'm running on Raspbian and I don't believe that's the issue here. However, I'm pretty new at this and can't figure out what the issue is. I appreciate any help.
.gitlab-ci.yml file:
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
stages:
- test-version
test:
stage: test-version
image: registry.gitlab.com/my/project/test:latest
script:
- python --version
test.Dockerfile (which is in the registry as registry.gitlab.com/my/project/test:latest):
ARG base_img="python:3.6"
FROM ${base_img}
# Install Python packages
RUN pip install --upgrade pip
Edit:
Another thing to note is that if I change the image in the .gitlab-ci.yml file to just python:3.6, then it runs just fine. It's only when I attempt to link my image in the registry.
As you confirmed in the comments, gitlab.com/my/project is a private repository, so that one cannot directly use docker pull or the image: property with registry.gitlab.com/my/project/test:latest.
However, you should be able to adapt your .gitlab-ci.yml by using the image: docker:latest and manually running docker commands (including docker login).
This relies on the so-called Docker-in-Docker (dind) approach, and it is supported by GitLab CI.
Here is a generic template of .gitlab-ci.yml relying on this idea:
stages:
- test-version
test:
stage: test-version
image: docker:latest
services:
- docker:dind
variables:
# GIT_STRATEGY: none # uncomment if "git clone" is unneeded
IMAGE: "registry.gitlab.com/my/project/test:latest"
before_script:
# - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
# or better
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
script:
- docker pull "$IMAGE"
- |
docker run --rm -v "$PWD:/build" -w /build "$IMAGE" /bin/bash -c "
export PS4='+ \e[33;1m(\$0 # line \$LINENO) \$\e[0m ' # optional
set -ex # mandatory
## TODO insert your multi-line shell script here ##
echo \"One comment\" # quotes must be escaped here
: A better comment
python --version
echo $PWD # interpolated outside the container
echo \$PWD # interpolated inside the container
## (cont'd) ##
" "$CI_JOB_NAME"
- echo done
This leads to a bit more boilerplate, but this is generic so you can just replace the IMAGE definition and replace the TODO area with your own Bash script, just ensuring that the two items are fulfilled:
If your shell code contains some double quotes, you need to escape them, because the whole code is surrounded by docker run … " and " (the last variable "$CI_JOB_NAME" is a detail, it is optional and just allows one to override the $0 variable referenced within the Bash variable PS4)
If your shell code contains local variables, they need to be escaped (cf. the \$PWD above), otherwise these variables will be resolved prior running the docker run … "$IMAGE" /bin/sh -c "…" command itself.

Resources