Gitlab CI - docker: command not found - docker

I am trying to build my docker image within the gitlab ci pipeline.
However it is not able to find the docker command.
/bin/bash: line 69: docker: command not found ERROR: Job failed: error
executing remote command: command terminated with non-zero exit code:
Error executing in Docker Container: 1
.gitlab-ci.yml
stages:
- quality
- test
- build
- deploy
image: node:8.11.3
services:
- mongo
- docker:dind
before_script:
- npm install
quality:
stage: quality
script:
- npm run-script lint
test:
stage: test
script:
- npm run-script test
build:
stage: build
script:
- docker build -t server .
deploy:
stage: deploy
script:
- echo "TODO deploy push docker image"

you need to choose an image including docker binaries
image: gitlab/dind
services:
- docker:dind

You have 2 options to fix this. You will need to edit your config.toml file (located wherever you installed your gitlab runner).
OPTION 1
in config.toml:
privileged = true
in .gitlab-ci.yml:
myjob:
stage: myjob
image: docker:latest
services:
- docker:18.09.7-dind # older version that does not need demand TLS (see below)
OPTION 2
in config.toml:
privileged = true
volumes = ["/certs/client", "/cache"]
in .gitlab-ci.yml:
myjob:
stage: myjob
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay2 # not sure if this is needed
DOCKER_TLS_CERTDIR: "/certs"
IMPORTANT: ONCE YOU HAVE MADE THE CHANGES TO config.toml YOU WILL PROBABLY NEED TO RESTART THE GITLAB RUNNER (which may vary depending on OS) - I DID RESTART MINE, NOT SURE WHAT WOULD HAPPEN IF YOU DID NOT RESTART IT!
Instructions for restarting gitlab runner are here ... https://docs.gitlab.com/runner/commands/ ... basically gitlab-runner restart but on Windows I had to use Windows "Services" to restart it
Why this problem?
priviledged=true gets rid of the docker: command not found problem
However, docker:dind now requires TLS certs (whatever they are). If you are happy with an older docker version then you can use OPTION 1. If you want the latest you need to setup Gitlab CLI to use them which is OPTION 2. J.E.S.U.S loves you :)
For more info ... https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03

Problem here is that node docker image does not embed docker binaries.
Two possibilities :
split stages to two jobs. One using node images for quality and test, one using docker image for building and deploying. See jobs documentation.
build a custom docker image that embed both node and docker and use this image to build your repo.
Note that in both case you will have to enable docker inside your agent. See documentation.

Related

Docker-in-Docker doesn't work on self hosted linux runner

I'm having trouble utilizing docker commands in af self hosted linux runner.
Reading the docs it should work more or less out of the box, just like when using atlassians own runners.
however, when running a docker command i get an error:
+ docker version
bash: docker: command not found
The relevant part of the pipelines yml file:
pipelines:
branches:
'master':
- step:
name: 'step1'
script:
- docker version //this works
services:
- docker
- step:
name: 'step2'
runs-on:
- self.hosted
- linux
script:
- docker version //this fails
services:
- docker
The only self hosted runner specific mentions of docker commands, is the new addition of using custom images to run the docker daemon inside a runner, but as i understand it, running the default should work, also on selfhosted runners.
https://support.atlassian.com/bitbucket-cloud/docs/configure-your-runner-in-bitbucket-pipelines-yml#Custom-docker-in-docker-image
Am i missing that should be done when starting the runner, or is this not supported (yet) ?
I've asked the same question on atlassians community: https://community.atlassian.com/t5/Bitbucket-questions/Selfhosted-runner-cannot-use-docker-commands/qaq-p/2186491#M87567
Will answer this question, if i get an answer from there.
My question was answered on Atlassians community, and the solution was to use the image docker:dind as the Docker image.
You can add the "definitions:" configuration below to your YAML file above the "pipelines:" config.
definitions:
services:
docker:
image: docker:dind
pipelines:
branches:
'master':
- step:
...

Can't connect to Docker daemon in my GitLab CI pipeline

I am trying to build a super-simple CI/CD pipeline using GitLab CI.
Upon running it I get presented with the error:
Server:
ERROR: Cannot connect to the Docker daemon at tcp://docker:2375.
Is the docker daemon running?
My .gitlab-ci.yml is :
image: docker:latest
variables:
DOCKER_HOST: tcp://docker:2375
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
before_script:
- docker --version
docker_build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker build -t arieltar/hubsec:1.1 .
- docker push arieltar/hubsec:1.1
Based on the error message I would ask, does the gitlab-runner user belong to the docker group?
You will need to decide if you want to use Docker-in-Docker with, or without TLS. This requires changing /etc/gitlab-runner/config.toml settings, and assigning the DOCKER_TLS_CERTDIR in your .gitlab-ci.yml file. See the Docker-in-docker section of the GitLab docs.
Please check below things as prelim.
Whether docker is running or not
Login with gitlab-user if you are running pipeline with gitlab user and check if that user can access or run docker ps without sudo :).
add below entry if pt1. and pt2 satisfied.
services:
name: docker:dind
entrypoint: ["dockerd-entrypoint.sh", "--tls=false"]
script:
export DOCKER_HOST=tcp://127.0.0.1:2375 && docker build -t arieltar/hubsec:1.1 .

Check docker run in Gitlab CICD pipeline

I'm using Gitlab CI/CD to build Docker images of our Node server.
I am wondering if there is a way to test that docker run of the image was ok.
We've had few occasions where the Docker builds but it is missing some files/env variables and it fails to start the server.
Is there any way to run the docker image and test if it is starting up correctly in the CI/CD pipeline?
Cheers.
With Gitlab you are able to use a docker-runner.
When you use the docker-runner, and not a shell runner, a docker-like
image and its services have to initiate, it should give an error if
something fails.
Chek this docs from gitlab:
This is a classic yml from that web:
default:
image:
name: ruby:2.2
entrypoint: ["/bin/bash"]
services:
- name: my-postgres:9.4
alias: db-postgres
entrypoint: ["/usr/local/bin/db-postgres"]
command: ["start"]
before_script:
- bundle install
test:
script:
- bundle exec rake spec
As you see, the test sections will be executed after building the image, so, you should not have to worry about. Gitlab should detect any errors when loading the image
If you are doing it with the shell gitlab-runner, you should call the
docker image start like this:
stages:
- dockerStartup
- build
- test
- deploy
- dockerStop
job 0:
stage: dockerStartup
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests
[...] //your jobs here
job 5:
stage: dockerStop
script: docker stop whatever

Gitlab CI - Build Docker Image With Shared Runner (cannot connect to Docker Daemon)

I am currently using Gitlab Shared Runners to build and deploy my project (at least I'm trying too !).
I have the gitlab-ci.yml below :
image: java:8-jdk
stages:
- build
- package
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
- docker info
cache:
paths:
- .gradle/wrapper
- .gradle/caches
build:
stage: build
script:
- ./gradlew build
artifacts:
paths:
- build/libs/*.jar
expire_in: 1 week
only:
- master
docker-build:
image: docker:stable
services:
- docker:dind
stage: package
script:
docker build -t registry.gitlab.com/my-project .
docker push registry.gitlab.com/my-project
after_script:
- echo "End CI"
First, build stage is doing great, but there is a problem with the second stage when I'm trying to build and push my docker image.
I get this log :
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
It seems that Gitlab is using a shared runner that can't build a docker image, but I don't know how I can change that. I cannot change the configuration of my runner, because I'm using shared runners. I also tried to put some tags to my second stages, in hope that a more suitable runner would have to take care of my job, but I'm still getting this error.
Thank you for your help.
I believe you need to set DOCKER_HOST to connect to the DinD running in another container:
docker-build:
image: docker:stable
services:
- docker:dind
stage: package
script:
- export DOCKER_HOST=tcp://docker:2375/
- docker build -t registry.gitlab.com/my-project .
- docker push registry.gitlab.com/my-project
If your shared runner executor is of type docker you may try this setup :
stages:
- build
- package
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
- docker info
cache:
paths:
- .gradle/wrapper
- .gradle/caches
build:
image: java:8-jdk
stage: build
script:
- ./gradlew build
artifacts:
paths:
- build/libs/*.jar
expire_in: 1 week
only:
- master
docker-build:
stage: package
script:
docker build -t registry.gitlab.com/my-project .
docker push registry.gitlab.com/my-project
after_script:
- echo "End CI"
Even we have faced the same problem in our org. We found that there is a long standing issue with the docker in docker area for gitlab which can be tracked in these issues #3612, #2408 and #2890 as well.
We have found that in our case using docker binding was much suitable for our usecase than the docker-in-docker one. so, we used the solution in their official page.
I know this has been already answered but this may help some one who have a similar usecase :)

GitLab CI runner can't connect to unix:///var/run/docker.sock in kubernetes

GitLab's running in kubernetes cluster. Runner can't build docker image with build artifacts. I've already tried several approaches to fix this, but no luck. Here are some configs snippets:
.gitlab-ci.yml
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
stages:
- build
- package
- deploy
maven-build:
image: maven:3-jdk-8
stage: build
script: "mvn package -B --settings settings.xml"
artifacts:
paths:
- target/*.jar
docker-build:
stage: package
script:
- docker build -t gitlab.my.com/group/app .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
- docker push gitlab.my.com/group/app
config.toml
concurrent = 1
check_interval = 0
[[runners]]
name = "app"
url = "https://gitlab.my.com/ci"
token = "xxxxxxxx"
executor = "kubernetes"
[runners.kubernetes]
privileged = true
disable_cache = true
Package stage log:
running with gitlab-ci-multi-runner 1.11.1 (a67a225)
on app runner (6265c5)
Using Kubernetes namespace: default
Using Kubernetes executor with image docker:latest ...
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Running on runner-6265c5-project-4-concurrent-0h9lg9 via gitlab-runner-3748496643-k31tf...
Cloning repository...
Cloning into '/group/app'...
Checking out 10d5a680 as master...
Skipping Git submodules setup
Downloading artifacts for maven-build (61)...
Downloading artifacts from coordinator... ok id=61 responseStatus=200 OK token=ciihgfd3W
$ docker build -t gitlab.my.com/group/app .
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1
What am I doing wrong?
Don't need to use this:
DOCKER_DRIVER: overlay
cause it seems like OVERLAY isn't supported, so svc-0 container is unable to start with it:
$ kubectl logs -f `kubectl get pod |awk '/^runner/{print $1}'` -c svc-0
time="2017-03-20T11:19:01.954769661Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2017-03-20T11:19:01.955720778Z" level=info msg="libcontainerd: new containerd process, pid: 20"
time="2017-03-20T11:19:02.958659668Z" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."
Also, add export DOCKER_HOST="tcp://localhost:2375" to the docker-build:
docker-build:
stage: package
script:
- export DOCKER_HOST="tcp://localhost:2375"
- docker build -t gitlab.my.com/group/app .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
- docker push gitlab.my.com/group/app
When using Kubernetes, you have to adjust your Build image to connect with the Docker engine.
Add to your build image:
DOCKER_HOST=tcp://localhost:2375
Quote from the docs:
Running the docker:dind also known as the docker-in-docker image is also
possible but sadly needs the containers to be run in privileged mode.
If you're willing to take that risk other problems will arise that might not
seem as straight forward at first glance. Because the docker daemon is started
as a service usually in your .gitlab-ci.yaml it will be run as a separate
container in your pod. Basically containers in pods only share volumes assigned
to them and an IP address by wich they can reach each other using localhost.
/var/run/docker.sock is not shared by the docker:dind container and the docker
binary tries to use it by default. To overwrite this and make the client use tcp
to contact the docker daemon in the other container be sure to include
DOCKER_HOST=tcp://localhost:2375 in your environment variables of the build container.
Gitlab-CI on Kubernetes
based on #Yarik 's comment what worked for me was
- export DOCKER_HOST=$DOCKER_PORT
no other answers worked.
I had the same problem, and I could not get the above workarounds to work for me (I did not try the volumes trick mentioned by #fkpwolf).
Now GitLab has an alternative solution by using Kaniko, which did work for me:
The .gitlab-ci.yaml could then be something like this, in that case:
stages:
- build
- package
- deploy
maven-build:
image: maven:3-jdk-8
stage: build
script: "mvn package -B --settings settings.xml"
artifacts:
paths:
- target/*.jar
docker-kaniko-build:
stage: package
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- echo "{\"auths\":{\"gitlab.my.com\":{\"username\":\"gitlab-ci-token\",\"password\":\"$CI_BUILD_TOKEN\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination gitlab.my.com/group/app
From the GitLab docs it is mentioned that:
kaniko solves two problems with using the docker-in-docker build method:
Docker-in-docker requires privileged mode in order to function, which is a significant security concern.
Docker-in-docker generally incurs a performance penalty and can be quite slow.
See: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html

Resources