docker build in gitlab can't find packages to build - docker

I have a local gitlab runner that is behind a proxy. When building docker images, it fails to access the repos.
Errors:
Err:1 http://deb.debian.org/debian buster InRelease
Could not connect to deb.debian.org:80 (199.232.98.132). - connect (111: Connection refused)
Relevant part of CI/CD
build_image:
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
HTTPS_PROXY: http://proxy.example.com:3128
HTTP_PROXY: http://proxy.example.com:3128
NO_PROXY: docker
stage: test
before_script:
- docker info
- docker login nexus-docker.example.com:5000 -u $NEXUS_USER -p $NEXUS_PASS
script:
- docker build -t nexus-docker.example.com:5000/occ/groupsyncer:latest .
except:
- master
gitlab runner config:
[[runners]]
name = "docker-runner"
url = "https://gitlab.example.com/"
token = "REDACTED"
executor = "docker"
cache_dir="/cache"
clone_url = "https://gitlab.example.com/"
environment = ["https_proxy=http://squidproxy.example.com:3128", "http_proxy=http://squidproxy.example.com:3128", "HTTPS_PROXY=squidproxy.example.com:3128", "HTTP_PROXY=squidproxy.example.com:3128", "no_proxy=gitlab.example.com", "NO_PROXY=gitlab.example.com,docker,thealias"]
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
volumes = ["/var/run/docker.sock:/var/run/docker.sock","/opt/gitlab-runner/cache:/cache:rw"]
tls_verify = false
image = "docker:20.10.16"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
shm_size = 0
/etc/systemd/system/docker.service.d/http-proxy.conf
Environment="HTTP_PROXY=http://squidproxy.example.com:3128" "HTTPS_PROXY=http://squidproxy.example.com:3128" "NO_PROXY=.example.com,.occdev.example.com" "http_proxy=http://squidproxy.example.com:3128" "https_proxy=http://squidproxy.example.com:3128"
Domain edited to example.com for posting

Your container build process does not inherit environment variables by default. You must either (1) configure your dockerfile with the proxy variables or (2) pass in these variables as part of the build process with build args.
For example, in your dockerfile, you might add the following ARG instructions to your dockerfile:
FROM foo:tag
ARG HTTP_PROXY=defaultvalue
ARG HTTPS_PROXY=defaultvalue
RUN apt update && apt install -y ...
# or whatever...
You can pass this variable through to the build process when calling docker build
docker build --build-arg HTTP_PROXY="${HTTP_PROXY}" \
--build-arg HTTPS_PROXY="${HTTPS_PROXY}" \
-t nexus-docker.example.com:5000/occ/groupsyncer:latest .

Related

Trouble connecting to Docker daemon in GitLab CI

I'm trying to create a fairly simple GitLab CI file to build out Docker images. Whenever I run the pipeline, I end up getting a Docker daemon connection issue. What can I do to properly build my image? Thanks!
GitLab CI:
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_HOST: tcp://docker:2375
iac-build:
stage: build
extends: .iac
rules:
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
when: always
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
when: never
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
script:
- docker build -t testfirstimage .
allow_failure: false
Error:
$ docker build -t testfirstimage .
failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused
Cleaning up project directory and file based variables
ERROR: Job failed: command terminated with exit code 1
First thing I would check is if you already have something running on that local host - I've literally tried running a server on a local host port for hours and kept having it refused, only to find out that I had forgotten to terminate my connection to that port.
If that isn't the issue, I had this issue before and had to run this command to get it to work:
concurrent = 1
check_interval = 0
[[runners]]
name = "#####"
url = "#####"
token = "#####"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = false
disable_cache = false
cache_dir = "cache"
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
[runners.cache]
Insecure = false
I spent forever trying to figure it out and couldn't get anything to work until I found out to add
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
I didn't figure it out by magic though - props to this issues page: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1986
Hopefully that works.
In order to get Docker-in-Docker working with GitLab CI, you will first need to decide if you want to use Docker-in-Docker with or without TLS. Then, change /etc/gitlab-runner/config.toml settings, and assign the DOCKER_TLS_CERTDIR in your .gitlab-ci.yml file. See the Docker-in-docker section of the GitLab docs.
Docker-in-docker with TLS:
# /etc/gitlab-runner/config.toml
[[runners]]
url = "https://gitlab.com/"
token = TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:20.10.16"
privileged = true
disable_cache = false
volumes = ["/certs/client", "/cache"]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
# .gitlab-ci.yml
image: docker:20.10.16
variables:
DOCKER_TLS_CERTDIR: "/certs"
services:
- docker:20.10.16-dind
before_script:
- docker info
# rest of .gitlab-ci.yml

Local gitlab pipeline error during connect

I have a local Gitlab setup and trying to build a pipeline that runs a SAST scan using MobSF. Upon trying to pull the image of MobSF in order to run it I get the following error:
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
The error comes up on any script line referencing a Docker command.
The whole output of the pipeline is:
Running with gitlab-runner 14.0.0 (3b6f852e)
on pipeline 5qvFbM4s
Preparing the "docker" executor 00:04
Preparing environment 00:01
Running on runner-5qvfbm4s-project-2-concurrent-0 via TheOneWhoKnocks...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/root/sast-dast-security-testing/.git/
Checking out e71038e1 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:25a1e57c774167d28c44d88fa296f3e1122c6d79e99b98653c899b170393bbd6 for docker:18.09.7-dind with digest docker#sha256:a490c83561c1cef49b6fe12aba2c31f908391ec3efe4eb173225809c981e50c3 ...
$ export DOCKER_HOST=tcp://docker:2375
$ docker pull opensecurity/mobile-security-framework-mobsf
Using default tag: latest
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
ERROR: Job failed: exit code 1
This is my .gitlab-ci.yaml:
stages:
- build
- mobsf
build:
image: docker:18.09.7-dind
stage: build
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
script:
- docker pull opensecurity/mobile-security-framework-mobsf
- docker run -i --env-file ./env.list -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest
mobsf:
image: owasp/glue:raw-latest
stage: mobsf
script:
- ./scan.sh
- docker run -it -v $(pwd):/app owasp/glue:raw-latest ruby bin/glue -t Dynamic -T /app/report.json --mapping-file mobsf --finding-file-path /app/android.json -z 2
And this is my runner's config.toml:
[[runners]]
name = "pipeline"
url = "http://192.168.179.129/"
token = "XXXXX"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Any help would be appreciated!
It's fairly obvious that Google's public DNS servers won't resolve your local DNS requests. "docker"
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
Try this answer, i was facing similar one when registering local gitlab-runner to local domain name (gitlab.local).
Docker cannot resolve dns on private network

DinD configuration for Gitlab CI with private Docker registry in Sonatype Nexus 3

I have done a setup of my own Gitlab(-p 7022:22, 7080:9080), Gitlab Runner(-p 7093:8093), Sonatype Nexus3(Maven, Docker, Helm)(-p 10081:8081, 10082:10082, 10083:10083, 10084:10084). All are running as docker images of their own. And till docker build everything runs great(below code snippets doesn't have docker build related code). Problem is, I want to upload the final docker image to my Nexus3 Docker Registry which I am unable to do so.
My Gitlab Runner config is below
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "testing dind runner"
url = "http://192.168.0.250:7080/" ----> Gitlab git repo external url
token = "SOME TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
My Docker registry in Nexus3 is hosted as below
Web UI = http://192.168.0.250:10081/ (container internal port is 8081 and exposed to host on 10081)
Docker(Group) = 10084 (exposed as the same port through docker)
Docker(Hosted) = 10082 (exposed as the same port through docker)
Docker(Proxy) = 10083 (exposed as the same port through docker)
My project CI config is below
image: docker:19.03.12
services:
- name: docker:19.03.12-dind
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
before_script:
- docker info
- docker login -u nx-uploader -p 1234 192.168.0.250:10082
stages:
- test docker reg
test-docker:
stage: test docker reg
script:
- docker images
- docker search httpd
I am constantly getting errors as
$ docker login -u nx-uploader -p 1234 192.168.0.250:10082
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post http://docker:2376/v1.40/auth: dial tcp: lookup docker on 192.168.0.1:53: no such host
ERROR: Job failed: exit code 1
Need help/pointers to fix this.
Thanks in advance.

Gitlab CI run code quality not generate .json file

I followed the official guid deploied a pipline to analyse my code quality. Although the pipline passed and the job successed, the json file didn't generated by docker. And these notis displaied:
WARNING: A new version (v0.83.0) is available. Upgrade instructions
are available at: https://github.com/codeclimate/codeclimate#packages
Uploading artifacts... WARNING: gl-code-quality-report.json: no
matching files ERROR: No files to upload
Job succeeded
And this is my /etc/gitlab-runner/config.toml file:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "code_quality"
url = "http://192.168.0.2/"
token = "pyKK8v9C6Bnay8yMmfph"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
cache_dir = "cache"
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
My Gitlab version is 11.7
And this is my .gitlab-ci.yml file:
code_quality:
tags:
- quality
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
before_script:
- ip -4 a
allow_failure: true
services:
- name: docker:stable-dind
command: ["--insecure-registry=192.168.0.1:5000"]
script:
- ip -4 a
- docker pull 192.168.0.1:5000/codeclimate:0.72.0
- docker tag 192.168.0.1:5000/codeclimate:0.72.0 codeclimate/codeclimate:0.72.0
- docker pull 192.168.0.1:5000/codeclimate-coffeelint:latest
- docker tag 192.168.0.1:5000/codeclimate-coffeelint:latest codeclimate/codeclimate-coffeelint:latest
- docker pull 192.168.0.1:5000/codeclimate-csslint:latest
- docker tag 192.168.0.1:5000/codeclimate-csslint:latest codeclimate/codeclimate-csslint:latest
- docker pull 192.168.0.1:5000/codeclimate-eslint:latest
- docker tag 192.168.0.1:5000/codeclimate-eslint:latest codeclimate/codeclimate-eslint:latest
- docker pull 192.168.0.1:5000/codeclimate-rubocop:latest
- docker tag 192.168.0.1:5000/codeclimate-rubocop:latest codeclimate/codeclimate-rubocop:latest
- docker pull 192.168.0.1:5000/codeclimate-fixme:latest
- docker tag 192.168.0.1:5000/codeclimate-fixme:latest codeclimate/codeclimate-fixme:latest
- docker pull 192.168.0.1:5000/codeclimate-structure:latest
- docker tag 192.168.0.1:5000/codeclimate-structure:latest codeclimate/codeclimate-structure:latest
- docker pull 192.168.0.1:5000/codeclimate-duplication:latest
- docker tag 192.168.0.1:5000/codeclimate-duplication:latest codeclimate/codeclimate-duplication:latest
- export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
- docker images
- docker run
--env SOURCE_CODE="$PWD"
--env CODECLIMATE_VERSION=0.72.0
--env TIMEOUT_SECONDS=900
--volume "$PWD":/code
--volume /var/run/docker.sock:/var/run/docker.sock
192.168.0.1:5000/codequality:latest /code
artifacts:
reports:
codequality: gl-code-quality-report.json
Due to I want increase the speed of pulling image, so I deployed a docker registry in localhost.
Can you give me some advice?
Thank you all!
ANyways, months later, if you are using Gitlab CE ths won't work, is a feature for EE only.

git runner is unable to access

I am trying to set up a gitlab runner to use gitlab-ci instead of my Jenkins.
I set up a docker container with linked docker.sock
docker run -d --name gitlab-runner --restart always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /config/file/on/host:/etc/gitlab-runner \
gitlab/gitlab-runner:latest
After the container run I registered a new runner to gitlab server that ends in following configuration
concurrent = 1
check_interval = 0
[[runners]]
name = "lianli"
url = "<https://gitlab_server.de"
token = "<secret>"
executor = "docker"
[runners.docker]
tls_verify = false
image = "debian:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
So now everything is connect. But when the pipeline is running it ends in an access error:
remote: Git access over HTTP is not allowed
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#gitlab_server.de/group/project.git/': The requested URL returned error: 403
ERROR: Job failed: exit code 1
My .gitlab-ci.yml looks like:
stages:
- test
variables:
NGINX: nginx:stable-alpine
before_script:
- docker info
test:
stage: test
script:
- docker build -t nginx_test .
I do not understand, why it could not access?
Note: The runner is Version 9.3.0 and gitlab is version 9.3.2
Is your GitLab instance configured to accept http requests to repository?
Are you a member of the project?

Resources