I am trying to set up a gitlab runner to use gitlab-ci instead of my Jenkins.
I set up a docker container with linked docker.sock
docker run -d --name gitlab-runner --restart always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /config/file/on/host:/etc/gitlab-runner \
gitlab/gitlab-runner:latest
After the container run I registered a new runner to gitlab server that ends in following configuration
concurrent = 1
check_interval = 0
[[runners]]
name = "lianli"
url = "<https://gitlab_server.de"
token = "<secret>"
executor = "docker"
[runners.docker]
tls_verify = false
image = "debian:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
So now everything is connect. But when the pipeline is running it ends in an access error:
remote: Git access over HTTP is not allowed
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#gitlab_server.de/group/project.git/': The requested URL returned error: 403
ERROR: Job failed: exit code 1
My .gitlab-ci.yml looks like:
stages:
- test
variables:
NGINX: nginx:stable-alpine
before_script:
- docker info
test:
stage: test
script:
- docker build -t nginx_test .
I do not understand, why it could not access?
Note: The runner is Version 9.3.0 and gitlab is version 9.3.2
Is your GitLab instance configured to accept http requests to repository?
Are you a member of the project?
Related
I am using testcontainers to start a Postgresql DB for my junit test. Locally everything works fine. This is my small test project:
https://gitlab.com/janning/tpj-testcontainer
I can run the test inside my IDE and on CLI with ./gradlew test
Now I want to run it in my Gitlab-CI Pipeline but without docker-in-docker (dind). So I need to mount the docker socket, which is documented here: https://www.testcontainers.org/supported_docker_environment/continuous_integration/gitlab_ci/
so I configred my gitlab-runner like this:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "kt103-tpj"
url = "https://gitlab.com"
token = "XXXXXXXXXXXXXXXXX"
executor = "docker"
[runners.docker]
tls_verify = false
image = "registry.gitlab.com/janning/tpj-testcontainer/debian:latest"
privileged = false
disable_entrypoint_overwrite = false
extra_hosts = ["host.docker.internal:host-gateway"]
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
First my gitlab-ci.yml builds a Dockerimage as I need openjdk-17 and some docker commands.
variables:
TESTCONTAINERS_HOST_OVERRIDE: "host.docker.internal"
stages:
- dockerimage
- test
dockerimage:
image: docker:latest
stage: dockerimage
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- tag=":latest"
- docker build --pull -t "$CI_REGISTRY_IMAGE/debian${tag}" .
- docker push "$CI_REGISTRY_IMAGE/debian${tag}"
test:
stage: test
image: $CI_REGISTRY_IMAGE/debian:latest
script:
- echo $DOCKER_HOST
- echo $DOCKER_TLS_VERIFY
- echo $DOCKER_CERT_PATH
- ./gradlew test
artifacts:
when: always
reports:
junit: build/test-results/test/**/TEST-*.xml
The job "test" failed (complete log here)
Task :test HalloTest > test() FAILED
java.lang.IllegalStateException at RyukResourceReaper.java:129 1 test completed, 1 failed
Task :test FAILED
If i dig deeper into the junit result I can see the following stacktrace:
java.lang.IllegalStateException: Could not connect to Ryuk at host.docker.internal:49158
at org.testcontainers.utility.RyukResourceReaper.maybeStart(RyukResourceReaper.java:129)
at org.testcontainers.utility.RyukResourceReaper.init(RyukResourceReaper.java:42)
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:203)
at ...
I guess it has something to do with the docker image, which I build. But I am stuck as I don't know what the problem is. I don't how to solve or debug this situation.
The ryuk container is started in my host which is running gitlab-runner. But it can't connect:
$ docker container logs -f ba62a173aafd
2022/06/24 09:42:51 Pinging Docker...
2022/06/24 09:42:51 Docker daemon is available!
2022/06/24 09:42:51 Starting on port 8080...
2022/06/24 09:42:51 Started!
panic: Timed out waiting for the first connection
goroutine 1 [running]:
main.main()
/go/src/github.com/testcontainers/moby-ryuk/main.go:50 +0x449
The problem was just a firewall problem blocking traffic between the docker containers. As I could reach any docker container from external, it took my quite a while until I realized it.
I have a local Gitlab setup and trying to build a pipeline that runs a SAST scan using MobSF. Upon trying to pull the image of MobSF in order to run it I get the following error:
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
The error comes up on any script line referencing a Docker command.
The whole output of the pipeline is:
Running with gitlab-runner 14.0.0 (3b6f852e)
on pipeline 5qvFbM4s
Preparing the "docker" executor 00:04
Preparing environment 00:01
Running on runner-5qvfbm4s-project-2-concurrent-0 via TheOneWhoKnocks...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/root/sast-dast-security-testing/.git/
Checking out e71038e1 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:25a1e57c774167d28c44d88fa296f3e1122c6d79e99b98653c899b170393bbd6 for docker:18.09.7-dind with digest docker#sha256:a490c83561c1cef49b6fe12aba2c31f908391ec3efe4eb173225809c981e50c3 ...
$ export DOCKER_HOST=tcp://docker:2375
$ docker pull opensecurity/mobile-security-framework-mobsf
Using default tag: latest
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
ERROR: Job failed: exit code 1
This is my .gitlab-ci.yaml:
stages:
- build
- mobsf
build:
image: docker:18.09.7-dind
stage: build
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
script:
- docker pull opensecurity/mobile-security-framework-mobsf
- docker run -i --env-file ./env.list -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest
mobsf:
image: owasp/glue:raw-latest
stage: mobsf
script:
- ./scan.sh
- docker run -it -v $(pwd):/app owasp/glue:raw-latest ruby bin/glue -t Dynamic -T /app/report.json --mapping-file mobsf --finding-file-path /app/android.json -z 2
And this is my runner's config.toml:
[[runners]]
name = "pipeline"
url = "http://192.168.179.129/"
token = "XXXXX"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Any help would be appreciated!
It's fairly obvious that Google's public DNS servers won't resolve your local DNS requests. "docker"
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
Try this answer, i was facing similar one when registering local gitlab-runner to local domain name (gitlab.local).
Docker cannot resolve dns on private network
I have done a setup of my own Gitlab(-p 7022:22, 7080:9080), Gitlab Runner(-p 7093:8093), Sonatype Nexus3(Maven, Docker, Helm)(-p 10081:8081, 10082:10082, 10083:10083, 10084:10084). All are running as docker images of their own. And till docker build everything runs great(below code snippets doesn't have docker build related code). Problem is, I want to upload the final docker image to my Nexus3 Docker Registry which I am unable to do so.
My Gitlab Runner config is below
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "testing dind runner"
url = "http://192.168.0.250:7080/" ----> Gitlab git repo external url
token = "SOME TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
My Docker registry in Nexus3 is hosted as below
Web UI = http://192.168.0.250:10081/ (container internal port is 8081 and exposed to host on 10081)
Docker(Group) = 10084 (exposed as the same port through docker)
Docker(Hosted) = 10082 (exposed as the same port through docker)
Docker(Proxy) = 10083 (exposed as the same port through docker)
My project CI config is below
image: docker:19.03.12
services:
- name: docker:19.03.12-dind
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
before_script:
- docker info
- docker login -u nx-uploader -p 1234 192.168.0.250:10082
stages:
- test docker reg
test-docker:
stage: test docker reg
script:
- docker images
- docker search httpd
I am constantly getting errors as
$ docker login -u nx-uploader -p 1234 192.168.0.250:10082
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post http://docker:2376/v1.40/auth: dial tcp: lookup docker on 192.168.0.1:53: no such host
ERROR: Job failed: exit code 1
Need help/pointers to fix this.
Thanks in advance.
I tried to configure a gitlab-runner.
This is the tutorial I used: https://docs.gitlab.com/runner/install/docker.html
The steps were no problem, but when I actually ran the Runner it stops with the error below. The only online resource referencing this problem is this one: https://gitlab.com/gitlab-org/gitlab-runner/issues/1762
But I can't see a solution in there.
The Gitlab Version is: 10.4.3
Output of the Runner:
Running with gitlab-runner 10.7.2 (b5e03c94)
on Runner for .NET Core 2 0945b842
Using Docker executor with image gitlab/gitlab-runner:latest ...
Pulling docker image gitlab/gitlab-runner:latest ...
Using docker image sha256:abe39c16d28c87fefad753d02a89f8764a7577439a4851ec1f7d5228df69e48e for gitlab/gitlab-runner:latest ...
Running on runner-0945b842-project-66-concurrent-0 via 32359ca86797...
Fetching changes...
HEAD is now at b180fbf -
From https://<redacted>
b180fbf..3acaee5 <redacted> -> origin/<redacted>
Checking out 3acaee5f as <redacted>...
Skipping Git submodules setup
FATAL: Command sh not found.
FATAL: Command sh not found.
ERROR: Job failed: exit code 1
The .gitlab-ci.yml:
variables:
GIT_SSL_NO_VERIFY: "1"
before_script:
- docker login <redacted>
runnertest:
tags:
- dotnet
script:
- echo 'test'
The config.toml:
concurrent = 1
check_interval = 0
[[runners]]
name = "Runner for .NET Core 2"
url = "<redacted>"
token = "0945b84283ca87d541418edb23bc8c"
executor = "docker"
tls_verify = false
[runners.docker]
environment = ['GIT_SSL_NO_VERIFY=true']
image = "gitlab/gitlab-runner:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
runner had been started docker container, the config.toml is:
concurrent = 1
check_interval = 0
[[runners]]
name = "spring-boot-scripts"
url = "http://xxxxx.com/ci"
token = "xxxxxx"
executor = "docker"
builds_dir = "/tmp/builds"
[runners.docker]
tls_verify = false
image = "spring-boot-demo:ci"
privileged = false
disable_cache = true
volumes = ["/cache"]
[runners.cache]
Insecure = false
and .gitlab-ci.yml is:
image: spring-boot-demo:ci
stages:
- build
before_scipts:
- mkdir /tmp/before_scripts
- echo "============before_scripts========="
job1:
stage: build
script:
- sh /home/admin/spring-boot-demo-application/bin/entrypoint.sh
after_scipts:
- mkdir /tmp/after_scripts
- echo "============after_scripts========="
gitlab's output like this,and building task cannot been stopped until cancelled:
Running with gitlab-ci-multi-runner 1.4.2 (bcc1794)
Using Docker executor with image spring-boot-demo:ci ...
Pulling docker image spring-boot-demo:ci ...
WARNING: Cannot pull the latest version of image spring-boot-demo:ci : Error: image library/spring-boot-demo:ci not found
WARNING: Locally found image will be used instead.
Running on runner-278e2660-project-114610-concurrent-0 via 6ca6af37d681...
Cloning repository...
Cloning into '/tmp/builds/spring-boot/startup-scripts'...
Checking out b58711bc as debug...
Now questions is:
if the before_scripts/script/after_scripts would been exec in container,why i can not find the directory /tmp/before_scripts,/tmp/after_scripts and jobs's script not work in container
You can't find the folders because the jobs are not running. You have misspelled "script" in both the before_script and after_script jobs. Also, note that the correct job is not plural and does not have the 's' at the end.
GitLab docs