Gitlab CI trying to run powershell when not asked to - docker

I am trying to build web application with gitlab-CI.
I created runner with this configuration:
name = "REDACTED"
url = "REDACTED"
token = REDACTED
executor = "docker-windows"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "mcr.microsoft.com/powershell"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["c:\\cache"]
shm_size = 0
Then my .gitlab-ci.yml looks like this
image: microsoft/dotnet:latest
stages:
- build
- test
before_script:
- "dotnet restore"
node_build:
stage: build
only:
- master
script:
- "echo Stage - Build started"
- "cd ./WebApplication"
- dir
- dotnet build
node_test:
stage: test
only:
- master
script:
- "echo Stage - Test started"
- "cd ./WebApplication"
- dir
- dotnet build
When the pipeline is ran, output looks like this
Running with gitlab-runner 13.11.0 (7f7a4bb0)
on REDACTED REDACTED
Preparing the "docker-windows" executor
Using Docker executor with image microsoft/dotnet:latest ...
Pulling docker image microsoft/dotnet:latest ...
Using docker image sha256:34f6f2295334d34567c67059f7c28836c79e014d0c4fadf54de3978798640003 for microsoft/dotnet:latest with digest microsoft/dotnet#sha256:61d86fc52893087df54b0579fcd9c33e144a4b3d34c543a94e6a6b376c74285d ...
Preparing environment
Running on REDACTED via
REDACTED ...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in C:/builds/REDACTED /c-sharp-ci-test/.git/
Checking out bbb22919 as master...
git-lfs/2.11.0 (GitHub; windows amd64; go 1.14.2; git 48b28d97)
Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:34f6f2295334d34567c67059f7c28836c79e014d0c4fadf54de3978798640003 for microsoft/dotnet:latest with digest microsoft/dotnet#sha256:61d86fc52893087df54b0579fcd9c33e144a4b3d34c543a94e6a6b376c74285d ...
Cleaning up file based variables
ERROR: Job failed (system failure): Error response from daemon: container e144f05bdd00b4e744554345666afbc008ee2437c7d56bf4a98fbd949a88b1b2 encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2)
[Event Detail: Provider: 00000000-0000-0000-0000-000000000000]
[Event Detail: Provider: 00000000-0000-0000-0000-000000000000]
[Event Detail: onecore\vm\compute\management\orchestration\vmhostedcontainer\processmanagement.cpp(173)\vmcomputeagent.exe!00007FF7D970B1D7: (caller: 00007FF7D96BE70B) Exception(6) tid(37c) 80070002 The system cannot find the file specified.
CallContext:[\Bridge_ProcessMessage\VmHostedContainer_ExecuteProcess]
Provider: 00000000-0000-0000-0000-000000000000] extra info: {"CommandLine":"powershell -NoProfile -NoLogo -InputFormat text -OutputFormat text -NonInteractive -ExecutionPolicy Bypass -Command -","User":"ContainerUser","WorkingDirectory":"C:\\","Environment"
When I look into log, it says it tried to run step_script stage of the job, which I never specified and it tries to run powershell. Why is that happening and how can I get rid of it ? I supose dotnet:latest does not have powershell in it as it is not needed for building.

First, it is always best to use a fixed tag instead of the shifting "latest": from one build to the next, "latest" can reference a new image version.
Second try a specific dotnet image like mcr.microsoft.com/dotnet/core/sdk:3. instead of microsoft/dotnet:xxx: note though they are likely to use Powershell, as seen in their Dockerfile
Try one of the .Dotnet Samples outside of GitLab to see if you can make it work manually, then include it in your gitlab-ci.yml.
Note: from gitlab-org/gitlab-runner issue 26418, step_script would be equivalent to build_script.

Related

gitlab - ci for composer package

i setup a dev-server in my homeoffice and installed gitlab via docker-compose. so far everything works fine, i can login, push commits and so on.
Now i wanted to setup a CI Pipeline to build composer packages when new tags are pushed. So i clicked the CI/CD Button and added the .gitlab-ci.yml file from the composer template. But the pipeline was only pending. So i figured i might need to register a runner first.
I installed gitlab-runner (via apt) on the same machine that runs the gitlab via docker and registered the runner with the domain and key given by gitlab (in the add runners page). I selected docker as executor, gave it a name and left everything else at its default value.
The runner is registered properly in gitlab and the ci pipeline is now working but it always fails.
The only output i have is:
Running with gitlab-runner 11.2.0 (11.2.0)
on **************
Using Docker executor with image curlimages/curl:latest ...
Pulling docker image gitlab-runner-helper:11.2.0 ...
The contents of the gitlab-ci file are:
# This file is a template, and might need editing before it works on your project.
# Publishes a tag/branch to Composer Packages of the current project
publish:
image: curlimages/curl:latest
stage: build
variables:
URL: "$CI_SERVER_PROTOCOL://$CI_SERVER_HOST:$CI_SERVER_PORT/api/v4/projects/$CI_PROJECT_ID/packages/composer?job_token=$CI_JOB_TOKEN"
script:
- version=$([[ -z "$CI_COMMIT_TAG" ]] && echo "branch=$CI_COMMIT_REF_NAME" || echo "tag=$CI_COMMIT_TAG")
- insecure=$([ "$CI_SERVER_PROTOCOL" = "http" ] && echo "--insecure" || echo "")
- response=$(curl -s -w "\n%{http_code}" $insecure --data $version $URL)
- code=$(echo "$response" | tail -n 1)
- body=$(echo "$response" | head -n 1)
# Output state information
- if [ $code -eq 201 ]; then
echo "Package created - Code $code - $body";
else
echo "Could not create package - Code $code - $body";
exit 1;
fi
Because i did not make any changes to the template file i suspect the gitlab-runner setup to need some configuration in order to work, maybe a group-assignment or something like that.
When running systemctl status gitlab-runner i can see:
Failed to create container volume for /builds/{group} Error response from daemon: pull access denied for gitlab-runner-helper, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (executor_docker.go:166:3s)" job=15 project=34 runner=******
So i went to the runners section in gitlab and enabled the runner fot the specific project. So i could avoid the error above but the pipeline still breaks.
The output in gitlab is still the same but the gitlab-runner log is different:
Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n
Sadly - i am not getting any furhter from here
Everytime i press the retry button for the pipeline i get the following syslog entries:
Checking for jobs... received" job=19 repo_url="correct-url-for-repo" runner=******
This message appears twice
Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n
Ignoring extra error returned from registry: unauthorized: authentication required
Failed to create container volume for /builds/{group} Error response from daemon: pull access denied for gitlab-runner-helper, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (executor_docker.go:166:3s)" job=19 project=34 runner=******
Job failed: Error response from daemon: pull access denied for gitlab-runner-helper, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (executor_docker.go:166:3s)" job=19 project=34 runner=******
Both messages appear twice
so either the gitlab-runner is not allowed to pull docker images or it is not allowed to access my gitlab project but i cant figure out the problem.
When running gitlab-runner restart as root i see the following "error"
ERRO[0000] Docker executor: prebuilt image helpers will be loaded from /var/lib/gitlab-runner.
Can someone please help me :) ?
Select the correct Docker image for the runner. Depending where are you executing it, and probably also depending on your GitLab version. Also, manually try it before executing the pipeline:
docker pull gitlab/gitlab-runner-helper:x86_64-latest
To use the selected image, modify the runner's config file:
[[runners]]
(...)
executor = "docker"
[runners.docker]
(...)
helper_image = "gitlab/gitlab-runner-helper:tag"
The images gitlab-runner-helper, gitlab/gitlab-runner-helper:11.2.0 do not exist. It seems the debian package installable in ubuntu is broken somehow... So i figured i might need to install the latest gitlab-runner version
Here is what i did: (I am on Ubuntu 20.04)
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
cat <<EOF | sudo tee /etc/apt/preferences.d/pin-gitlab-runner.pref
Explanation: Prefer GitLab provided packages over the Debian native ones
Package: gitlab-runner
Pin: origin packages.gitlab.com
Pin-Priority: 1001
EOF
Source
So i was able to update gitlab-runner to the latest version.
But still no success, now the service won't start without any error message, systemctl only tells mit that the process exited.
the syslog told me:
chdir /var/lib/gitlab-runner: no such file or directory
opening /etc/init.d/gitlab-runner showed me that path as --working-directory parameter for the service.
So i created that directory and changed its ownership to gitlab-runner
This time i could run the ci pipeline!
Still got an error
fatal: unable to access 'http://{mylocaldomain}/isat/typo3-gdpr.git/': Could not resolve host: {mylocaldomain}
Okay - dns resolution not possible because i use a local domain.
As stated here you can pass an extra_host to the docker executor.
To do so, simply adjust the /etc/gitlab-runner/config.toml file and add the extra_hosts option:
concurrent = 1
check_interval = 0
[[runners]]
name = "lab"
url = "http://{localDomain}/"
token = "******"
executor = "docker"
[runners.docker]
tls_verify = false
image = "ruby:2.1"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
extra_hosts = ["{localDomain}:{ip}"]
[runners.cache]
Now i could sucessfully run the ci pipeline and my package is listed in the composer registry!

WARNING: Checking for jobs... failed in docker executer in gitlab-runner

I am trying to run gitlab-ci on a local running using docker executer
This is the config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
listen_address = "0.0.0.0:8093"
[[runners]]
url = "https://gitlab.com/<ACCOUNT>/my-static-website"
token = XXXXXX
executor = "docker"
builds_dir = ""
clone_url = "https://gitlab.com/<ACCOUNT>/my-static-website.git"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = true
disable_cache = false
volumes = ["/cache"]
[runners.cache]
Insecure = false
My .gitlab-ci.yml:
image: node
stages:
- build
- test
build website:
stage: build
script:
- npm install
- npm install -g gatsby-cli
- gatsby build
artifacts:
paths:
- ./public
tags:
- trials
test artifacts:
image: alpine
stage: test
script:
- grep -q "Gatsby" ./public/index.html
Here is the error I am getting:
Runtime platform arch=amd64 os=linux
pid=28815 revision=4c96e5ad version=12.9.0
Starting multi-runner from ./config.toml... builds=0
Running in system-mode.
Configuration loaded builds=0
listen_address not defined, metrics & debug endpoints disabled builds=0
Session server listening address=0.0.0.0:8093
builds=0
WARNING: Checking for jobs... failed runner=kYtFEV-i
status=404 Not Found
WARNING: Checking for jobs... failed runner=kYtFEV-i
status=404 Not Found
WARNING: Checking for jobs... failed runner=kYtFEV-i
status=404 Not Found
I am using gitlab-runner version 12.9 and gitlab server: 12.10.0-pre
I have my runner on the server as follows:
I am running the command: gitlab-runner run -c ./config.toml
What did I miss here?
Your runner is not able to check for jobs. Can you double check the endpoint URL?
If your repository is on gitlab.com, you should be using the endpoint https://gitlab.com/
In your GitLab Web UI, go to Settings -> CI/CD -> Runners -> Set up a specific Runner manually
You'll see the endpoint URL and the token you'll need to register your runner.
This is covered in my GitLab CI tutorial at https://gitpitch.com/atsaloli/cicd/master?grs=gitlab#/41 (it takes a few seconds to load)
Let me know if that helps?

Gitlab - Job with "docker in docker" service results in "Cannot connect to the Docker daemon"

I've added:
services:
- docker:dind
to my .gitlab-ci.yaml file, however it still does not seem to have the docker daemon running. Is additional configuration required?
I'm using the Gitlab Runner on Kubernetes - installed directly via the Gitlab web interface / admin.
image: docker:latest
services:
- docker:dind
stages:
- build
- deploy
build-web:
stage: build
script:
- docker info
only:
- master
deploy-web:
stage: deploy
script:
- docker info
- exit 1
only:
- master
Build output:
Running with gitlab-runner 12.2.0 (a987417a)
on runner-gitlab-runner-857c466884-kzql9 BB7a6A8L
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image docker:latest ...
Waiting for pod gitlab-managed-apps/runner-bb7a6a8l-project-1-concurrent-0prm8w to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-bb7a6a8l-project-1-concurrent-0prm8w to be running, status is Pending
Running on runner-bb7a6a8l-project-1-concurrent-0prm8w via runner-gitlab-runner-857c466884-kzql9...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/root/cinema/.git/
Created fresh repository.
From http://mygitlab.ddns.net/root/cinema
* [new branch] master -> origin/master
Checking out e138a25e as master...
Skipping Git submodules setup
$ docker info
Client:
Debug Mode: false
Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
ERROR: Job failed: command terminated with exit code 1
I see the following config in the gitlab runner pod:
bash-4.4$ cat .gitlab-runner/config.toml
listen_address = "[::]:9252"
concurrent = 4
check_interval = 3
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "runner-gitlab-runner-857c466884-kzql9"
request_concurrency = 1
url = "http://mygitlab.ddns.net/"
token = "BB7a6A8LRvZ4Y-9KsLvj"
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "ubuntu:16.04"
namespace = "gitlab-managed-apps"
namespace_overwrite_allowed = ""
privileged = true
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.pod_security_context]
[runners.kubernetes.volumes]
Warning: The below might not be 100% secure
Seems to be the issue mentioned here: https://gitlab.com/gitlab-org/gitlab-runner/issues/4501
A workaround is provided here: https://gitlab.com/gitlab-org/gitlab-runner/issues/4501#note_194648457
Essentially set the following:
# .gitlab-ci.yml
variables:
DOCKER_TLS_CERTDIR: ""

Gitlab CI Runner: FATAL: Command sh not found

I tried to configure a gitlab-runner.
This is the tutorial I used: https://docs.gitlab.com/runner/install/docker.html
The steps were no problem, but when I actually ran the Runner it stops with the error below. The only online resource referencing this problem is this one: https://gitlab.com/gitlab-org/gitlab-runner/issues/1762
But I can't see a solution in there.
The Gitlab Version is: 10.4.3
Output of the Runner:
Running with gitlab-runner 10.7.2 (b5e03c94)
on Runner for .NET Core 2 0945b842
Using Docker executor with image gitlab/gitlab-runner:latest ...
Pulling docker image gitlab/gitlab-runner:latest ...
Using docker image sha256:abe39c16d28c87fefad753d02a89f8764a7577439a4851ec1f7d5228df69e48e for gitlab/gitlab-runner:latest ...
Running on runner-0945b842-project-66-concurrent-0 via 32359ca86797...
Fetching changes...
HEAD is now at b180fbf -
From https://<redacted>
b180fbf..3acaee5 <redacted> -> origin/<redacted>
Checking out 3acaee5f as <redacted>...
Skipping Git submodules setup
FATAL: Command sh not found.
FATAL: Command sh not found.
ERROR: Job failed: exit code 1
The .gitlab-ci.yml:
variables:
GIT_SSL_NO_VERIFY: "1"
before_script:
- docker login <redacted>
runnertest:
tags:
- dotnet
script:
- echo 'test'
The config.toml:
concurrent = 1
check_interval = 0
[[runners]]
name = "Runner for .NET Core 2"
url = "<redacted>"
token = "0945b84283ca87d541418edb23bc8c"
executor = "docker"
tls_verify = false
[runners.docker]
environment = ['GIT_SSL_NO_VERIFY=true']
image = "gitlab/gitlab-runner:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]

Gitlab-CI multi runner start docker container, scripts now work

runner had been started docker container, the config.toml is:
concurrent = 1
check_interval = 0
[[runners]]
name = "spring-boot-scripts"
url = "http://xxxxx.com/ci"
token = "xxxxxx"
executor = "docker"
builds_dir = "/tmp/builds"
[runners.docker]
tls_verify = false
image = "spring-boot-demo:ci"
privileged = false
disable_cache = true
volumes = ["/cache"]
[runners.cache]
Insecure = false
and .gitlab-ci.yml is:
image: spring-boot-demo:ci
stages:
- build
before_scipts:
- mkdir /tmp/before_scripts
- echo "============before_scripts========="
job1:
stage: build
script:
- sh /home/admin/spring-boot-demo-application/bin/entrypoint.sh
after_scipts:
- mkdir /tmp/after_scripts
- echo "============after_scripts========="
gitlab's output like this,and building task cannot been stopped until cancelled:
Running with gitlab-ci-multi-runner 1.4.2 (bcc1794)
Using Docker executor with image spring-boot-demo:ci ...
Pulling docker image spring-boot-demo:ci ...
WARNING: Cannot pull the latest version of image spring-boot-demo:ci : Error: image library/spring-boot-demo:ci not found
WARNING: Locally found image will be used instead.
Running on runner-278e2660-project-114610-concurrent-0 via 6ca6af37d681...
Cloning repository...
Cloning into '/tmp/builds/spring-boot/startup-scripts'...
Checking out b58711bc as debug...
Now questions is:
if the before_scripts/script/after_scripts would been exec in container,why i can not find the directory /tmp/before_scripts,/tmp/after_scripts and jobs's script not work in container
You can't find the folders because the jobs are not running. You have misspelled "script" in both the before_script and after_script jobs. Also, note that the correct job is not plural and does not have the 's' at the end.
GitLab docs

Resources