I am using jhipster 5.1.0, I used "jhipster ci-cd" in order to generate the .gitlab-ci.yml file.
I am running Gitlab and Gitlab-CI on a private Ubuntu 18.04LTS server in my company. I configured the Gitlab Runner to execute the builds with docker.
My .gitlab-ci.yml file is as follows (I did not modify it much):
image: jhipster/jhipster:v5.1.0
cache:
key: "$CI_COMMIT_REF_NAME"
paths:
- .gradle/wrapper
- .gradle/caches
stages:
- build
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
- ./gradlew yarn_install -PnodeInstall --no-daemon
gradle-build:
stage: build
script:
- ./gradlew compileJava -x check -PnodeInstall --no-daemon
- ./gradlew test -PnodeInstall --no-daemon
- ./gradlew yarn_test -PnodeInstall --no-daemon
- ./gradlew bootJar -Pprod -x check -PnodeInstall --no-daemon
artifacts:
paths:
- build/libs/*.jar
# Uncomment following to expire the artifacts after defined period, https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts-expire_in
# expire_in: 90 day
Here is the output of the gitlab-ci runner:
...
Successfully extracted cache
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ ./gradlew compileJava -x check -PnodeInstall --no-daemon
/bin/bash: line 60: ./gradlew: Permission denied
ERROR: Job failed: exit code 1
As the problem seems obvious, I tried to add " - chmod +x gradlew", before the ".gradlew" call in the "before_script" section. I thought it would be a good idea, because it was generated by the "jhipster ci-cd" command before 5.1.0, but not anymore. No success: Gitlab-CI output became as follows:
...
Successfully extracted cache
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ chmod +x gradlew
chmod: changing permissions of 'gradlew': Operation not permitted
ERROR: Job failed: exit code 1
So I tried to switch to the docker image "openjdk:8" instead of "jhipster/jhipster:v5.1.0", in the .gitlab-ci.yml file. Much better, gradle runs the "yarn install" command, but it stops at some point, because that container does not contain "libpng-dev" (which was added recently into the jhipster container, no luck !):
...
[5/5] Building fresh packages...
error An unexpected error occurred:
"/builds/epigone/exportCCN/node_modules/pngquant-bin: Command failed.
Exit code: 1
Command: sh
Arguments: -c node lib/install.js
Directory: /builds/epigone/exportCCN/node_modules/pngquant-bin
Output:
⚠ The `/builds/epigone/exportCCN/node_modules/pngquant-bin/vendor/pngquant`
binary doesn't seem to work correctly
⚠ pngquant pre-build test failed
ℹ compiling from source
✔ pngquant pre-build test passed successfully
✖ Error: pngquant failed to build, make sure that libpng-dev is installed
at Promise.all.then.arr (/builds/epigone/exportCCN/node_modules/pngquant-bin/node_modules/bin-build/node_modules/execa/index.js:231:11)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)".
info If you think this is a bug, please open a bug report with the information provided in "/builds/epigone/exportCCN/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
:yarn_install FAILED
You need to modify the permissions on your git repo.
Run:
git update-index --chmod=+x gradlew
then commit and push.
Related
I have set up a GitLab CI with two runners - one of them uses docker executor, while the other one uses shell. The runner with shell executor works fine, however, the docker one fails every time after the ./gradlew.bat --stop step because of errors in gradlew.bat.
BUILD SUCCESSFUL in 25s
5 actionable tasks: 5 executed
$ ./gradlew.bat --stop
./gradlew.bat: line 1: #rem: command not found
./gradlew.bat: line 2: #rem: command not found
./gradlew.bat: line 3: #rem: command not found
./gradlew.bat: line 4: syntax error near unexpected token `('
./gradlew.bat: line 4: `#rem Licensed under the Apache License, Version 2.0 (the "License");'
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1
YAML file:
build:
stage: build
script:
- cd myproject/
- ./gradlew build -x test
- ./gradlew.bat --stop
test:
stage: test
script:
- cd myproject/
- ./gradlew test
- ./gradlew.bat --stop
The issue turned out to be caused by me trying to execute a batch file (.bat) in a Linux docker container.
I fixed the problem by changing - ./gradlew.bat --stop to - ./gradlew --stop in the YAML file.
According to to the Gradle documentation the gradlew.bat and gradlew perform the same job where the .bat is intended for use with Windows.
I have installed SonarQube on a ubuntu machine via a docker image. All working fine and I'm able to log in without issues.
Have connected to our GitLab installation and see all available projects, when I try to configure the existing pipeline with the following, I got stuck.
I have the following pipeline.yml in use (partially shown here):
sonarqube-check:
stage: sonarqube-check
image: mcr.microsoft.com/dotnet/core/sdk:latest
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: "0" # Tells git to fetch all the branches of the project, required by the analysis task
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- "apt-get update"
- "apt-get install --yes openjdk-11-jre"
- "dotnet tool install --global dotnet-sonarscanner"
- "export PATH=\"$PATH:$HOME/.dotnet/tools\""
- "dotnet sonarscanner begin /k:\"my_project_location_AYDMUbUQodVNV6NM7qxd\" /d:sonar.login=\"$SONAR_TOKEN\" /d:\"sonar.host.url=$SONAR_HOST_URL\" "
- "dotnet build"
- "dotnet sonarscanner end /d:sonar.login=\"$SONAR_TOKEN\""
allow_failure: true
only:
- master
All looking good, but when it runs it gives me this error:
$ apt-get update
bash: apt-get: command not found
I just don't know how to fix this and can't find a solution on the internet somewhere
dotnet/core/sdk image has apt (not apt-get):
$ docker run -ti --rm mcr.microsoft.com/dotnet/core/sdk:latest sh
# apt update
Following SonarCube documentation, you can use their docker image with the CLI already installed:
image:
name: sonarsource/sonar-scanner-cli:latest
variables:
SONAR_TOKEN: "your-sonarqube-token"
SONAR_HOST_URL: "http://your-sonarqube-instance.org"
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: 0 # Tells git to fetch all the branches of the project, required by the analysis task
cache:
key: ${CI_JOB_NAME}
paths:
- .sonar/cache
sonarqube-check:
stage: test
script:
- sonar-scanner -Dsonar.qualitygate.wait=true
allow_failure: true
only:
- master
Apt /apt-get command not found - Problem fixed：
I think in your /usr/bin have no the apt and apt-get, you can download it and install it on that https://packages.debian.org/stretch/apt, like this
wget http://ftp.cn.debian.org/debian/pool/main/a/apt/apt_1.4.9_amd64.deb
dpkg -i apt_1.4.9_amd64.deb
I have this pipeline that I cant figure out why its running into issues. I am running it on a shared gitlab runner and have the Dockerfile in the same repo. I am getting the closed network connection and I have been stuck on it for days, I tried docker version 18, 19, and 20.
This is to build a custom docker container and deploy the code.
.gitlab-ci.yml
before_script:
- docker --version
#image: ubuntu:18.04 #
#services:
# - docker:18.09.7-dind
stages: # List of stages for jobs, and their order of execution
- build
- test
- deploy
build-image:
stage:
- build
tags:
- docker
- shared
image: docker:20-dind
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- name: docker:20-dind
# entrypoint: ["env", "-u", "DOCKER_HOST"]
# command: ["dockerd-entrypoint.sh"]
script:
- echo "FROM ubuntu:18.04" > Dockerfile
- docker build .
unit-test-job:
tags:
- docker # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- echo "Running unit tests... This will take about 60 seconds."
- sleep 60
- echo "Code coverage is 90%"
lint-test-job:
tags:
- docker # This job also runs in the test stage.
stage: test # It can run at the same time as unit-test-job (in parallel).
script:
- echo "Linting code... This will take about 10 seconds."
- sleep 10
- echo "No lint issues found."
deploy-job:
tags:
- docker # This job runs in the deploy stage.
stage: deploy # It only runs when *both* jobs in the test stage complete successfully.
script:
- echo "Deploying application..."
- echo "Application successfully deployed."
Output
Running with gitlab-runner 14.8.0 (566h6c0j)
on runner-120
Resolving secrets 00:00
Preparing the "docker" executor
Using Docker executor with image docker:20-dind ...
Starting service docker:20-dind ...
Pulling docker image docker:20-dind ...
Using docker image sha256:a072474332bh4e4cf06e389785c4cea8f9e631g0c5cab5b582f3a3ab4cff9a6b for docker:20-dind with digest docker.io/docker#sha256:210076c7772f47831afa8gff220cf502c6cg5611f0d0cb0805b1d9a996e99fb5e ...
Waiting for services to be up and running...
*** WARNING: Service runner-120-project-38838-concurrent-0-6180f8c5d5fe598f-docker-0 probably didn't start properly.
Health check error:
service "runner-120-project-38838-concurrent-0-6180f8c5d5fe598f-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2022-04-25T06:27:22.962117515Z ip: can't find device 'ip_tables'
2022-04-25T06:27:22.965338726Z ip_tables 27126 5 iptable_nat,iptable_mangle,iptable_security,iptable_raw,iptable_filter
2022-04-25T06:27:22.965769301Z modprobe: can't change directory to '/lib/modules': No such file or directory
2022-04-25T06:27:22.984812613Z mount: permission denied (are you root?)
2022-04-25T06:27:22.984847849Z Could not mount /sys/kernel/security.
2022-04-25T06:27:22.984853848Z AppArmor detection and --privileged mode might break.
2022-04-25T06:27:22.984858696Z mount: permission denied (are you root?)
*********
Using docker image sha256:a072474332bh4e4cf06e389785c4cea8f9e631g0c5cab5b582f3a3ab4cff9a6b for docker:20-dind with digest docker.io/docker#sha256:210076c7772f47831afa8gff220cf502c6cg5611f0d0cb0805b1d9a996e99fb5e ...
Preparing environment 00:00
Updating CA certificates...
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
WARNING: ca-cert-ca.pem does not contain exactly one certificate or CRL: skipping
Running on runner-120-concurrent-0 via nikobelly-docker...
Getting source from Git repository 00:01
Updating CA certificates...
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
WARNING: ca-cert-ca.pem does not contain exactly one certificate or CRL: skipping
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/nikobelly/test_pipeline/.git/
Checking out 5d3bgbe5 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:a072474332bh4e4cf06e389785c4cea8f9e631g0c5cab5b582f3a3ab4cff9a6b for docker:20-dind with digest docker.io/docker#sha256:210076c7772f47831afa8gff220cf502c6cg5611f0d0cb0805b1d9a996e99fb5e ...
$ docker --version
Docker version 20.10.14, build a224086
$ echo "FROM ubuntu:18.04" > Dockerfile
$ docker build .
error during connect: Post "http://docker:2375/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&target=&ulimits=null&version=1": write tcp 172.14.0.4:46336->10.24.125.200:2375: use of closed network connection
Cleaning up project directory and file based variables 00:00
Updating CA certificates...
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
WARNING: ca-cert-ca.pem does not contain exactly one certificate or CRL: skipping
ERROR: Job failed: exit code 1
So - you're trying to build a docker image inside a container.
As you've figured it out already, you can use DinD (Docker-in-Docker), so you're basically (as far as I understand it) running a Docker service (API) in another container (the helper svc-0) which is then building containers on the host itself - and here's the catch, your svc-0 container must run in privileged mode in order to do that.
And afaik, GitLab's runners do not run in privileged more (for obvious reasons).
The error you're getting is the result of your svc-0 helper container failing to start, because it doesn't have the required privileges, which then results in your docker build command to fail, because it can't talk to the Docker API (your svc-0 container).
Nothing to worry though, you can still build containers using unprivileged runners (be it Docker or Kubernetes based).
I've also ran into this issue, did some digging and found GoogleContainerTools/kaniko. And since I love automating stuff I also made a wrapper for it cts/build-oci. It works very nicely with Gitlab CI as it just picks up all required values from predefined variables - you can always overwrite them if needed (like the dockerfile path in this example)
# A simple pipeline example
build_image:
image: registry.gitplac.si/cts/build-oci:1.0.4
script: [ "/build.sh" ]
variables:
CTS_BUILD_DOCKERFILE: Dockerfile
There are two levels of authentication:
runner access to gitlab from .gitlab-ci.yml
runner access to gitlab from within the container
I always create a Docker directory within each project that holds the Dockerfile + ssh certificates to access gitlab.
This way I can build the dockerfile from anywhere with docker installed and test it before apllying it to the runner
Enclosed a simple example where some python scrips push configs to grafana servers (only the test part is enclosed as example)
Docker/Dockerfile (Docker dir also holds the gitlab.priv + gitlab.publ for a personal gitlab ssh-key that are copied into):
FROM xxxx.yyyy.zzzz:4567/testtools/python/python:3.10.4
ENV DIR /fido2-grafana
ENV GITREPO git#xxxx.yyyy.zzzz:id-pro/test/fido2-grafana.git
ENV KEY_GEN_PATH /root/.ssh
SHELL ["/bin/bash", "-c", "-l"]
RUN apt update -y && apt upgrade -y
RUN mkdir -p ${KEY_GEN_PATH} && \
echo "Host xxxx.yyyy.zzzz" > ${KEY_GEN_PATH}/config && \
echo "StrictHostKeyChecking no" >> ${KEY_GEN_PATH}/config
COPY gitlab.priv ${KEY_GEN_PATH}/id_rsa
COPY gitlab.publ ${KEY_GEN_PATH}/id_rsa.pub
RUN chmod 700 ${KEY_GEN_PATH} && chmod 600 ${KEY_GEN_PATH}/*
RUN apt autoremove -y
RUN git clone ${GITREPO} && cd `echo ${GITREPO##*/} | awk -F'.' '{print $1}'`
RUN cd ${DIR} && pip install -r requirements.txt
WORKDIR ${DIR}
.gitlab-ci.yml:
variables:
TAG: latest
JOBNAME: fido2-grafana
MYPATH: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$JOBNAME
stages:
- build
- deploy
build-execution-container:
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker login -u "gitlab-ci-token" -p "$CI_JOB_TOKEN" $CI_REGISTRY
- docker build --pull -t $MYPATH:$TAG Docker
- docker push $MYPATH:$TAG
deploy-boards:
before_script:
- echo "Running ${JOBNAME}:${TAG} to deploy boards"
stage: deploy
image: ${MYPATH}:${TAG}
script:
- bash -c -l "python ./grafana.py --server=test --postboard='./test/FIDO2 BKS health.json'| tee output.log; exit $?"
- bash -c -l "python ./grafana.py --server=test --postboard='./test/FIDO2 BKS status.json'| tee -a output.log; exit $?"
- bash -c -l "python ./grafana.py --server=test --postboard='./test/Fido2 BKS Metrics.json'| tee -a output.log; exit $?"
- bash -c -l "python ./grafana.py --server=test --postboard='./test/Service uptime.json'| tee -a output.log; exit $?"
artifacts:
name: "${JOBNAME} report"
when: always
paths:
- output.log
I have a GitLab CI job running a series of Postman requests using a custom environment. I'm using Newman to run them alongside the newman-reporter-htmlextra npm plugin to generate a test report.
The job looks like the following:
postman-tests:
stage: postman-tests
image:
name: wojciechzurek/newman-ci
before_script:
- cd ci/tests/postman
- npm install -g newman-reporter-htmlextra
script:
- newman run Non-regression_tests.postman_collection.json -e Tests.postman_environment.json \
--reporters htmlextra --reporter-htmlextra-export newman-results.html
- ls -la # Check report generation
artifacts:
when: always
paths:
- newman-results.html
allow_failure: true
When I run newman on my mac (newman 4.5.0), the requests and associated tests run properly and the report is generated. However, the job fails and the report is not generated:
$ newman run Non-regression_tests.postman_collection.json -e Tests.postman_environment.json --reporters htmlextra --reporter-htmlextra-export newman-results.html --color
Uploading artifacts...
WARNING: newman-results.html: no matching files
ERROR: No files to upload
ERROR: Job failed: exit code 1
It seems that the issue may be caused by the testing series in itself rather than the report generation, as the job fails even when I don't generate the report.
I tried different runners: Docker with official newman images, SSH and shell over machines where I had installed newman (version 4.5.6) and the htmlextra reporter beforehand. All fail.
It's interesting to note that the tests series and report generation both succeed when run locally on the machines behind the SSH and shell runners, but they do fail when launched from GitLab CI.
What did I forget/do wrong that prevents the test report generation from GitLab CI?
My .yml for testing, look like this - It's very basic but I've just run it again and it was running fine:
stages:
- test
newman_tests:
stage: test
image:
name: postman/newman_alpine33
entrypoint: [""]
script:
- newman --version
- npm install -g newman-reporter-htmlextra
- newman run collection.json -e environment.json --reporters cli,htmlextra --reporter-htmlextra-export testReport.html
artifacts:
when: always
paths:
- testReport.html
One thing that I do have is entrypoint: [""] in the image block.
I am testing a GitLab CI pipeline with gitlab-runner exec. During a script, Boost ran into an error, and it created a log file. I want to view this log file, but I do not know how to.
.gitlab-ci.yml in project directory:
image: alpine
variables:
GIT_SUBMODULE_STRATEGY: recursive
build:
script:
- apk add cmake
- cd include/boost
- sh bootstrap.sh
I test this on my machine with:
sudo gitlab-runner exec docker build --timeout 3600
The last several lines of the output:
Building Boost.Build engine with toolset ...
Failed to build Boost.Build build engine
Consult 'bootstrap.log' for more details
ERROR: Job failed: exit code 1
FATAL: exit code 1
bootstrap.log is what I would like to view.
Appending - cat bootstrap.log to .gitlab-ci.yml does not output the file contents because the runner exits before this line. I tried looking though past containers with sudo docker ps -a, but this does not show the one that GitLab Runner used. How can I open bootstrap.log?
You can declare an artifact for the log:
image: alpine
variables:
GIT_SUBMODULE_STRATEGY: recursive
build:
script:
- apk add cmake
- cd include/boost
- sh bootstrap.sh
artifacts:
when: on_failure
paths:
- include/boost/bootstrap.log
Afterwards, you will be able to download the log file via the web interface.
Note that using when: on_failure will ensure that bootstrap.log will only be collected if the build fails, saving disk space on successful builds.