I am running this .yml file
kind: pipeline
type: docker
name: default
steps:
- name: test
image: ubuntu
commands:
- apt-get update
- apt-get install git -y
- ./build.sh bin
first, two commands are running correctly but in the last command it fails and the drone doesn't give me a message
![drone messages get while testing][1]
This fails after echo if1
PROJECT_PATH=$PWD
RUN=$1
ENVIRONMENT=${ENVIRONMENT:="local"}
if [ "$ENVIRONMENT" == "local" ]; then
echo if1
GIT_AUTHOR="$(git config --global --get user.name)"
GIT_AUTHOR_EMAIL="$(git config --global --get user.email)"
GIT_VERSION="$(git describe --tags --always --dirty)"
VERSION=$GIT_VERSION
echo endif
else
echo else1
DRONE_COMMIT_SHORT=${DRONE_COMMIT:0:8}
VERSION=${DRONE_TAG:=$DRONE_COMMIT_SHORT}
echo endelse
``
[1]: https://i.stack.imgur.com/ChrUo.png
You need shebang
#!/usr/bin/env bash
Your bash
Related
I'm using AWS ECR to host a private Dockerfile image, and I would like to use it in GitLab CI.
Accordingly to the documentation I need to set docker-credential-ecr-login to fetch the private image, but I have no idea how to do that before anything else. That's my .gitlab-ci file:
image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest
tests:
stage: test
before_script:
- echo "before_script"
- apt install amazon-ecr-credential-helper
- apk add --no-cache curl jq python py-pip
- pip install awscli
script:
- echo "script"
- bundle install
- bundle exec rspec
allow_failure: true # for now as we do not have tests
Thank you.
I confirm the feature at stake is not yet available in GitLab CI; however I've recently seen it is possible to implement a generic workaround to run a dedicated CI script within a container taken from a private Docker image.
The template file .gitlab-ci.yml below is adapted from the OP's example, using the Docker-in-Docker approach I suggested in this other SO answer, itself inspired by the GitLab CI doc dealing with dind:
stages:
- test
variables:
IMAGE: "0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest"
REGION: "ap-northeast-1"
tests:
stage: test
image: docker:latest
services:
- docker:dind
variables:
# GIT_STRATEGY: none # uncomment if "git clone" is unneeded for this job
before_script:
- ': before_script'
- apt install amazon-ecr-credential-helper
- apk add --no-cache curl jq python py-pip
- pip install awscli
- $(aws ecr get-login --no-include-email --region "$REGION")
- docker pull "$IMAGE"
script:
- ': script'
- |
docker run --rm -v "$PWD:/build" -w /build "$IMAGE" /bin/bash -c "
export PS4='+ \e[33;1m($CI_JOB_NAME # line \$LINENO) \$\e[0m ' # optional
set -ex
## TODO insert your multi-line shell script here ##
echo \"One comment\" # quotes must be escaped here
: A better comment
echo $PWD # interpolated outside the container
echo \$PWD # interpolated inside the container
bundle install
bundle exec rspec
## (cont'd) ##
"
- ': done'
allow_failure: true # for now as we do not have tests
This example assumes the Docker $IMAGE contains the /bin/bash binary, and relies on the so-called block style of YAML.
The above template already contains comments, but to be self-contained:
You need to escape double quotes if your Bash commands contain them, because the whole code is surrounded by docker run … " and ";
You also need to escape local Bash variables (cf. the \$PWD above), otherwise these variables will be resolved prior running the docker run … "$IMAGE" /bin/bash -c "…" command itself.
I replaced the echo "stuff" or so commands with their more effective colon counterpart:
set -x
: stuff
: note that these three shell commands do nothing
: but printing their args thanks to the -x option.
[Feedback is welcome as I can't directly test this config (I'm not an AWS ECR user), but I'm puzzled by the fact the OP's example contained at the same time some apt and apk commands…]
Related remark on a pitfall of set -e
Beware that the following script is buggy:
set -e
command1 && command2
command3
Namely, write instead:
set -e
command1 ; command2
command3
or:
set -e
( command1 && command2 )
command3
To be convinced about this, you can try running:
bash -e -c 'false && true; echo $?; echo this should not be run'
→ 1
→ this should not be run
bash -e -c 'false; true; echo $?; echo this should not be run'
bash -e -c '( false && true ); echo $?; echo this should not be run'
From GitLab documentation. In order to interact with your AWS account, the GitLab CI/CD pipelines require both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be defined in your GitLab settings under Settings > CI/CD > Variables. Then add to your before script:
image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest
tests:
stage: test
before_script:
- echo "before_script"
- apt install amazon-ecr-credential-helper
- apk add --no-cache curl jq python py-pip
- pip install awscli
- $( aws ecr get-login --no-include-email )
script:
- echo "script"
- bundle install
- bundle exec rspec
allow_failure: true # for now as we do not have tests
Also, you had a typo is awscli, not awsclir.Then add the builds, tests and push accordingly.
I think that you have some sort of logic error in the case. image in the build configuration is a CI scripts runner image, not image you build and deploy.
I think you don't have to use it in any case since it is just an image which has utilities & connections to the GitLab CI & etc. The image shouldn't have any dependencies of your project normally.
Please check examples like this one https://gist.github.com/jlis/4bc528041b9661ae6594c63cd2ef673c to get it more clear how to do it a correct way.
I faced the same problem using docker executor mode of gitlab runner.
SSH into the EC2 instance showed that docker-credential-ecr-login was present in /usr/bin/. To pass it to the container I had to mount this package to the gitlab runner container.
gitlab-runner register -n \
--url '${gitlab_url}' \
--registration-token '${registration_token}' \
--template-config /tmp/gitlab_runner.template.toml \
--executor docker \
--tag-list '${runner_name}' \
--description 'gitlab runner for ${runner_name}' \
--docker-privileged \
--docker-image "alpine" \
--docker-disable-cache=true \
--docker-volumes "/var/run/docker.sock:/var/run/docker.sock" \
--docker-volumes "/cache" \
--docker-volumes "/usr/bin/docker-credential-ecr-login:/usr/bin/docker-credential-ecr-login" \
--docker-volumes "/home/gitlab-runner/.docker:/root/.docker"
More information on this thread as well: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583#note_375018948
We have a similar setup where we need to run CI jobs based off of an Image that is hosted on ECR.
Steps to follow:-
follow this guide here>> https://github.com/awslabs/amazon-ecr-credential-helper
gist of this above link is if you are on "Amazon Linux 2"
sudo amazon-linux-extras enable docker
sudo yum install amazon-ecr-credential-helper
open the ~/.docker/config.json on your gitlab runner in VI editor
Paste this code in the ~/.docker/config.json
{
"credHelpers":
{
"aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login"
}
}
source ~/.bashrc
systemctl restart docker
also remove any references of DOCKER_AUTH_CONFIG from your GitLab>>CI/CD>> Variables
That's it
Hello i have simple configuration in my project:
version: 2
jobs:
build:
docker:
- image: circleci/node:7
steps:
- checkout
- run:
name: install-dependencies
command: npm install
- run:
name: tests
command: npm test
- deploy:
name: digital-ocean
command: ssh -o "StrictHostKeyChecking no" user#hostname "cd ~/profile-store; git pull; npm install; forever start app.js"
The problem is it need multiply command:
cd client
npm start
cd ..
(in second iteration should install packages from server and in the next run unit tests in client)
I tried these syntax:
command: ["cd client", "npm install", "cd .."]
But getting an error. The question is :
How can i write to execute 3 commands in one command instruction?
command: cd client && npm install && cd ..
For enhanced readability, you can use a folded block scalar (folds linebreaks into spaces):
command: >-
cd client &&
npm install &&
cd ..
Note that you do not really need the final cd .. since the shell instance executing the command is not re-used.
I am building my project on CircleCI and I have a build job that looks like this:
build:
<<: *defaults
steps:
- checkout
- setup_remote_docker
- run:
name: Install pip
command: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && sudo python get-pip.py
- run:
name: Install AWS CLI
command: curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip && sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
- run:
name: Login to Docker Registry
command: aws ecr get-login --no-include-email --region us-east-1 | sh
- run:
name: Install Dep
command: curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
- run:
name: Save Version Number
command: echo "export VERSION_NUM=${CIRCLE_TAG}.${CIRCLE_BUILD_NUM}" > deployment/dev/.env
- run:
name: Build App
command: source deployment/dev/.env && docker-compose -f deployment/dev/docker-compose.yml build
- run:
name: Test App
command: |
git config --global url."https://${GITHUB_PERSONAL_ACCESS_TOKEN} :x-oauth-basic#github.com/".insteadOf "https://github.com/"
dep ensure
go test -v ./...
- run:
name: Push Image
command: |
if [[ "${CIRCLE_TAG}" =~ ^v[0.9]+(\.[0-9]+)*-[a-z]*$ ]]; then
source deployment/dev/.env
docker-compose -f deployment/dev/docker-compose.yml push
else
echo 'No tag, not deploying'
fi
- persist_to_workspace:
root: .
paths:
- deployment/*
- tools/*
When I push a change to a branch, the build fails every time with Couldn't connect to Docker daemon at ... - is it running? when it reaches the Build App step of the build job.
Please help me figure out why branch builds are failing but tag builds are not.
I suspect you are hitting this docker-compose bug: https://github.com/docker/compose/issues/6050
The bug reports a misleading error (the one you're getting) when an image name in the docker-compose file is invalid.
If you use an environment variable for the image name or image tag, and that variable is set from a branch name, then it would fail on some branches, but not others.
The problem was occurring on the Save Version Number step. Sometimes that version would be .${CIRCLE_BUILD_NUM} since no tag was passed. Docker dislikes these tags starting with ., so I added a conditional check to see if CIRCLE_TAG was empty, and if it was, use some default version: v0.1.0-build.
I am trying to install conda inside a circleci docker image and add it to the .bashrc file.
Here're some relevant portions from /.circleci/config.yml:
version: 2
jobs:
build:
docker:
- image: circleci/python:3.5
environment:
BASH_ENV: /home/circleci/.bashrc
DISTRIB: "conda"
PYTHON_VERSION: "3.5"
NUMPY_VERSION: "*"
SCIPY_VERSION: "*"
SCIKIT_LEARN_VERSION: "*"
MATPLOTLIB_VERSION: "*"
steps:
- checkout
- run: wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
- run: chmod +x ~/miniconda.sh && ~/miniconda.sh -b
- run: echo "export PATH=/home/circleci/miniconda3/bin:$PATH" >> $BASH_ENV
- run: echo $PATH # Added for troubleshooting.
However here is the output I get:
====>> echo "export PATH=/home/circleci/miniconda3/bin:$PATH" >> $BASH_ENV
#!/bin/bash -eo pipefail
echo "export PATH=/home/circleci/miniconda3/bin:$PATH" >> $BASH_ENV
====>> echo $PATH
#!/bin/bash -eo pipefail
echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
So the path to conda is not getting added to .bashrc.
How do I solve this? I am at my wits end.
This is supposed to work.
Appreciate the help!
Got the answer from CircleCI Support. Posting it here for anyone else who comes to SO looking for help:
Please not the default behavior of the shell is non-login meaning the the .bashrc file is not read.
https://circleci.com/docs/2.0/configuration-reference/#default-shell-options
You can change the default shell to /bin/sh -eo --login pipefail
You could also use BASH_ENV alternatively.
https://circleci.com/docs/2.0/env-vars/#using-bash_env-to-set-environment-variables
I am trying to put together a CI environment for a .NET application using the following stack (just the relevant ones):
Debian + mono
Docker
Gitlab CI
Gitlab-multi-runner (as a docker container)
Sonarqube + Postgre
I've used docker-compose to create the container for sonarqube and postgre, both are running and working. I am sadly stuck with executing sonarqube analysis for my build executed by the gitlab runner and all examples I found were using Maven. I've tried to use sonar-scanner as well, no luck so far.
Here are the contents of my gitlab-ci.yml:
image: mono:latest
cache:
paths:
- ./src/T_GitLabCi/packages/
stages:
- build
.shared: &restriction
only:
- master
tags:
- docker
build:
<<: *restriction
stage: build
script:
- nuget restore ./src/T_GitLabCi
- MONO_IOMAP=case xbuild /t:Build /p:Configuration="Release" /p:Platform="Any CPU" ./src/T_GitLabCi/T_GitLabCi.sln
- mono ./tools/NUnitConsoleRunner/nunit3-console.exe ./src/T_GitLabCi/T_GitLabCi.sln --work=./src/T_GitLabCi/test --config=Release
- << EXECUTE SONAR ANALYSIS >>
I am definitely missing something here. Could somebody point me the right direction?
I have projects written in PHP but that shouldn't matter. Here's what I did.
I enabled a private registry hosted on my GitLab installation
In this registry I have a "sonar-scanner" image built from this Dockerfile (it's based on one of the images available on Docker hub):
FROM java:alpine
ENV SONAR_SCANNER_VERSION 2.8
RUN apk add --no-cache wget && \
wget https://sonarsource.bintray.com/Distribution/sonar-scanner-cli/sonar-scanner-${SONAR_SCANNER_VERSION}.zip && \
unzip sonar-scanner-${SONAR_SCANNER_VERSION} && \
cd /usr/bin && ln -s /sonar-scanner-${SONAR_SCANNER_VERSION}/bin/sonar-scanner sonar-scanner && \
apk del wget
COPY files/sonar-scanner-run.sh /usr/bin
and here's the files/sonar-scanner-run.sh file:
#!/bin/sh
URL="<YOUR SONARQUBE URL>"
USER="<SONARQUBE USER THAT CAN ACCESS THE PROJECTS>"
PASSWORD="<USER PASSWORD>"
if [ -z "$SONAR_PROJECT_KEY" ]; then
echo "Undefined \"projectKey\"" && exit 1
else
COMMAND="sonar-scanner -Dsonar.host.url=\"$URL\" -Dsonar.login=\"$USER\" -Dsonar.password=\"$PASSWORD\" -Dsonar.projectKey=\"$SONAR_PROJECT_KEY\""
if [ ! -z "$SONAR_PROJECT_VERSION" ]; then
COMMAND="$COMMAND -Dsonar.projectVersion=\"$SONAR_PROJECT_VERSION\""
fi
if [ ! -z "$SONAR_PROJECT_NAME" ]; then
COMMAND="$COMMAND -Dsonar.projectName=\"$SONAR_PROJECT_NAME\""
fi
if [ ! -z $CI_BUILD_REF ]; then
COMMAND="$COMMAND -Dsonar.gitlab.commit_sha=\"$CI_BUILD_REF\""
fi
if [ ! -z $CI_BUILD_REF_NAME ]; then
COMMAND="$COMMAND -Dsonar.gitlab.ref_name=\"$CI_BUILD_REF_NAME\""
fi
if [ ! -z $SONAR_BRANCH ]; then
COMMAND="$COMMAND -Dsonar.branch=\"$SONAR_BRANCH\""
fi
if [ ! -z $SONAR_ANALYSIS_MODE ]; then
COMMAND="$COMMAND -Dsonar.analysis.mode=\"$SONAR_ANALYSIS_MODE\""
if [ $SONAR_ANALYSIS_MODE="preview" ]; then
COMMAND="$COMMAND -Dsonar.issuesReport.console.enable=true"
fi
fi
eval $COMMAND
fi
Now in my project in .gitlab-ci.yml I have something like this:
SonarQube:
image: <PATH TO YOUR IMAGE ON YOUR REGISTRY>
variables:
SONAR_PROJECT_KEY: "<YOUR PROJECT KEY>"
SONAR_PROJECT_NAME: "$CI_PROJECT_NAME"
SONAR_PROJECT_VERSION: "$CI_BUILD_ID"
script:
- /usr/bin/sonar-scanner-run.sh
That't pretty much all. The above example of .gitlab-ci.yml is simplified since I'm using diffrent builds for master and other branches (like when: manual) and I use this plugin to get feedback in GitLab: https://gitlab.talanlabs.com/gabriel-allaigre/sonar-gitlab-plugin
Feel free to ask if you have any questions. It took me some time to put this all together the way I want it :) Actually I'm still finetuning it.
You need to install sonar-scanner first. You can find portage of sonar-scanner for almost any recent language, for example for npm you don't have to use directly the java executor:
I only add to do this :
npm install --save sonar-scanner
Then I needed to add this in my package.json
"scripts": {
"sonar-scanner": "node_modules/sonar-scanner/bin/sonar-scanner"
}
This is my job in .gitlab-ci.yml:
job_testmaster:
stage: test
script:
- PACKAGE_VERSION=$(node -p "require('./package.json').version")
- echo sonar.projectVersion=${PACKAGE_VERSION} >> sonar-project.properties
- npm run build
- npm run sonar-scanner -- -Dsonar.login=${SONAR_LOGIN}
only:
- master
tags:
- docker
With this, I am able to start sonar analysis, but I am not able to use the quality gates after.
Hope this help.