I am trying to install conda inside a circleci docker image and add it to the .bashrc file.
Here're some relevant portions from /.circleci/config.yml:
version: 2
jobs:
build:
docker:
- image: circleci/python:3.5
environment:
BASH_ENV: /home/circleci/.bashrc
DISTRIB: "conda"
PYTHON_VERSION: "3.5"
NUMPY_VERSION: "*"
SCIPY_VERSION: "*"
SCIKIT_LEARN_VERSION: "*"
MATPLOTLIB_VERSION: "*"
steps:
- checkout
- run: wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
- run: chmod +x ~/miniconda.sh && ~/miniconda.sh -b
- run: echo "export PATH=/home/circleci/miniconda3/bin:$PATH" >> $BASH_ENV
- run: echo $PATH # Added for troubleshooting.
However here is the output I get:
====>> echo "export PATH=/home/circleci/miniconda3/bin:$PATH" >> $BASH_ENV
#!/bin/bash -eo pipefail
echo "export PATH=/home/circleci/miniconda3/bin:$PATH" >> $BASH_ENV
====>> echo $PATH
#!/bin/bash -eo pipefail
echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
So the path to conda is not getting added to .bashrc.
How do I solve this? I am at my wits end.
This is supposed to work.
Appreciate the help!
Got the answer from CircleCI Support. Posting it here for anyone else who comes to SO looking for help:
Please not the default behavior of the shell is non-login meaning the the .bashrc file is not read.
https://circleci.com/docs/2.0/configuration-reference/#default-shell-options
You can change the default shell to /bin/sh -eo --login pipefail
You could also use BASH_ENV alternatively.
https://circleci.com/docs/2.0/env-vars/#using-bash_env-to-set-environment-variables
Related
I am downloading and unzipping binaryen in a run step.
- run: wget -c https://github.com/WebAssembly/binaryen/releases/download/version_101/binaryen-version_101-x86_64-linux.tar.gz -O - | tar -xz -C /tmp/
I am then updating the path in $BASH_ENV.
- run: echo "export PATH=/tmp/binaryen-version_101/bin/wasm-opt:\${PATH}" >> $BASH_ENV
However, I still get a command not found for wasm-opt.
How can I install the downloaded wasm-opt binary such that another run step can use it?
The main issue is that the PATH variable should contain a list of directories. You added the actual binary itself to the path instead of the directory it resides in.
So for example, instead of /tmp/binaryen-version_101/bin/wasm-opt you want /tmp/binaryen-version_101/bin/. Also, after you add a directory to the PATH you won't be able to run those binaries until the next step.
Here's an example config I made:
version: 2.1
workflows:
main:
jobs:
- build
jobs:
build:
docker:
- image: cimg/base:stable
steps:
- checkout
- run: curl -sSL "https://github.com/WebAssembly/binaryen/releases/download/version_101/binaryen-version_101-x86_64-linux.tar.gz" | tar -xz -C /tmp/
- run: echo 'export PATH=/tmp/binaryen-version_101/bin/:${PATH}' >> $BASH_ENV
- run: wasm-opt
I am running this .yml file
kind: pipeline
type: docker
name: default
steps:
- name: test
image: ubuntu
commands:
- apt-get update
- apt-get install git -y
- ./build.sh bin
first, two commands are running correctly but in the last command it fails and the drone doesn't give me a message
![drone messages get while testing][1]
This fails after echo if1
PROJECT_PATH=$PWD
RUN=$1
ENVIRONMENT=${ENVIRONMENT:="local"}
if [ "$ENVIRONMENT" == "local" ]; then
echo if1
GIT_AUTHOR="$(git config --global --get user.name)"
GIT_AUTHOR_EMAIL="$(git config --global --get user.email)"
GIT_VERSION="$(git describe --tags --always --dirty)"
VERSION=$GIT_VERSION
echo endif
else
echo else1
DRONE_COMMIT_SHORT=${DRONE_COMMIT:0:8}
VERSION=${DRONE_TAG:=$DRONE_COMMIT_SHORT}
echo endelse
``
[1]: https://i.stack.imgur.com/ChrUo.png
You need shebang
#!/usr/bin/env bash
Your bash
When running a sh script in docker file, i got the following error:
./upload.sh: 5: ./upload.sh: sudo: not found ./upload.sh: 21:
./upload.sh: Bad substitution
sudo chmod 755 upload.sh # line 5
version=$(git rev-parse --short HEAD)
echo "version $version"
echo "Uploading file"
for path in $(find public/files -name "*.txt"); do
echo "path $path"
WORDTOREMOVE="public/"
echo "WORDTOREMOVE $WORDTOREMOVE"
# cause of the error
newpath=${path//$WORDTOREMOVE/} # Line 21
echo "new path $path"
url=http://localhost:3000/${newpath}
...
echo "Uploading file"
...
done
DockerFile
FROM node:10-slim
EXPOSE 3000 4001
WORKDIR /prod/code
...
COPY . .
RUN ./upload.sh
RUN npm run build
CMD ./DockerRun.sh
Any idea?
If anyone faces the same issue, here how I fixed it
chmod +x upload.sh
git update-index --chmod=+x upload.sh (mandatory if you pushed the file to remote branch before changing its permission)
The docker image you are using (node:10-slim) has no sudo installed on it because this docker image runs processes as user root:
docker run -it node:10-slim bash
root#68dcffceb88c:/# id
uid=0(root) gid=0(root) groups=0(root)
root#68dcffceb88c:/# which sudo
root#68dcffceb88c:/#
When your Dockerfile runs RUN ./upload.sh it will run:
sudo chmod 755 upload.sh
Using sudo inside the docker fails because sudo is not installed, there is no need to use sudo inside the docker because all of the commands inside the docker run as user root.
Simply remove the sudo from line number 5.
If you wish to update the running PATH variable run:
PATH=$PATH:/directorytoadd/bin
This will append the directory "/directorytoadd/bin" to the current path.
The app I am working on is currently using a buildpack that is using ffmpegthumbnailer 2.0.8 and I need a feature that was added in version 2.1.2. I have forked the repo of the buildpack we are currently using https://github.com/akomic/heroku-buildpack-ffmpegthumbnailer and updated the bin/compile file to point the download_url to "https://github.com/dirkvdb/ffmpegthumbnailer/archive/2.2.0.tar.gz" but when I add my forked repo to the app and run heroku run "ffmpegthumbnailer -version" to verify that it worked I get a bash: ffmpegthumbnailer: command not found error.
Here is the original bin/compile:
#!/bin/sh
indent() {
sed -u 's/^/ /'
}
echo "-----> Install ffmpegthumbnailer"
BUILD_DIR=$1
VENDOR_DIR="vendor"
DOWNLOAD_URL="http://www.aksiom.net/stuff/ffmpegthumbnailer_2.0.8-2.bin.tar.gz"
echo "DOWNLOAD_URL = " $DOWNLOAD_URL | indent
cd $BUILD_DIR
mkdir -p $VENDOR_DIR
cd $VENDOR_DIR
curl -L --silent $DOWNLOAD_URL | tar zx
echo "exporting PATH and LIBRARY_PATH" | indent
PROFILE_PATH="$BUILD_DIR/.profile.d/ffmpeg.sh"
mkdir -p $(dirname $PROFILE_PATH)
echo 'export PATH="$PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/bin"' >> $PROFILE_PATH
echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/lib"' >> $PROFILE_PATH
Here is my updated bin/compile in the forked repo:
#!/bin/sh
indent() {
sed -u 's/^/ /'
}
echo "-----> Install ffmpegthumbnailer"
BUILD_DIR=$1
VENDOR_DIR="vendor"
DOWNLOAD_URL="https://github.com/dirkvdb/ffmpegthumbnailer/archive/2.2.0.tar.gz"
echo "DOWNLOAD_URL = " $DOWNLOAD_URL | indent
cd $BUILD_DIR
mkdir -p $VENDOR_DIR
cd $VENDOR_DIR
curl -L --silent $DOWNLOAD_URL | tar zx
echo "exporting PATH and LIBRARY_PATH" | indent
PROFILE_PATH="$BUILD_DIR/.profile.d/ffmpeg.sh"
mkdir -p $(dirname $PROFILE_PATH)
echo 'export PATH="$PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/bin"' >> $PROFILE_PATH
echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/lib"' >> $PROFILE_PATH
Other info: I add the buildpack using heroku buildpacks:add <link_to_forked_repo>
Thank you for any help!
This is because the original akomic tarball specified is not the source code, but rather a custom zip of compiled binaries and libraries. Notice that the compile script has no make step, for instance. There is no documentation for how this custom tarball created, but presumably if you go through the same process with a newer version of the source code, and then upload it to a place this script can access it, it will work again. I'm trying to figure this out now.
I am trying to put together a CI environment for a .NET application using the following stack (just the relevant ones):
Debian + mono
Docker
Gitlab CI
Gitlab-multi-runner (as a docker container)
Sonarqube + Postgre
I've used docker-compose to create the container for sonarqube and postgre, both are running and working. I am sadly stuck with executing sonarqube analysis for my build executed by the gitlab runner and all examples I found were using Maven. I've tried to use sonar-scanner as well, no luck so far.
Here are the contents of my gitlab-ci.yml:
image: mono:latest
cache:
paths:
- ./src/T_GitLabCi/packages/
stages:
- build
.shared: &restriction
only:
- master
tags:
- docker
build:
<<: *restriction
stage: build
script:
- nuget restore ./src/T_GitLabCi
- MONO_IOMAP=case xbuild /t:Build /p:Configuration="Release" /p:Platform="Any CPU" ./src/T_GitLabCi/T_GitLabCi.sln
- mono ./tools/NUnitConsoleRunner/nunit3-console.exe ./src/T_GitLabCi/T_GitLabCi.sln --work=./src/T_GitLabCi/test --config=Release
- << EXECUTE SONAR ANALYSIS >>
I am definitely missing something here. Could somebody point me the right direction?
I have projects written in PHP but that shouldn't matter. Here's what I did.
I enabled a private registry hosted on my GitLab installation
In this registry I have a "sonar-scanner" image built from this Dockerfile (it's based on one of the images available on Docker hub):
FROM java:alpine
ENV SONAR_SCANNER_VERSION 2.8
RUN apk add --no-cache wget && \
wget https://sonarsource.bintray.com/Distribution/sonar-scanner-cli/sonar-scanner-${SONAR_SCANNER_VERSION}.zip && \
unzip sonar-scanner-${SONAR_SCANNER_VERSION} && \
cd /usr/bin && ln -s /sonar-scanner-${SONAR_SCANNER_VERSION}/bin/sonar-scanner sonar-scanner && \
apk del wget
COPY files/sonar-scanner-run.sh /usr/bin
and here's the files/sonar-scanner-run.sh file:
#!/bin/sh
URL="<YOUR SONARQUBE URL>"
USER="<SONARQUBE USER THAT CAN ACCESS THE PROJECTS>"
PASSWORD="<USER PASSWORD>"
if [ -z "$SONAR_PROJECT_KEY" ]; then
echo "Undefined \"projectKey\"" && exit 1
else
COMMAND="sonar-scanner -Dsonar.host.url=\"$URL\" -Dsonar.login=\"$USER\" -Dsonar.password=\"$PASSWORD\" -Dsonar.projectKey=\"$SONAR_PROJECT_KEY\""
if [ ! -z "$SONAR_PROJECT_VERSION" ]; then
COMMAND="$COMMAND -Dsonar.projectVersion=\"$SONAR_PROJECT_VERSION\""
fi
if [ ! -z "$SONAR_PROJECT_NAME" ]; then
COMMAND="$COMMAND -Dsonar.projectName=\"$SONAR_PROJECT_NAME\""
fi
if [ ! -z $CI_BUILD_REF ]; then
COMMAND="$COMMAND -Dsonar.gitlab.commit_sha=\"$CI_BUILD_REF\""
fi
if [ ! -z $CI_BUILD_REF_NAME ]; then
COMMAND="$COMMAND -Dsonar.gitlab.ref_name=\"$CI_BUILD_REF_NAME\""
fi
if [ ! -z $SONAR_BRANCH ]; then
COMMAND="$COMMAND -Dsonar.branch=\"$SONAR_BRANCH\""
fi
if [ ! -z $SONAR_ANALYSIS_MODE ]; then
COMMAND="$COMMAND -Dsonar.analysis.mode=\"$SONAR_ANALYSIS_MODE\""
if [ $SONAR_ANALYSIS_MODE="preview" ]; then
COMMAND="$COMMAND -Dsonar.issuesReport.console.enable=true"
fi
fi
eval $COMMAND
fi
Now in my project in .gitlab-ci.yml I have something like this:
SonarQube:
image: <PATH TO YOUR IMAGE ON YOUR REGISTRY>
variables:
SONAR_PROJECT_KEY: "<YOUR PROJECT KEY>"
SONAR_PROJECT_NAME: "$CI_PROJECT_NAME"
SONAR_PROJECT_VERSION: "$CI_BUILD_ID"
script:
- /usr/bin/sonar-scanner-run.sh
That't pretty much all. The above example of .gitlab-ci.yml is simplified since I'm using diffrent builds for master and other branches (like when: manual) and I use this plugin to get feedback in GitLab: https://gitlab.talanlabs.com/gabriel-allaigre/sonar-gitlab-plugin
Feel free to ask if you have any questions. It took me some time to put this all together the way I want it :) Actually I'm still finetuning it.
You need to install sonar-scanner first. You can find portage of sonar-scanner for almost any recent language, for example for npm you don't have to use directly the java executor:
I only add to do this :
npm install --save sonar-scanner
Then I needed to add this in my package.json
"scripts": {
"sonar-scanner": "node_modules/sonar-scanner/bin/sonar-scanner"
}
This is my job in .gitlab-ci.yml:
job_testmaster:
stage: test
script:
- PACKAGE_VERSION=$(node -p "require('./package.json').version")
- echo sonar.projectVersion=${PACKAGE_VERSION} >> sonar-project.properties
- npm run build
- npm run sonar-scanner -- -Dsonar.login=${SONAR_LOGIN}
only:
- master
tags:
- docker
With this, I am able to start sonar analysis, but I am not able to use the quality gates after.
Hope this help.