Building Go app with "vendor" directory on Jenkins with Docker - docker

I'm trying to set up a Jenkins Pipeline to build and deploy my first Go project using a Jenkinsfile and docker.image().inside . I can't figure out how to get go to pick up the dependencies in the vendor/ directory.
When I run the build, I get a bunch of errors:
+ goapp test ./...
src/dao/demo_dao.go:8:2: cannot find package "github.com/dgrijalva/jwt-go" in any of:
/usr/lib/go_appengine/goroot/src/github.com/dgrijalva/jwt-go (from $GOROOT)
/usr/lib/go_appengine/gopath/src/github.com/dgrijalva/jwt-go (from $GOPATH)
/workspace/src/github.com/dgrijalva/jwt-go
...why isn't it picking up the Vendor directory?
When I throw in some logging, it seems that after running sh "cd /workspace/src/bitbucket.org/nalbion/go-demo" the next sh command is still in the original ${WORKSPACE} directory. I really like the idea of the Jenkins file, but I can't find any decent documentation for it.
(Edit - there is decent documentation here but dir("/workspace/src/bitbucket.org/nalbion/go-demo") {} doesn't seem to work within docker.image().inside)
My Docker file resembles:
FROM golang:1.6.2
# Google's App Engine Go SDK
RUN wget https://storage.googleapis.com/appengine-sdks/featured/go_appengine_sdk_linux_amd64-1.9.40.zip -q -O go_appengine_sdk.zip && \
unzip -q go_appengine_sdk.zip -d /usr/lib/ && \
rm go_appengine_sdk.zip
ENV PATH /usr/lib/go_appengine:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV GOPATH /usr/lib/go_appengine/gopath
# Add Jenkins user
RUN groupadd -g 132 jenkins && useradd -d "/var/jenkins_home" -u 122 -g 132 -m -s /bin/bash jenkins
And my Jenkinsfile:
node('docker') {
currentBuild.result = "SUCCESS"
try {
stage 'Checkout'
checkout scm
stage 'Build and Test'
env.WORKSPACE = pwd()
docker.image('nalbion/go-web-build:latest').inside(
"-v ${env.WORKSPACE}:/workspace/src/bitbucket.org/nalbion/go-demo " +
"-e GOPATH=/usr/lib/go_appengine/gopath:/workspace") {
// Debugging
sh 'echo GOPATH: $GOPATH'
sh "ls -al /workspace/src/bitbucket.org/nalbion/go-demo"
sh "cd /workspace/src/bitbucket.org/nalbion/go-demo"
sh "pwd"
sh "go vet ./src/..."
sh "goapp test ./..."
}
stage 'Deploy to DEV'
docker.image('nalbion/go-web-build').inside {
sh "goapp deploy --application go-demo --version v${v} app.yaml"
}
timeout(time:5, unit:'DAYS') {
input message:'Approve deployment?', submitter: 'qa'
}
stage 'Deploy to PROD'
docker.image('nalbion/go-web-build').inside {
sh "goapp deploy --application go-demo --version v${v} app.yaml"
}
} catch (err) {
currentBuild.result = "FAILURE"
// send notifications
throw err
}
}

I managed to get it working by including the cd in the same sh statement:
docker.image('nalbion/go-web-build:latest')
.inside("-v ${env.WORKSPACE}:/workspace/src/bitbucket.org/nalbion/go-demo " +
"-e GOPATH=/usr/lib/go_appengine/gopath:/workspace") {
sh """
cd /workspace/src/bitbucket.org/nalbion/go-demo
go vet ./src/...
goapp test ./...
"""
}

Related

How can I load credentials.json file into jenkins pipeline job?

I have a script which runs against google Directory API.
The purpose of this script is to download all users from Google's Directory of our company.
But when I run that script it give an error
FileNotFoundError: [Errno 2] No such file or directory: 'credentials.json'
Although that file is in the folder.
Do I need to put that file into the Credential manager or ....?
I also have a groovy file which just install the pip and activate venv and nothing else.
Here is groovy code
stage('Check activity') {
steps {
sh 'pwd'
sh '''#!/bin/bash
set -e
if ! which pipenv >/dev/null; then
echo \'no pipenv, installing...\'
pip3 install --user pipenv
if ! which pipenv >/dev/null; then
# default location for: /home/jenkins/.local/bin/pipenv
my_pip_env="/home/${USER}/.local/bin/pipenv"
fi
else
echo \'pipenv already installed, nothing to do.\'
my_pip_env=$(which pipenv)
fi
# pipenv version, check & install
${my_pip_env} --version
${my_pip_env} install
# run script
PYTHONPATH=$(pwd):${PYTHONPATH} \\
PIPENV_PIPFILE=$(realpath ./Pipfile) \\
${my_pip_env} run -v python3 ./it/google-users/google_user.py -v "${VERSION}" -dr "${DRY_RUN}" -et "${EXCLUDED_TYPES}"
# Remove virtualenv project
${my_pip_env} --rm
'''
}
}
do I need to define environment variables in groovy script?
When reading a file from your workspace, it is best to use readFile function
readFile('credentials.json')
In your case, you could read it into a variable and then pass it into the next steps of your script. Something like this:
pipeline {
agent any
stages {
stage('Check activity')
{
steps {
script {
sh '''#!/bin/bash
echo "hello" > hello.txt
'''
def mydata = readFile('hello.txt')
sh "echo My file data: ${mydata}"
}
}
}
}
}

wget: command not found in Jenkins Pipeline

in my Mac, wget command working. How to fix this issue?
Error Message
wget
https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip
/Users/don/.jenkins/workspace/demo#tmp/durable-2702e009/script.sh:
line 1: wget: command not found
Full Pipeline Script
node('master') {
def home = sh(script: "echo $ANDROID_HOME",returnStdout: true).trim()
def SDKPath = "$home/Android/sdk"
stage("Preparing SDK"){
// Check SDK Downloaded
def isSDKDownloaded = sh(script: "test -e sdk-tools-linux-4333796.zip && echo true || echo false",returnStdout: true).trim()
if(isSDKDownloaded == "false"){
// Download SDK
sh "wget 'https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip'"
}
// Check if SDK is Extracted
def isExtracted = sh(script: "test -e $SDKPath/tools && echo true || echo false",returnStdout: true).trim()
if(isExtracted == "false"){
sh "mkdir -p $SDKPath"
//Unzip SDK
sh "unzip sdk-tools-linux-4333796.zip -d $SDKPath"
}
// Install SDK Tools
sh "yes | $SDKPath/tools/bin/sdkmanager 'build-tools;28.0.3' 'platform-tools' 'platforms;android-27'"
sh "ls $SDKPath/licenses"
// See installed And Available SDK
sh "$SDKPath/tools/bin/sdkmanager --list"
// Accept All SDK Licences
sh "yes | $SDKPath/tools/bin/sdkmanager --licenses"
}
def selectedBranch = SELECTED_RELEASE_BRANCH
stage('Checkout') {
git branch: selectedBranch, url: 'git#gitlab.com:o-apps/demo.git'
// Remove Existing local properties
sh 'rm local.properties ||:'
// Write sdk.dir Path into local properties file
sh "echo 'sdk.dir=$SDKPath' >> local.properties"
}
stage('Setup Tools') {
withCredentials([file(credentialsId: 'android_keystore', variable: 'KEYFILE')]) {
sh "cp \$KEYFILE app/key.jks"
}
}
stage('Build Release APK') {
sh "./gradlew clean assembleRelease"
}
stage('Upload to Play Store') {
androidApkUpload googleCredentialsId: 'key', apkFilesPattern: '**/*-release.apk', trackName: 'alpha'
}
stage('Cleanup Credential') {
sh "rm app/key.jks"
}
}
This is probably due to the $PATH environment variable which is different between your user and the user running Jenkins. Your user may be altering its $PATH by expanding it in the shell resource file (~/.bashrc, ~/.zshrc).
Not to worry, you can use the full path.
To find out the full path to wget, run this on the machine that runs the pipeline (the one labelled master):
% which wget
/usr/local/bin/wget
(Your path may naturally be different.)
Now use the full path:
// Download SDK
sh "/usr/local/bin/wget 'https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip'"

Jenkins pipeline doesn't copy files generated in previous stage

I am setting up a Jenkins pipeline to deploy a PHP application. The application is using composer, so I am running composer install -o in the script to ensure that all dependencies are there. The test setup also ensures that the vendor/autoload.php is generated (it's in the phpunit.xml bootstrap config)
My scripts are based on https://modess.io/jenkins-php/ and http://jenkins-php.org/
My issue is that the vendor folder is not included, and the generated config.inc.php is not included.
The Jenkins log shows that my deployment line sh "cp -rp ${SOURCE_DIR}/* ${DEPLOY_DIR}" is being expanded into cp -rp src/globals.template.inc.php src/index.php src/phpinfo.php /usr/nasShare/htdocs/sometest which is not including the mentioned files. (in fact those are the only files in the src directory in SCM. Looking in the working directory on the jenkins server, the files are generated... )
Jenkinsfile
#!groovy
pipeline {
agent any
environment {
SOURCE_DIR="src"
TEMPLATE_FILE="globals.template.inc.php"
CONFIG_FILE="globals.inc.php"
}
stages {
stage ('Testing'){
steps {
echo "Running ant clean"
sh 'ant clean'
echo "running composer"
sh "composer install -o -d ${SOURCE_DIR}"
echo ""
sh 'ant quick-build'
}
}
stage ('Staging'){
steps {
echo "Building config file"
script {
def inptext = readFile file: "${SOURCE_DIR}/${TEMPLATE_FILE}"
inptext = inptext.replaceAll(~/¤GIT_BRANCH¤/, "${GIT_BRANCH_NAME}")
inptext = inptext.replaceAll(~/¤GIT_COMMIT¤/, "${sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()}")
inptext = inptext.replaceAll(~/¤GIT_TAG¤/, "${sh(returnStdout: true, script: "git -C . describe --tags").trim()}")
writeFile file: "${SOURCE_DIR}/${CONFIG_FILE}", text: inptext
}
}
}
stage ('Remote Deploy'){
agent any
when{
//https://stackoverflow.com/a/44231270/1725871
environment name: 'DEPLOY_TYPE', value: 'remote'
}
steps {
echo "Deploying via SSH on ${SSH_SERVER_NAME}:${DEPLOY_DIR}"
//TODO: rename backup file
sh "ssh ${SSH_USERNAME}#${SSH_SERVER_NAME} tar -cvpzf BACKUP_FNAME ${DEPLOY_DIR}/* "
sh "ssh ${SSH_USERNAME}#${SSH_SERVER_NAME} rm -R ${DEPLOY_DIR}/*"
sh "scp -rp ${SOURCE_DIR}/* ${SSH_USERNAME}#${SSH_SERVER_NAME}:${DEPLOY_DIR}"
//TODO: delete backup on success
}
}
stage ('Local Deploy'){
agent any
when{
environment name: 'DEPLOY_TYPE', value: 'local'
}
steps {
echo "Deploying to ${DEPLOY_DIR} "
//TODO: backup existing files
sh "rm -R ${DEPLOY_DIR}/*"
sh "cp -rp ${SOURCE_DIR}/* ${DEPLOY_DIR}"
}
}
}
}
I found the error of my way. Adding ${WORKSPACE}/ to my SOURCE_DIR variable solved it. now all the expected files are being copied.
Working environment from jenkinsfile
environment {
SOURCE_DIR="${WORKSPACE}/src"
TEMPLATE_FILE="globals.template.inc.php"
CONFIG_FILE="globals.inc.php"
}

Build and Run Docker Container in Jenkins

I need to run docker container in Jenkins so that installed libraries like pycodestyle can be runnable in the following steps.
I successfully built Docker Container (in Dockerfile)
How do I access to the container so that I can use it in the next step? (Please look for >> << code in Build step below)
Thanks
stage('Build') {
// Install python libraries from requirements.txt (Check Dockerfile for more detail)
sh "docker login -u '${DOCKER_USR}' -p '${DOCKER_PSW}' ${DOCKER_REGISTRY}"
sh "docker build \
--tag '${DOCKER_REGISTRY}/${DOCKER_TAG}:latest' \
--build-arg HTTPS_PROXY=${PIP_PROXY} ."
>> sh "docker run -ti ${DOCKER_REGISTRY}/${DOCKER_TAG}:latest sh" <<<
}
}
stage('Linting') {
sh '''
awd=$(pwd)
echo '===== Linting START ====='
for file in $(find . -name '*.py'); do
filename=$(basename $file)
if [[ ${file:(-3)} == ".py" ]] && [[ $filename = *"test"* ]] ; then
echo "perform PEP8 lint (python pylint blah) for $filename"
cd $awd && cd $(dirname "${file}") && pycodestyle "${filename}"
fi
done
echo '===== Linting END ====='
'''
}
You need to mount the workspace of your Jenkins job (containing your python project) as volume (see "docker run -v" option) to your container and then run the "next step" build step inside this container. You can do this by providing a shell script as part of your project's source code, which does the "next step" or write this script in a previous build stage.
It would be something like this:
sh "chmod +x build.sh"
sh "docker run -v $WORKSPACE:/workspace ${DOCKER_REGISTRY}/${DOCKER_TAG}:latest /workspace/build.sh"
build.sh is an executable script, which is part of your project's workspace and performans the "next step".
$WORKSPACE is the folder that is used by your jenkins job (normally /var/jenkins_home/jobs//workspace - it is provided by Jenkins as a build variable.
Please note: This solution requires that the Docker daemon is running on the same host as Jenkins! Otherwise the workspace will not be available to your container.
Another solution would be to run Jenkins as Docker container, so you can share the Jenkins home/workspaces easily with the containers you run within your build jobs, like described here:
Running Jenkins tests in Docker containers build from dockerfile in codebase

Cannot conditionally remove a directory inside the workspace using basic Jenkins pipeline steps

I am trying to remove the directory junit located in the workspace of my Jenkins job using scripted Pipeline which looks somewhat like this:
node {
stage('Build') {
checkout scm
app = docker.build("...")
}
stage('Test') {
app.withRun("--name = ${CONTAINER_ID} ...") {
// sh "mkdir -p junit"
// sh "rm -rf junit/"
dir "junit" {
deleteDir
}
sh "docker exec ${CONTAINER_ID} /bin/bash -c 'source venv/bin/activate && python run.py test -x junit'"
sh "docker cp ${CONTAINER_ID}:/home/foo/junit junit"
}
}
junit 'junit/*.xml'
}
However I am getting the following (red haring?) error, e.g.
java.lang.ClassCastException:
hudson.tasks.junit.pipeline.JUnitResultsStep.testResults expects class
java.lang.String but received class
org.jenkinsci.plugins.workflow.cps.CpsClosure2
However when I am using the shell steps:
sh "mkdir -p junit"
sh "rm -rf junit/"
It works as expected. What am I doing wrong?
Try to use parentheses:
dir ("junit") {
deleteDir()
}

Resources