Jenkins using docker agent with environment declarative pipeline - jenkins

I would like to install maven and npm via docker agent using Jenkins declarative pipeline. But When I would like to use below script Jenkins throws an error as below. It might be using agent none but how can I use node with docker agent via declarative pipeline jenkins.
ERROR: Attempted to execute a step that requires a node context while
‘agent none’ was specified. Be sure to specify your own ‘node { ... }’
blocks when using ‘agent none’.
I try to set agent any but this time I received an error "Still waiting to schedule task
Waiting for next available executor"
pipeline {
agent none
// environment{
proxy = https://
// stable_revision = sh(script: 'curl -H "Authorization: Basic $base64encoded"
// }
stages {
stage('Build') {
agent {
docker { image 'maven:3-alpine'}
}
steps {
sh 'mvn --version'
echo "$apigeeUsername"
echo "Stable Revision: ${env.stable_revision}"
}
}
stage('Test') {
agent { docker { image 'maven:3-alpine' image 'node:8.12.0' } }
environment {
HOME = '.'
}
steps {
script{
try{
sh 'npm install'
sh 'node --version'
//sh 'npm test/unit/*.js'
}catch(e){
throw e
}
}
}
}
// stage('Policy-Code Analysis') {
// steps{
// sh "npm install -g apigeelint"
// sh "apigelint -s wiservice_api_v1/apiproxy/ -f codeframe.js"
// }
// }
stage('Promotion'){
steps{
timeout(time: 2, unit: 'DAYS') {
input 'Do you want to Approve?'
}
}
}
stage('Deployment'){
steps{
sh "mvn -f wiservice_api_v1/pom.xml install -Ptest -Dusername=${apigeeUsername} -Dpassword=${apigeePassword} -Dapigee.config.options=update"
//sh "mvn apigee-enterprise:install -Ptest -Dusername=${apigeeUsername} -Dpassword=${apigeePassword} "
}
}
}
}

Basically your error message tells you everything you need to know:
ERROR: Attempted to execute a step that requires a node context while
‘agent none’ was specified. Be sure to specify your own ‘node { ... }’
blocks when using ‘agent none’.
so what is the issue here? You use agent none for your pipeline which means you do not specify a specific agent for all stages. An agent executes a specific stage. If a stage has no agent it can't be executed and this is your issue here.
The following 2 stage have no agent which means no docker-container / server or whatever where it can be executed.
stage('Promotion'){
steps{
timeout(time: 2, unit: 'DAYS') {
input 'Do you want to Approve?'
}
}
}
stage('Deployment'){
steps{
sh "mvn -f wiservice_api_v1/pom.xml install -Ptest -Dusername=${apigeeUsername} -Dpassword=${apigeePassword} -Dapigee.config.options=update"
//sh "mvn apigee-enterprise:install -Ptest -Dusername=${apigeeUsername} -Dpassword=${apigeePassword} "
}
}
so you have to add agent { ... } to both stage seperately or use a global agent like following and remove the agent from your stages:
pipeline {
agent {
docker { image 'maven:3-alpine'}
} ...
For further information see guide to set up master and agent machines or distributed jenkins builds or the official documentation.

I think you meant to add agent any instead of agent none, because each stage requires at least one agent (either declared at the top for the pipeline or per stage).
Also, I see some more issues.
Your Test stage specifies two images for the same stage.
agent { docker { image 'maven:3-alpine' image 'node:8.12.0' } } although, your stage is executing only npm commands. I believe only one of the image will be downloaded.
To clarify bit more on mkemmerz answer, your Promotion stage is designed correctly. If you plan to have an input step in the pipeline, do not add an agent for the pipeline because input steps block the executor context. See this link https://jenkins.io/blog/2018/04/09/whats-in-declarative/

Related

Jenkins declarative pipeline: npm command not found

So I have set up this jenkins ec2 instance, ssh into it, globally installed node and set PATH. But when executing my pipeline, it gives me npm command not found error.
I put echo $PATH in my pipeline and the result is:
/home/ec2-user/.nvm/versions/node/v10.15.1/bin:/sbin:/usr/sbin:/bin:/usr/bin
Which looks correct.
For reference, here's my very simple pipeline:
pipeline {
agent { label 'master' }
environment {
PATH = "/home/ec2-user/.nvm/versions/node/v10.15.1/bin:${env.PATH}"
}
stages {
stage('Test npm') {
steps {
sh """
echo $PATH
npm --version
"""
}
}
}
}
Appreciate with any help.
As #Dibakar Adtya pointed, the problem is when jenkins executes a pipeline, it's under the user jenkins, whereas I configured node under another user, ec2-user, and jenkins doesn't have access to ec2-user's bin. Thank you #Dibakar!
A more elegant solution is to use Jenkins NodeJS Plugin. It saves you from the environment hassles. Now the pipeline is:
pipeline {
agent { label 'master' }
tools { nodejs "nodejs" }
stages {
stage('Test npm') {
steps {
sh """
npm --version
"""
}
}
}
}

How to setup sonar scanner in Jenkins Declarative Pipeline

I'm facing a problem in implementing SonarQube scanner for my repository in Jenkinsfile. I don't know where should I add the properties of SonarQube scanner in the Jenkinsfile.
I've set Jenkins locally on my windows system. The projects are purely based on Python, Ruby & React.
agent {label 'master'}
triggers {
GenricTrigger ([
genricVariables: [
key: 'pr_from_branch', value: '$.pullrequest.source.branch.name'],
[
expressionType: 'JsonPath',
regexpFilter: '',
defaultValue: ''],
token: 'test'])
}
options {
buildDiscarder (
logRotator(numToKeepStr:'5'))
}
stages {
stage ('Initialize & SonarQube Scan') {
steps {
def scannerHome = tool 'sonarScanner';
withSonarQubeEnv('My SonarQube Server') {
bat """
${scannerHome}/bin/sonar-runner.bat
pip install -r requirements.txt
"""
}
}
}
stage('Quality Gate') {
sleep time: 3000, unit: 'MILLISECONDS'
timeout(time: 1, unit: 'MINUTES') { // Just in case something goes wrong, pipeline will be killed after a timeout
def qg = waitForQualityGate() // Reuse taskId previously collected by withSonarQubeEnv
if (qg.status != 'OK') {
error "Pipeline aborted due to quality gate failure: ${qg.status}"
}
}
}
stage ('Smoke Test') {
steps {
bat """
pytest -s -v tests/home/login_test.py
currentBuild.result = 'SUCCESS'
"""
}
}
}
}
The properties include:
-----------------Sonarqube configuration........................
sonar.projectKey=<*****>
sonar.projectName=<project name>
sonar.projectVersion=1.0
sonar.login=<sonar-login-token>
sonar.sources=src
sonar.exclusions=**/*.doc,**/*.docx,**/*.ipch,/node_modules/,
sonar.host.url=http://<url>/
-----------------Sonar for bitbucket plugin configuration...................
sonar.bitbucket.repoSlug=<project name>
sonar.bitbucket.accountName=<name>
sonar.bitbucket.oauthClientKey=<OAuth_Key>
sonar.bitbucket.oauthClientSecret=<OAuth_secret>
sonar.analysis.mode=issues
I can manually add these properties in sonar-project.properties file and set this file in my project root directly but it will be running locally not on the server. So to avoid that I want to add these properties to Jenkinsfile
We run Sonar scanner as a Docker container but it should give you a fair idea of how to use your properties for the same in Jenkinsfile.
stage("Sonar Analysis"){
sh "docker pull docker.artifactory.company.com/util-sonar-runner:latest"
withSonarQubeEnv('sonarqube'){
sh "docker run --rm -v ${workspace}:/opt/spring-service -w /opt/spring-service -e SONAR_HOST_URL=${SONAR_HOST_URL} -e SONAR_AUTH_TOKEN=${SONAR_AUTH_TOKEN} docker.artifactory.company.com/util-sonar-runner:latest /opt/sonar-scanner/bin/sonar-scanner -Dsonar.host.url=${SONAR_HOST_URL} -Dsonar.login=${SONAR_AUTH_TOKEN} -Dsonar.projectKey=spring-service -Dsonar.projectName=spring-service -Dsonar.projectBaseDir=. -Dsonar.sources=./src -Dsonar.java.binaries=./build/classes -Dsonar.junit.reportPaths=./build/test-results/test -Dsonar.jacoco.reportPaths=./build/jacoco/test.exec -Dsonar.exclusions=src/test/java/**/* -Dsonar.fortify.reportPath=fortifyResults-${IMAGE_NAME}.fpr -Dsonar.password="
}
}
You run the pipeline step like this. The sonar server properties can be defined under the profile of the pom.xml file.
steps {
withSonarQubeEnv('SonarQube') {
sh 'mvn -Psonar -Dsonar.sourceEncoding=UTF-8 org.sonarsource.scanner.maven:sonar-maven-plugin:3.0.2:sonar'
}
}
The SonarQube scanner needs to be defined on Jenkins Global tool Configuration section.

Jenkins declarative pipline multiple slave

I have a pipeline with multiple stages, some of them are in parallel. Up until now I had a single code block indicating where the job should run.
pipeline {
triggers { pollSCM '0 0 * * 0' }
agent { dockerfile { label 'jenkins-slave'
filename 'Dockerfile'
}
}
stages{
stage('1'){
steps{ sh "blah" }
} // stage
} // stages
} // pipeline
What I need to do now is run a new stage on a different slave, NOT in docker.
I tried by adding an agent statement for that stage but it seems like it tries to run that stage withing a docker container on the second slave.
stage('test new slave') {
agent { node { label 'e2e-aws' } }
steps {
sh "ifconfig"
} // steps
} // stage
I get the following error message
13:14:23 unknown flag: --workdir
13:14:23 See 'docker exec --help'.
I tried setting the agent to none for the pipeline and using an agent for every step and have run into 2 issues
1. My post actions show an error
2. The stages that have parallel stages also had an error.
I can't find any examples that are similar to what I am doing.
You can use the node block to select a node to run a particular stage.
pipeline {
agent any
stages {
stage('Init') {
steps {
node('master'){
echo "Run inside a MASTER"
}
}
}
}
}

Jenkins pipeline step happens on master instead of slave

I am getting started with Jenkins Pipeline. My pipeline has one simple step that is supposed to run on a different agent - like the "Restrict where this project can be run" option.
My problem is that it is running on master.
They are both Windows machines.
Here's my Jenkinsfile:
pipeline {
agent {label 'myLabel'}
stages {
stage('Stage 1') {
steps {
echo pwd()
writeFile(file: 'test.txt', text: 'Hello, World!')
}
}
}
}
pwd() prints C:\Jenkins\workspace\<pipeline-name>_<branch-name>-Q762JIVOIJUFQ7LFSVKZOY5LVEW5D3TLHZX3UDJU5FWYJSNVGV4Q.
This folder is on master. This is confirmed by the presence of the test.txt file.
I expected test.txt to be created on the slave agent.
Note 1
I can confirm that the pipeline finds the agent because the logs contain:
[Pipeline] node
Running on MyAgent in C:\Jenkins\workspace\<pipeline-name>_<branch-name>-Q762JIVOIJUFQ7LFSVKZOY5LVEW5D3TLHZX3UDJU5FWYJSNVGV4Q
But this folder does not exist on MyAgent, which seems related to the problem.
Note 2
This question is similar to Jenkins pipeline not honoring agent specification
, except that I'm not using the build instruction so I don't think the answer applies.
Note 3
pipeline {
agent any
stages {
stage('Stage 1') {
steps {
echo "${env.NODE_NAME}"
}
}
stage('Stage 2') {
agent {label 'MyLabel'}
steps {
echo "${env.NODE_NAME}"
}
}
}
}
This prints the expected output - master and MyAgent. If this is correct, then why is the workspace located in a different folder on master instead of being on MyAgent?
here is an example
pipeline {
agent none
stages {
stage('Example Build') {
agent { label 'build-label' }
steps {
sh 'env'
sh ' sleep 8'
}
}
stage('Example Test') {
agent { label 'deploy-label' }
steps {
sh 'env'
sh ' sleep 5'
}
}
}
}
I faced similar issue and the following pipeline code worked for me (i.e. the file got created on the Windows slave instead of Windows master),
pipeline {
agent none
stages {
stage("Stage 1") {
steps {
node('myLabel'){
script {
writeFile(file: 'test.txt', text: 'Hello World!', encoding: 'UTF-8')
}
// This should print the file content on slave (Hello World!)
bat "type test.txt"
}
}
}
}
}
I'm debugging a completely unrelated issue and this fact was thrown in my face. Apparently the pipeline is processed in the built-in node (previously known as the master node), with the steps being forwarded to the agent.
So even though echo runs on the agent, but pwd() will run on the built-in node. You can do sh 'pwd' to get the path on the agent.

Pass variables between Jenkins stages

I want to pass a variable which I read in stage A towards stage B somehow. I see in some examples that people write it to a file, but I guess that is not really a nice solution. I tried writing it to an environment variable, but I'm not really successful on that. How can I set it up properly?
To get it working I tried a lot of things and read that I should use the """ instead of ''' to start a shell and escape those variables to \${foo} for example.
Below is what I have as a pipeline:
#!/usr/bin/env groovy
pipeline {
agent { node { label 'php71' } }
environment {
packageName='my-package'
packageVersion=''
groupId='vznl'
nexus_endpoint='http://nexus.devtools.io'
nexus_username='jenkins'
nexus_password='J3nkins'
}
stages{
// Package dependencies
stage('Install dependencies') {
steps {
sh '''
echo Skip composer installation
#composer install --prefer-dist --optimize-autoloader --no-interaction
'''
}
}
// Unit tests
stage('Unit Tests') {
steps {
sh '''
echo Running PHP code coverage tests...
#composer test
'''
}
}
// Create artifact
stage('Package') {
steps {
echo 'Create package refs'
sh """
mkdir -p ./build/zpk
VERSIONTAG=\$(grep 'version' composer.json)
REGEX='"version": "([0-9]+.[0-9]+.[0-9]+)"'
if [[ \${VERSIONTAG} =~ \${REGEX} ]]
then
env.packageVersion=\${BASH_REMATCH[1]}
/usr/bin/zs-client packZpk --folder=. --destination=./build/zpk --name=${env.packageName}-${env.packageVersion}.zpk --version=${env.packageVersion}
else
echo "No version found!"
exit 1
fi
"""
}
}
// Publish ZPK package to Nexus
stage('Publish packages') {
steps {
echo "Publish ZPK Package"
sh "curl -u ${env.nexus_username}:${env.nexus_password} --upload-file ./build/zpk/${env.packageName}-${env.packageVersion}.zpk ${env.nexus_endpoint}/repository/zpk-packages/${groupId}/${env.packageName}-${env.packageVersion}.zpk"
archive includes: './build/**/*.{zpk,rpm,deb}'
}
}
}
}
As you can see the packageVersion which I read from stage Package needs to be used in stage Publish as well.
Overall tips against the pipeline are of course always welcome as well.
A problem in your code is that you are assigning version of environment variable within the sh step. This step will execute in its own isolated process, inheriting parent process environment variables.
However, the only way of passing data back to the parent is through STDOUT/STDERR or exit code. As you want a string value, it is best to echo version from the sh step and assign it to a variable within the script context.
If you reuse the node, the script context will persist, and variables will be available in the subsequent stage. A working example is below. Note that any try to put this within a parallel block can be of failure, as the version information variable can be written to by multiple processes.
#!/usr/bin/env groovy
pipeline {
environment {
AGENT_INFO = ''
}
agent {
docker {
image 'alpine'
reuseNode true
}
}
stages {
stage('Collect agent info'){
steps {
echo "Current agent info: ${env.AGENT_INFO}"
script {
def agentInfo = sh script:'uname -a', returnStdout: true
println "Agent info within script: ${agentInfo}"
AGENT_INFO = agentInfo.replace("/n", "")
env.AGENT_INFO = AGENT_INFO
}
}
}
stage("Print agent info"){
steps {
script {
echo "Collected agent info: ${AGENT_INFO}"
echo "Environment agent info: ${env.AGENT_INFO}"
}
}
}
}
}
Another option which doesn't involve using script, but is just declarative, is to stash things in a little temporary environment file.
You can then use this stash (like a temporary cache that only lives for the run) if the workload is sprayed out across parallel or distributed nodes as needed.
Something like:
pipeline {
agent any
stages {
stage('first stage') {
steps {
// Write out any environment variables you like to a temporary file
sh 'echo export FOO=baz > myenv'
// Stash away for later use
stash 'myenv'
}
}
stage ("later stage") {
steps {
// Unstash the temporary file and apply it
unstash 'myenv'
// use the unstashed vars
sh 'source myenv && echo $FOO'
}
}
}
}

Resources