Copying files between jenkins slaves In pipeline - jenkins

Our goal is to divide our pipeline with multiple agents.
We have a slave called slave1, that has a sole purpose of checkout out on git,
and building executables.
Eventually when slave1 finishes, we'd like to pass it's output to slave2,
which has a sole purpose of testing slave1's executable.
Please notice that the idea here is not to split jobs, but achieving the
files in the same pipeline.
Heres an example of a Jenkinsfile that'll make more sense:
pipeline
{
agent
{
label 'slave1'
}
stages
{
stage("Initialize & Build")
{
steps
{
script
{
println("Im starting the pipeline with slave1!")
// Builds Files
// ....
// Has many files that needs to pass to slave2
}
}
}
stage("Execute & Test")
{
agent
{
label 'slave2'
}
steps
{
script
{
println("Im in the new slave - slave2!")
// How does this slave get the files?
}
}
}
}
}
How is it possible to pass these files between the agents?
I read about artifacts, but it seems like it's goal is to return the objects from a job, this isn't necessarily what is needed.

If both of your agents are on linux servers, you can simply scp the build output of agent 1 to agent 2.
Also, first you will need to establish passwordless SSH connection between these 2 agents.
Here's an example.
pipeline
{
agent
{
label 'slave1'
}
stages
{
stage("Initialize & Build")
{
steps
{
script
{
println("Im starting the pipeline with slave1!")
// Builds Files
// ....
// option 1: copy files in the workspace of agent 2.
scp $WORKSPACE/build_output/* <user>#agent2:/home/<user>/workspace/<job_name>/
// option 2: copy files to any known location of agent 2
scp $WORKSPACE/build_output/* <user>#agent2:/<destination_path>
}
}
}
stage("Execute & Test")
{
agent
{
label 'slave2'
}
steps
{
script
{
println("Im in the new slave - slave2!")
//option 1
dir($WORKSPACE) {
// Test execution steps
}
// option 2
dir(<destination_path>) {
// Test execution steps
}
}
}
}
}
}

Related

Running script (stash) prior to parallel stages being invoked

I have a parallel stage setup, and would like to know if it's possible to run a script prior to the nested stages, so something like this:
stage('E2E-PR-CYPRESS') {
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
steps {
script {
stash name: 'cypress-dir', includes: 'cypress/**/*'
}
}
parallel {
stage('Cypress Tests 1') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
stage('Cypress Tests 2') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
}
I know the above is wrong, I get the error Only one of "matrix", "parallel", "stages", or "steps" allowed for stage "E2E-PR-CYPRESS"
I already have the stash script in a setup stage at the beginning of my pipeline, but I'd like to be able to restart from this stage above on Jenkins, and so need the stash part in this stage as the parallel stages need to unstash the contents.
Updated Answer:
After playing a bit with the Restart from a Stage option there is seems to be a nice feature designed exactly for your needs called Preserving stashes for Use with Restarted Stages:
Normally, when you run the stash step in your Pipeline, the resulting
stash of artifacts is cleared when the Pipeline completes, regardless
of the result of the Pipeline. Since stash artifacts aren’t accessible
outside of the Pipeline run that created them, this has not created
any limitations on usage. But with Declarative stage restarting, you
may want to be able to unstash artifacts from a stage which ran before
the stage you’re restarting from.
To enable this, there is a job property that allows you to configure a
maximum number of completed runs whose stash artifacts should be
preserved for reuse in a restarted run. You can specify anywhere from
1 to 50 as the number of runs to preserve.
This job property can be configured in your Declarative Pipeline’s options section, as below:
options {
preserveStashes()
// or
preserveStashes(buildCount: 5)
}
This built in feature is exactly what you need to solve your issue without any special modifications to your code, as it will allow you to rerun the pipeline from any stage and still use the existing file that were previously stashed.
Original Answer:
You can actually achieve this quite simply using the scripted syntax for the parallel command, and it will also allow you to avoid the duplicate code in the parallel stages.
parallel: Execute in parallel
Takes a map from branch names to closures and an optional argument failFast which will terminate all branches upon a failure in any other branch:
parallel firstBranch: {
// do something
}, secondBranch: {
// do something else
},
failFast: true|false
In your case it can look like:
stage('E2E-PR-CYPRESS') {
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
steps {
script {
stash name: 'cypress-dir', includes: 'cypress/**/*'
// Define the parallel execution stages
def stages = ['Cypress Tests 1', 'Cypress Tests 2']
// Create the parallel executions and run them
parallel stages.collectEntries {
["Running ${it}": {
node('aws_micro_slave_e2e') {
skipDefaultCheckout()
runE2eTests()
}
}]
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
}
This way you can easily add more parallel steps by updating the stages list, or even receive it as an input parameter. In addition you can create the parallel executions by different labels or tests suits, instead of the stage name.
You can add a Prepare stage at the top like this:
stages{
stage('Preperation'){
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
steps {
script {
stash name: 'cypress-dir', includes: 'cypress/**/*'
}
}
}
stage('E2E-PR-CYPRESS') {
parallel {
stage('Cypress Tests 1') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
stage('Cypress Tests 2') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
An out of the box concept
Propose splitting the job into 2 parts taking the following into consideration:
Currently use an EC2 plugin, as the current agents are EC2
Running the parallel stages with the same stashed content ready to unstash
Create jenkins pipeline job 1:
This job will checkout the workspace with any type of agent
Create a packer json to create a customised AMI for the EC2
The customised AMI will stash the contents and move to a directory that will appear on the EC2 when the agent is built
Output the AMI ID, run a groovy job to update the EC2 plugin AMI ID with the customised AMI ID to temporarily set the AMI in memory on Jenkins
pipeline {
agent {
docker {
test-container
}
}
options {
buildDiscarder(
logRotator(
numToKeepStr: '10',
artifactNumToKeepStr: '10'
)
)
ansiColor('xterm')
gitConnection("git")
}
stages {
stage('Run Stash Cypress Functional Test') {
steps {
dir('functional-test') {
// develop branch is canary build, all other branches are stable builds
script {
sh """
# script to stash cypress tests
"""
}
}
}
}
stage('Functional Test AMI Build') {
steps {
dir('functional-test/packer') {
withAWS(role: 'PackerBuild', roleAccount: '123456789012', roleSessionName: 'Jenkins-Workflow-FunctionalTest-Packer') {
script {
sh """
# packer json script will require to copy contents from workspace, run the script to stash content
# packer json script will require to capture new AMI ID
# https://discuss.devopscube.com/t/how-to-get-the-ami-id-after-a-packer-build/36
# https://www.packer.io/docs/post-processors/manifest
packer validate FunctionalTestPacker.json
packer build -debug FunctionalTestPacker.json
# grab AMI ID and export as jenkins env variable
"""
}
}
}
}
}
stage('run groovy script to update AMI ID on EC2 plugin') {
steps {
dir(groovy job dir) {
script {
sh """
# run groovy job to update AMI on Jenkins EC2 plugin
# https://gist.github.com/vrivellino/97954495938e38421ba4504049fd44ea
"""
}
}
}
}
stage('Kickoff Functional Test Deploy') {
// pipeline checkbox parameter, when ticked it will automatically kick off the functional test pipeline
when {
expression {params.RUN_TESTS.toBoolean()}
}
steps {
script{
env.branch = params.BRANCH
sh """
echo "Branch is ${branch}"
"""
}
build job: 'workflow/CypressFunctionaTestDeployAndRun',
parameters: [
string(name: 'BRANCH', value: env.branch)
],
wait : false
}
}
}
post {
always {
cleanWs()
}
}
}
Create jenkins pipeline job 2:
This job will create the EC2 agents via the plugin from the customised AMI from pipeline job 1
This means your agents will have the same workspace ready to unstash - so you can execute a parallel run
Also you could move a lot of your user data script that is in the EC2 plugin as part of the customised AMI build, thus cut down the time for each EC2 agent to get ready to carry out execution
pipeline {
stages {
stage('E2E-PR-CYPRESS') {
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
}
parallel {
stage('Cypress Tests 1') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
stage('Cypress Tests 2') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
}

Run postbuild action for every test on specific jenkins node

I have a remote agent and multiple local agents on my jenkins server. I have a script that I want to run only after those test that are builded on the remote agent. Is it possible somehow?
Thanks
Using jenkins pipeline you have the possiblity of running actions according to the result of the build. Take a look here:
https://jenkins.io/doc/book/pipeline/syntax/#post
You can even separate your build in "stages" and run actions according to the result of the stage using the same method.
On the entire pipeline, or on a specific stage, or even on your post actions, you can choose which node does the job.
Considering you run a stage on a specific node you could:
pipeline {
stages {
stage ('Build') {
agent { label "SLAVE1" }
steps {
// Stuff to do
}
post {
always {
// stuff
}
}
}
}
}
Or at end end of your pipeline in a post block:
pipeline {
stages {
stage ("Build") {
agent { label "SLAVE" }
steps {
// stuff
}
}
}
post {
// Or failure, unstable, success...
always {
node('SLAVE1'){
// stuff
}
}
}
}

Jenkins. Use a shared library on the options phase

So I have created a shared library in jenkins with a listener that gets triggered each time the pipelines reads a FlowNode so I can run groovy code before and after each stage, step, etc...
I'm able to call the shared library in a step phase like this:
pipeline {
agent any
stages {
stage('prepare') {
steps{
prepareStepsWrapper()
}
}
stage('step1') {
steps {
echo 'step1'
}
}
stage('step2') {
steps {
echo 'step2'
}
}
stage('step3') {
steps {
echo 'step3'
// fail on purpose
sh 'notfoundexecutablelol'
}
}
stage('step4') {
steps {
echo 'step4'
}
}
}
post{
always{
println env.getEnvironment()
}
}
}
And works pretty great!
With this approach the 'prepare' stage needs to be filtered out so I've switched to the options directive:
pipeline {
agent any
options {
prepareStepsWrapper()
}
stages {
stage('step1') {
steps {
echo 'step1'
}
}
...
}
}
But the pipeline fails with
WorkflowScript: 4: Invalid option type "prepareStepsWrapper"
tl;dr; How can I load a shared library within the options directive?
What does the option-stage do?
The options directive allows configuring Pipeline-specific options
from within the Pipeline itself.
You can't call the shared-library in the options-stage. This stage should not be used for execute any logic, rather it sets configurations for the pipeline. All availables options and the documentation can be found here.
You could try to create a stage that simply calls your prepareStepsWrapper() and use locks to avoid that other stages are executed before this stage.

Re-use agent in parallel stages of declarative pipeline

I'm using Declarative Pipelines 1.3.2 plugin and I want to use the same agent (as in only specifying the agent directive once) in multiple parallel stages:
stage('Parallel Deployment')
{
agent { dockerfile { label 'docker'; filename 'Dockerfile'; } }
parallel
{
stage('A') { steps { ... } }
stage('B') { steps { ... } }
}
}
However, Jenkins complains:
"agent" is not allowed in stage "Parallel Deployment" as it contains parallel stages
A solution is to duplicate the agent directive for each parallel stage, but this is tedious and leads to lot of duplicated code with many parallel stages:
stage('Parallel Deployment')
{
parallel
{
stage('A') {
agent { dockerfile { label 'docker'; filename 'Dockerfile'; } }
steps { ... }
}
stage('B') {
agent { dockerfile { label 'docker'; filename 'Dockerfile'; } }
steps { ... }
}
}
}
Is there a more idiomatic solution, or is duplicating agent directive necessary for each of the parallel stages?
Specifying the agent at pipeline level can be a solution, but has the potential downside that the agent is up & running for the whole duration of the build.
Also note that this means each stage (that doesn't define its own agent) is run on the same agent instance, not agent type. If the parallel processes are CPU / resource intensive, this might not be what you want.
Still, if you want to run parallel stages on one instance, and can't or want not to define the agent at pipeline level, here's a workaround for the declarative syntax:
stage('Parallel Deployment') {
agent { dockerfile { label 'docker'; filename 'Dockerfile'; } }
stages {
stage('A & B') {
parallel {
stage('A') { steps { ... } }
stage('B') { steps { ... } }
}
}
}
}
Or you go for a scripted pipeline, which doesn't have this limitation.
Declare the agent at Pipeline level so all stages run on the same agent.

How can I declare multiple agents in my Jenkins file and then refer to them in subsequent stages?

I'm trying to run a jenkins file with multiple agents in it, but I'm running into errors. Here's a snippet of my jenkins file:
pipeline {
agent {
docker {
label 'agentAAA'
...
}
node {
label 'agentBBB'
...
}
}
...
stages {
stage('to run on AAA') {
agent {
label 'agentAAA'
}
...
}
stage('to run on BBB') {
agent {
label 'agentBBB'
}
...
}
stage('to run on BBB') {
agent {
label 'agentBBB'
}
...
}
I'm getting these errors:
Only one agent type is allowed per agent section
No agent type specified. Must be one of [any, docker, dockerfile, label, none]
I can't find any examples in the documentation of how to refer to a previously declared agent. I see how to declare the agent in each individual stage, but I'd end up with many repeated declarations in my file.
You need to specify agent as none for overall pipeline, then you can specify agent for each stage explicitly as shown in below example. Populate the details as and what required.
pipeline {
agent none
stages {
stage ('Stage-1') {
agent { label 'agent-1' }
steps {
script {
}
}
}
stage ('Stage-2') {
agent { label 'agent-2' }
steps {
script {
}
}
}
}
}
Refer link for further details - https://jenkins.io/doc/book/pipeline/jenkinsfile/#using-multiple-agents

Resources