I have a step in a Jenkins pipeline to copy some source files to the workspace.
stage('Copy Files') {
script {
echo 'Staging files'
sh "cp -ar /home/dev/src/ ${env.WORKSPACE}"
}
}
Yet, when I rerun the build it uses the old code. The only solution is to delete the workspace prior to the copy. In a normal Linux file system a copy will overwrite the destination. Why does Jenkins behave differently--i.e., old files are not overwritten? From the syntax it seems like it is just running a shell command, so why does this not have the expected behavior?
It is because, Jenkins run on master node and workspace will be on the worker node.
when checkout scm and sh "" code blocks are in different stages, files will not be save from first stage to other. You should use stash & unstash. when you stash a directory path, files in that dir will be available to the unstashed step in later stages.
Jenkins doc - here
Related
I am trying to find out where Jenkins clone the GIT repo but I can't find it.
In my script in Jenkins pipeline I have:
stage('Do something'){
steps {
sh '''
#!/bin/bash
YAML_FILE="${JENKINS_HOME}/jobs/<JENKINS_JOB_NAME>/workspace/GIT_REPO_NAME/path/to/file.yaml"
...
However when I am trying to buid it, it can't find it in this path:
Error: open /efsmnt/jobs/<JENKINS_JOB_NAME>/workspace/GIT_REPO_NAME/path/to/file.yaml: no such file or directory
I also found that it could be in:
20:13:55 Error: open /home/jenkins/agent/workspace/<JENKINS_JOB_NAME>/GIT_REPO_NAME/path/to/file.yaml: no such file or directory
but with no luck.
The global environment WORKSPACE represents the job workspace where your repo will be clone into.
And Jenkins won't create GIT_REPO_NAME folder in job workspace when run git clone.
You can try below script
sh '''
pwd
ls -l
YAML_FILE="${WORKSPACE}/path/to/file.yaml"
'''
Use double quote in case job workspace path includes space when Jenkins job or Jenkins folder name includes space
I have some windows slave at my Jenkins so I need to copy file to them in pipeline. I heard about Copy To Slave and Copy Artifact plugins, but they doesn't have pipeline syntax manual. So I don't know how to use them in pipeline.
Direct copy doesn't work.
def inputFile = input message: 'Upload file', parameters: [file(name: 'parameters.xml')]
new hudson.FilePath(new File("${ENV:WORKSPACE}\\parameters.xml")).copyFrom(inputFile)
This code returns and error:
Caused: java.io.IOException: Failed to copy /var/lib/jenkins/jobs/_dev/jobs/(TEST)job/builds/107/parameters.xml to d:\Jenkins\workspace\_dev\(TEST)job\parameters.xml
Is there any way to copy file from master to slave in Jenkins Pipeline?
As I understand copyFrom is executed on your Windows node, therefore the source path is not accessible.
I think you want to look into the stash/unstash steps (Jenkins Pipeline: Basic Steps), which work across different nodes. Also this example might be helpful.
Pipeline DSL context runs on master node even that your write node('someAgentName') in your pipeline.
Try to use stash/unstash, but it is bad for large files.
Try External Workspace Manager Plugin. It has
pipelines steps and good for large files.
Try to use an intermediate storage. archive() and sh("wget $url") will be helpful.
If the requirement is to copy an executable to the test slave and to publish the test results, this is easy to do without the Copy to Slave plugin.
A shared folder should be created on each test slave (normal Windows shared folder).
After build: Build script copies the executable to the shared directory on each slave. A simple batch script using copy command is sufficient for this.
stage ('Copy to slaves') {
steps {
bat 'call "copy-to-slave.bat"'
}
}
During test: The test script copies the executable to another directory and runs it.
After test: Post-build action "Publish Robot Framework test results" can be used to report the test results. It is not necessary to copy the test result files back to the master first.
I recommend on Pipeline: Phoenix AutoTest plugin
Jenkins plugin website:
https://plugins.jenkins.io/phoenix-autotest/#documentation
GitHub repository of plugin:
https://github.com/jenkinsci/phoenix-autotest-plugin
My parallel steps need access to a local file in the workspace of the job. But it seems they can access it?
I tried listing the workspace in each stream:
powershell "ls ${workspace}"
they are all empty! the output of each stream ls C:\workspace\branch_name#<stream#> shows no files.
How they get access to the workspace? they're pretty much useless to me if they can't even access local files.
Is there a feature to copy files from the main workspace to the stream workspaces?
Since Powershell support was recently introduced, and there is a powershell step (issue JENKINS-34581), do check you are in the same path each time:
node {
powershell '$(pwd).Path'
bat 'echo %cd%'
}
And that you have checked out a repository first (or your workspace would be empty anyway)
The way you handle this in Jenkins is to stash and unstash the files for each parallel job.
https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-stash-code-stash-some-files-to-be-used-later-in-the-build
I have a Jenkins pipeline A that looks something like this
Run prebuild tests
Build project
Run post-build tests
Run another pipeline B with parameters extracted from current build
I was wondering if there was a way to get test results from pipeline B and aggregate them with the tests results of pipeline A.
Currently, I have to open the console output and open the Url to the external build.
If the above is not possible, is it possible to display this Url somewhere else than the console (e.g. as an artifact ).
I believe what you are looking for is "stash". Below is copied directly from https://jenkins.io/doc/pipeline/examples/
Synopsis
This is a simple demonstration of how to unstash to a different directory than the root directory, so that you can make sure not to overwrite directories or files, etc.
// First we'll generate a text file in a subdirectory on one node and stash it.
stage "first step on first node"
// Run on a node with the "first-node" label.
node('first-node') {
// Make the output directory.
sh "mkdir -p output"
// Write a text file there.
writeFile file: "output/somefile", text: "Hey look, some text."
// Stash that directory and file.
// Note that the includes could be "output/", "output/*" as below, or even
// "output/**/*" - it all works out basically the same.
stash name: "first-stash", includes: "output/*"
}
// Next, we'll make a new directory on a second node, and unstash the original
// into that new directory, rather than into the root of the build.
stage "second step on second node"
// Run on a node with the "second-node" label.
node('second-node') {
// Run the unstash from within that directory!
dir("first-stash") {
unstash "first-stash"
}
// Look, no output directory under the root!
// pwd() outputs the current directory Pipeline is running in.
sh "ls -la ${pwd()}"
// And look, output directory is there under first-stash!
sh "ls -la ${pwd()}/first-stash"
}
Basically you can copy your artifacts, say .xml files that result from running unit tests, from the first job to the node running the second job. Then have the Unit test processor run on both the results from the first and the second job.
I have a jenkins pipeline that builds a java artifact,
copies it to a directory and then attempts to execute a external script.
I am using this syntax within the pipeline script to execute the external script
dir('/opt/script-directory') {
sh './run.sh'
}
The script is just a simple docker build script, but the build will fail
with this exception:
java.io.IOException: Failed to mkdirs: /opt/script-directory#tmp/durable-ae56483c
The error is confusing because the script does not create any directories. It is just building a docker image and placing the freshly built java artifact in that image.
If I create a different job in jenkins that executes the external script as
its only build step and then call that job from my pipeline script using this syntax:
build 'docker test build'
everything works fine, the script executes within the other job and the pipeline
continues as expected.
Is this the only way to execute a script that is external to the workspace?
What am I doing wrong with my attempt at executing the script from within
the pipeline script?
The issue is that the jenkins user (or whatever the user is that runs the Jenkins slave process) does not have write permission on /opt and the sh step wants to create the script-directory#tmp/durable-ae56483c sub-directory there.
Either remove the dir block and use the absolute path to the script:
sh '/opt/script-directory/run.sh'
or give write permission to jenkins user to folder /opt (not preferred for security reasons)
Looks like a bug in Jenkins, durable directories are meant to store recovery information e.g. before executing an external script using sh.
For now all you can do is make sure that /opt/script-directory has +r +w and +x set for jenkins user.
Another workaround would be not to change the current directory, just execute sh with it:
sh '/opt/script-directory/run.sh'
I had a similar concern when trying to execute a script in a Jenkins pipeline using a Jenkinsfile.
I was trying to run a script restart_rb.sh with sudo.
To run it I specified the present working directory ($PWD):
sh 'sudo sh $PWD/restart_rb.sh'