My jenkins is located at myjenkins:8080/
After I run my Job it generates one file and I would like to get the complete path of it to give it to the user.
Example:
myjenkins:8080/job/FUTURE/job/GetFullPath/2/execution/node/3/ws/
I want to give this path to the user and the user would see the file generated there.
pipeline {
agent { label env.Machine}
stages {
stage('PREPARE'){
steps{
script{
env.custom_stage_name = "PREPARE"
bat '%FOLDER%\\CreateFile.bat'
}
}
}
stage('BUILD'){
steps{
ws("c:\\jenkins\\workspace\\Test") {
bat 'xcopy ' + '%FOLDER%\\File.txt ' + "c:\\jenkins\\workspace\\Test /I /s /e /h /y"
}
}
}
}
}
Based on your title it's not clear to me if you just need a way to get the current working directory or you want to expose the directory so you can access it using a browser.
If you meant the second case: First at all, I think this is not possible without using a workaround. And there are some problems you need to work around!
First of all let me show you how to get the path to the workspace (if it is no obvious)
You can get the Path to your workspace by using the variable ${env.WORKSPACE}
Example:
pipeline {
agent any
stages {
stage('Hello') {
steps {
echo "${env.WORKSPACE}"
}
}
}
}
Concatenate the file:
def my_file = "${env.WORKSPACE}/my.file"
To your actual problem
Unless your filename differs every run, it will get overwritten. There is no guarantee that the file in your workspace folder is preserved. To keep it you should tell Jenkins to archive (see point 2) your artifacts.
I can't image any good reason to expose your workspace like you want to do.
So, let me give you some alternative first examples:
You want to give or sent Jenkins's log-files to someone: You can do this using the email-ext plugin which allows attaching the log.
This is a neat way to get information about the build state and sending out the log to people.
You want to save your build artifacts - and I think this is what you actually want.
In this case you should archive them. See jenkins-docs on how to do this.
If the "archive artifacts" also do not fit your needs: You can use a separate Freestyle project. A Freestyle projects allows viewing and downloading (as a zip-file) its workspace content. But: This is by far not the best solution!
Basically you copy your files from your pipeline project to this freestyle project. You can access it using the freestyle project's URL.
If you really want to expose the workspace using an URL
This involves a separate webserver, since I can't think of a way on how to tell Jenkins to expose it's working directories as a webservice.
If you just want to gain access to your workspace folder, you can expose it using a separate webserver. The ideas is to run a very simple web-server to serve the current workspace directory.
Here is one example on how to deploy exactly this using python.
I need to get GitVersion.exe variables in my Jenkins pipeline.
The GitVersion documentation gives a hint on how to do that. Essentially call gitversion /output buildserver.
This call does add the variables to the current step and they are lost once the step completes. I can show this call executes when combining a set command in the same bat execution. The second set shows the variables are gone from the environment.
bat 'nuget install GitVersion.CommandLine -OutputDirectory c:/packages -Version 3.6.5'
bat 'c:/packages/GitVersion.CommandLine.3.6.5/tools/GitVersion.exe /output buildserver && set'
bat 'set'
The documentation of GitVersion is aware of that and suggests to use EnvInject.
Installing the plugin and executing the same pipeline did not change the result. I read that the Plugin is not made for pipelines so that may have something to do with it.
Pipelines support a syntax for environment.
Following that syntax I can set static variables at the top of my pipeline like this:
environment {
ASuperVariable = 'MySuperVariable'
}
What I need is combining those calls so that I can add run time variables to the Jenkinsfile pupeline.
environment {
bat 'gitversion /output buildserver'
}
Now obviously the above call is not even syntax correct. Is there a way to mark a section so that the contained environment changes are available for other steps?
EDIT:
This is still unsolved. At the moment I need to create a batch script and pass the tool into it as an argument. Inside the batch I can call the tool to add to the environment of the batch script and use that wile the batch is running. A Multi line batch in the Jenkins file could be a solution if the process remains the same over all the multiple lines.
Not sure whether you would be able to use scripted pipeline or at least a script block inside declarative. It'd be quite easy doing so:
withEnv(['ASuperVariable=MySuperVariable']) {
echo env.ASuperVariable
}
Or when calling a windows cmd script:
node('win') {
withEnv(['ASuperVariable=MySuperVariable']) {
bat 'echo %ASuperVariable%'
}
}
At this moment we use JJB to compile Jenkins jobs (mostly pipelines already) in order to configure about 700 jobs but JJB2 seems not to scale well to build pipelines and I am looking for a way to drop it from the equation.
Mainly i would like to be able to have all these pipelines stored in a single centralized repository.
Please note that keeping the CI config (Jenkinsfile) inside each repository and branch is not possible in our use case, we need to keep all pipelines in a single "jenkins-jobs.git" repo.
As far as I know this is not possible yet, but in progress. See: https://issues.jenkins-ci.org/browse/JENKINS-43749
I think this is the purpose of jenkins shared libraries
I didn't dev such library my-self but I am using some. Basically:
Develop the "shared code" of the jenkins pipeline in a shared library
it can contains the whole pipeline (seq of steps)
Add this library to the jenkins server
In each project, add a jenkinsfile that "import" those using #Library
as #Juh_ said, you can use jenkins shared libraries, here is a complete steps, Suppose that we have three branches:
master
develop
stage
and we want to create a single Jenkins file so that we can change in only one place. All you need is creating a new branch ex: common. This branch MUST have this structure. What we are interested for now is adding a new groovy file in vars directory, ex: common.groovy. Here we can put the common Jenkins file that you wish to be used across all branches.
Here is a sample:
def call() {
node {
stage("Install Stage from common file") {
if (env.BRANCH_NAME.equals('master')){
echo "npm install from common files master branch"
}
else if(env.BRANCH_NAME.equals('develop')){
echo "npm install from common files develop branch"
}
}
stage("Test") {
echo "npm test from common files"
}
}
}
You must wrap your code call function in order to be used in other branches. now we have finished work in common branch we need to use it in our branches. go to any branch you wish to use this pipline ex: master and create Jenkinsfile and put this one line of code:
common()
This will call the common function that you have created before in common branch and will execute the pipeline.
I have an open-source project, that resides in GitHub and is built using a build farm, controlled by Jenkins.
I want to build it branch-wise using a pipeline, but I don't want to store Jenkinsfile inside the code. Is there a way to accomplish this?
I have encountered the same issue as you. While the idea of having the build process as part of the code is good, there is information that the Jenkinsfile would include that are not intrinsic to the project build itself, but rather are specific to the build environment instance, which may change.
The way I accomplished this is:
Encapsulate the core build process in a single script (build.py or build.sh). This may call specific build tools like Make, CMake, Ant, etc.
Tell Jenkins via the Jenkinsfile to call a function defined in a single global library
Define the global Jenkins build function to call the build script (e.g. build.py) with appropriate environment settings. For example, using custom tools and setting up the PATH.
So for step 2, create a Jenkinsfile in your project containing just the line
build_PROJECTNAME()
where PROJECTNAME is based on the name of your project.
Then use the Pipeline Shared Groovy Libraries Plugin and create a Groovy script in the shared library repository called vars/build_PROJECTNAME.groovy containing the code that sets up the environment and calls the project build script (e.g. build.py):
def call() {
node('linux') {
stage("checkout") {
checkout scm
}
stage("build") {
withEnv([
"PATH+CMAKE=${tool 'CMake'}/bin",
"PATH+PYTHON=${tool 'Python-3'}",
"PATH+NINJA=${tool 'Ninja'}",
]) {
execute 'python build.py'
}
}
}
}
First of all, why do you not want a Jenkinsfile in your code? The pipeline is just as much part of the code as would be your build file.
Other then that, you can load groovy files to be evaluated as a pipeline script. You can do this either from a different location with the from SCM option and then checkout the actual code. But this will force you to manually take care of the branch builds.
Another option would be to have a very basic Jenkinsfile that merely checkouts an external pipeline.
You would get something like this:
node{
deleteDir()
git env.flowScm
def flow = load 'pipeline.groovy'
stash includes: '**', name: 'flowFiles'
stage 'Checkout'
checkout scm // short hand for checking out the "from scm repository"
flow.runFlow()
}
Where the pipeline.groovy file would contain the actual pipeline would look like this:
def runFlow() {
// your pipeline code
}
// Has to exit with 'return this;' in order to be used as library
return this;
When writing jenkins pipelines it seems to be very inconvenient to commit each new change in order to see if it works.
Is there a way to execute these locally without committing the code?
You cannot execute a Pipeline script locally, since its whole purpose is to script Jenkins. (Which is one reason why it is best to keep your Jenkinsfile short and limited to code which actually deals with Jenkins features; your actual build logic should be handled with external processes or build tools which you invoke via a one-line sh or bat step.)
If you want to test a change to Jenkinsfile live but without committing it, use the Replay feature added in 1.14.
JENKINS-33925 tracks the feature request for an automated test framework.
I have a solution that works well for me. It consists of a local jenkins running in docker and a git web hook to trigger the pipeline in the local jenkins on every commit. You no longer need to push to your github or bitbucket repository to test the pipeline.
This has only been tested in a linux environment.
It is fairly simple to make this work although this instruction is a tad long. Most steps are there.
This is what you need
Docker installed and working. This is not part of this instruction.
A Jenkins running in docker locally. Explained how below.
The proper rights (ssh access key) for your local Jenkins docker user to pull from your local git repo. Explained how below.
A Jenkins pipeline project that pulls from your local git repository. Explained below.
A git user in your local Jenkins with minimal rights. Explained below.
A git project with a post-commit web hook that triggers the pipeline project. Explained below.
This is how you do it
Jenkins Docker
Create a file called Dockerfile in place of your choosing. I'm placing it in /opt/docker/jenkins/Dockerfile fill it with this:
FROM jenkins/jenkins:lts
USER root
RUN apt-get -y update && apt-get -y upgrade
# Your needed installations goes here
USER jenkins
Build the local_jenkins image
This you will need to do only once or after you have added something to the Dockerfile.
$ docker build -t local_jenkins /opt/docker/jenkins/
Start and restart local_jenkins
From time to time you want to start and restart jenkins easily. E.g. after a reboot of your machine. For this I made an alias that I put in .bash_aliases in my home folder.
$ echo "alias localjenkinsrestart='docker stop jenkins;docker rm jenkins;docker run --name jenkins -i -d -p 8787:8080 -p 50000:50000 -v /opt/docker/jenkins/jenkins_home:/var/jenkins_home:rw local_jenkins'" >> ~/.bash_aliases
$ source .bash_aliases # To make it work
Make sure the /opt/docker/jenkins/jenkins_home folder exists and that you have user read and write rights to it.
To start or restart your jenkins just type:
$ localjenkinsrestart
Everything you do in your local jenkins will be stored in the folder /opt/docker/jenkins/jenkins_home and preserved between restarts.
Create a ssh access key in your docker jenkins
This is a very important part for this to work. First we start the docker container and create a bash shell to it:
$ localjenkinsrestart
$ docker exec -it jenkins /bin/bash
You have now entered into the docker container, this you can see by something like jenkins#e7b23bad10aa:/$ in your terminal. The hash after the # will for sure differ.
Create the key
jenkins#e7b23bad10aa:/$ ssh-keygen
Press enter on all questions until you get the prompt back
Copy the key to your computer. From within the docker container your computer is 172.17.0.1 should you wonder.
jenkins#e7b23bad10aa:/$ ssh-copy-id user#172.17.0.1
user = your username and 172.17.0.1 is the ip address to your computer from within the docker container.
You will have to type your password at this point.
Now lets try to complete the loop by ssh-ing to your computer from within the docker container.
jenkins#e7b23bad10aa:/$ ssh user#172.17.0.1
This time you should not need to enter you password. If you do, something went wrong and you have to try again.
You will now be in your computers home folder. Try ls and have a look.
Do not stop here since we have a chain of ssh shells that we need to get out of.
$ exit
jenkins#e7b23bad10aa:/$ exit
Right! Now we are back and ready to continue.
Install your Jenkins
You will find your local Jenkins in your browser at http://localhost:8787.
First time you point your browser to your local Jenkins your will be greated with a Installation Wizard.
Defaults are fine, do make sure you install the pipeline plugin during the setup though.
Setup your jenkins
It is very important that you activate matrix based security on http://localhost:8787/configureSecurity and give yourself all rights by adding yourself to the matrix and tick all the boxes. (There is a tick-all-boxes icon on the far right)
Select Jenkins’ own user database as the Security Realm
Select Matrix-based security in the Authorization section
Write your username in the field User/group to add: and click on the [ Add ] button
In the table above your username should pop up with a people icon next to it. If it is crossed over you typed your username incorrectly.
Go to the far right of the table and click on the tick-all-button or manually tick all the boxes in your row.
Please verify that the checkbox Prevent Cross Site Request Forgery exploits is unchecked. (Since this Jenkins is only reachable from your computer this isn't such a big deal)
Click on [ Save ] and log out of Jenkins and in again just to make sure it works.
If it doesn't you have to start over from the beginning and emptying the /opt/docker/jenkins/jenkins_home folder before restarting
Add the git user
We need to allow our git hook to login to our local Jenkins with minimal rights. Just to see and build jobs is sufficient. Therefore we create a user called git with password login.
Direct your browser to http://localhost:8787/securityRealm/addUser and add git as username and login as password.
Click on [ Create User ].
Add the rights to the git user
Go to the http://localhost:8787/configureSecurity page in your browser. Add the git user to the matrix:
Write git in the field User/group to add: and click on [ Add ]
Now it is time to check the boxes for minimal rights to the git user. Only these are needed:
overall:read
job:build
job:discover
job:read
Make sure that the Prevent Cross Site Request Forgery exploits checkbox is unchecked and click on [ Save ]
Create the pipeline project
We assume we have the username user and our git enabled project with the Jenkinsfile in it is called project and is located at /home/user/projects/project
In your http://localhost:8787 Jenkins add a new pipeline project. I named it hookpipeline for reference.
Click on New Item in the Jenkins menu
Name the project hookpipeline
Click on Pipeline
Click [ OK ]
Tick the checkbox Poll SCM in the Build Triggers section. Leave the Schedule empty.
In the Pipeline section:
select Pipeline script from SCM
in the Repository URL field enter user#172.17.0.1:projects/project/.git
in the Script Path field enter Jenkinsfile
Save the hookpipeline project
Build the hookpipeline manually once, this is needed for the Poll SCM to start working.
Create the git hook
Go to the /home/user/projects/project/.git/hooks folder and create a file called post-commit that contains this:
#!/bin/sh
BRANCHNAME=$(git rev-parse --abbrev-ref HEAD)
MASTERBRANCH='master'
curl -XPOST -u git:login http://localhost:8787/job/hookpipeline/build
echo "Build triggered successfully on branch: $BRANCHNAME"
Make this file executable:
$ chmod +x /home/user/projects/project/.git/hooks/post-commit
Test the post-commit hook:
$ /home/user/projects/project/.git/hooks/post-commit
Check in Jenkins if your hookpipeline project was triggered.
Finally make some arbitrary change to your project, add the changes and do a commit. This will now trigger the pipeline in your local Jenkins.
Happy Days!
TL;DR
Jenkins Pipeline Unit testing framework
Jenkinsfile Runner
Long Version
Jenkins Pipeline testing becomes more and more of a pain. Unlike the classic declarative job configuration approach where the user was limited to what the UI exposed the new Jenkins Pipeline is a full fledged programming language for the build process where you mix the declarative part with your own code. As good developers we want to have some unit tests for this kind of code as well.
There are three steps you should follow when developing Jenkins Pipelines. The step 1. should cover 80% of the uses cases.
Do as much as possible in build scripts (eg. Maven, Gradle, Gulp etc.). Then in your pipeline scripts just calls the build tasks in the right order. The build pipeline just orchestrates and executes the build tasks but does not have any major logic that needs a special testing.
If the previous rule can't be fully applied then move over to Pipeline Shared libraries where you can develop and test custom logic on its own and integrate them into the pipeline.
If all of the above fails you, you can try one of those libraries that came up recently (March-2017). Jenkins Pipeline Unit testing framework or pipelineUnit (examples). Since 2018 there is also Jenkinsfile Runner, a package to execution Jenkins pipelines from a command line tool.
Examples
The pipelineUnit GitHub repo contains some Spock examples on how to use Jenkins Pipeline Unit testing framework
Jenkins has a 'Replay' feature, which enables you to quickly replay a job without updating sources:
At the moment of writing (end of July 2017) with the Blue Ocean plugin you can check the syntax of a declarative pipeline directly in the visual pipeline editor. The editor, works from the Blue Ocean UI when you click "configure" only for github projects (this is a known issue and they are working to make it work also on git etc).
But, as explained in this question you can open the editor browsing to:
[Jenkins URL]/blue/organizations/jenkins/pipeline-editor/
Then click in the middle of the page, and press Ctrl+S, this will open a textarea where you can paste a pipeline declarative script. When you click on Update, if there is a syntax error, the editor will let you know where the syntax error is. Like in this screenshot:
If there is no syntax error, the textarea will close and the page will visualize your pipeline. Don't worry it won't save anything (if it's a github project it would commit the Jenkinsfile change).
I'm new to Jenkins and this is quite helpful, without this I had to commit a Jenkinsfile many times, till it works (very annoying!). Hope this helps. Cheers.
A bit late to the party, but that's why I wrote jenny, a small reimplementation of some core Jenkinsfile steps. (https://github.com/bmustiata/jenny)
In my development setup – missing a proper Groovy editor – a great deal of Jenkinsfile issues originates from simple syntax errors. To tackle this issue, you can validate the Jenkinsfile against your Jenkins instance (running at $JENKINS_HTTP_URL):
curl -X POST -H $(curl '$JENKINS_HTTP_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)') -F "jenkinsfile=<Jenkinsfile" $JENKINS_HTTP_URL/pipeline-model-converter/validate
The above command is a slightly modified version from
https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Validating-(or-linting)-a-Declarative-Jenkinsfile-from-the-command-line
As far as i know this Pipeline Plugin is the "Engine" of the new Jenkinsfile mechanics, so im quite positive you could use this to locally test your scripts.
Im not sure if there is any additional steps needed when you copy it into a Jenkinsfile, however the syntax etc should be exactly the same.
Edit: Found the reference on the "engine", check this feature description, last paragraph, first entry.
For simplicity, you can create a Jenkinsfile at the root of the git repository, similar to the below example 'Jenkinsfile' based on the groovy syntax of the declarative pipeline.
pipeline {
agent any
stages {
stage('Build the Project') {
steps {
git 'https://github.com/jaikrgupta/CarthageAPI-1.0.git'
echo pwd()
sh 'ls -alrt'
sh 'pip install -r requirements.txt'
sh 'python app.py &'
echo "Build stage gets finished here"
}
}
stage('Test') {
steps {
sh 'chmod 777 ./scripts/test-script.sh'
sh './scripts/test-script.sh'
sh 'cat ./test-reports/test_script.log'
echo "Test stage gets finished here"
}
}
}
https://github.com/jaikrgupta/CarthageAPI-1.0.git
You can now set up a new item in Jenkins as a Pipeline job.
Select the Definition as Pipeline script from SCM and Git for the SCM option.
Paste the project's git repo link in the Repository URL and Jenkinsfile in the script name box.
Then click on the lightweight checkout option and save the project.
So whenever you pushed a commit to the git repo, you can always test the changes running the Build Now every time in Jenkins.
Please follow the instructions in the below visuals for easy setup a Jenkins Pipeline's job.
Aside from the Replay feature that others already mentioned (ditto on its usefulness!), I found the following to be useful as well:
Create a test Pipeline job where you can type in Pipeline code or point to your repo/branch of a Jenkinsfile to quickly test out something. For more accurate testing, use a Multibranch Pipeline that points to your own fork where you can quickly make changes and commit without affecting prod. Stuff like BRANCH_NAME env is only available in Multibranch.
Since Jenkinsfile is Groovy code, simply invoke it with "groovy Jenkinsfile" to validate basic syntax.
Put your SSH key into your Jenkins profile, then use the declarative linter as follows:
ssh jenkins.hostname.here declarative-linter < Jenkinsfile
This will do a static analysis on your Jenkinsfile. In the editor of your choice, define a keyboard shortcut that runs that command automatically. In Visual Studio Code, which is what I use, go to Tasks > Configure Tasks, then use the following JSON to create a Validate Jenkinsfile command:
{
"version": "2.0.0",
"tasks": [
{
"label": "Validate Jenkinsfile",
"type": "shell",
"command": "ssh jenkins.hostname declarative-linter < ${file}"
}
]
}
You can just validate your pipeline to find out syntax issues. Jenkins has nice API for Jenkisfile validation - https://jenkins_url/pipeline-model-converter/validate
Using curl and passing your .Jenkinsfile, you will get syntax check instantly
curl --user username:password -X POST -F "jenkinsfile=<jenkinsfile" https://jenkins_url/pipeline-model-converter/validate
You can add this workflow to editors:
VS Code
Sublime Text
Using the VS Code Jenkins Jack extension, you can have a way to test your Jenkinsfiles without use the git push way, from your local files to a local or remote running Jenkins. And you will have the running log of the job inside VS Code, the ability to create jobs in Jenkins and more staff. I hope this help to more people looking for a way to develop Jenkinsfiles.
i am using replay future , to do some update and run quickly .
With some limitations and for scripted pipelines I use this solution:
Pipeline job with an inlined groovy script:
node('master') {
stage('Run!') {
def script = load('...you job file...')
}
}
Jenkinsfile for testing have same structure as for lesfurets:
def execute() {
... main job code here ...
}
execute()
This is a short solution that lets me test Pipeline code very quickly:
pipeline {
agent any
options {
skipDefaultCheckout true
timestamps()
}
parameters {
text(name: 'SCRIPT', defaultValue: params.SCRIPT,
description: 'Groovy script')
}
stages {
stage("main") {
steps {
script {
writeFile file: 'script.groovy',
text: params.SCRIPT
def groovyScript = load 'script.groovy'
echo "Return value: " + groovyScript
}
} // steps
} // stage
} // stages
} // pipeline
skipDefaultCheckout true because we do not need the files in this tool git repository.
defaultValue: params.SCRIPT sets the default to the latest execution. If used by just one user, it allows for a very quick cycle of short tests.
The given script is written to a file, and loaded and with load.
With this setup I can test everything I can do inside my other Jenkinsfiles, including using shared libraries.