I am trying out the Jenkins Workflow plugin (https://github.com/jenkinsci/workflow-plugin) and using the Workflow Global Library (https://github.com/jenkinsci/workflow-plugin/tree/master/cps-global-lib) that comes with it.
I wanted to embed some script calls in my shared functions (bash/python). The obvious way to do this seems to be by using sh """...""". However this leads to some escaping being required ($ has to be escaped). Also its a bit messy to develop a script inside string quotes.
Is there a way to access resource files (e.g. a .sh or .py file) stored in the Global Library during a workflow execution?
So that I can do something like
sh getScript("script.sh")
''' syntax might be more helpful, since it reduces the need to escape. There are other Groovy syntaxes that are even better for long blocks, with various tradeoffs on the escape character.
But yes it would be helpful to be able to reference a static resource in the “classpath”. Feel free to file an RFE for this.
Related
We have our shared libraries on gitlab called mainlibrary and it has a lot of groovy files.
Example in mainlibrary gitlab repo we have the following files.
startup_pipeline.groovy
cleanup_pipeline.groovy
In one of our Jenkins job we need to include multiple groovy files in the Jenkinsfile. Is this possible?
This is how the Jenkinsfile looks like:
#Library('mainlibrary')_
startup_pipeline(email:'example#example.com')
Can I include the second groovy function file into this Jenkinsfile like this?
#Library('mainlibrary')_
startup_pipeline(email:'example#example.com'),
cleanup_pipeline(email:'example#example.com')
Considering the cleanup_pipeline.groovy is located under the vars folder and your code already has a complete declaration inside, your example may work, and the second file can be included. The only modification is the extra comma:
#Library('mainlibrary')_
startup_pipeline(email:'example#example.com')
cleanup_pipeline(email:'example#example.com')
Or can be under src and imported, but I never used this approach.
Usually, I keep the main logic inside vars, and other complex things go to src.
Based on Jenkins documentation, see more in Directory structure and Defining custom steps.
Video step by step building a shared library, you can build and test something similar if you are not sure about the structure.
Jenkins declarative pipeline is too powerful for us, often users can abuse it. We are thinking to use an opinionated YAML to describe CI/CD pipeline. And it seems there are two choices.
Write a plugin and consume YAML and dynamically create stage / steps.
Write a plugin to convert a YAML to Jenkins pipeline.
I am not expert on Jenkins, so I hope some expert can give some guidance and maybe an example.
using official plugin pipeline-as-yaml, but it has a fixed grammar.
using or customization wolox-ci
create your own shared libaray. However, they are easy from beginning but full grammer design is required when used widely. Here is a psudo code based on curry.
// create a file named yamlCompiler.groovy in shared library,
def call(str){
def rawMap = readYaml(text: str)
// consume yaml and get a lambda function
return {
stage{
steps.each{it ->
it."$type"(it)
}
}
}
}
Use yamlCompiler in your jenkinsfile code block.
#Library('your libs name')
def str =
'''
steps:
- type: sh
script: ls -la
- type: echo
message: xxx
'''
Closure closure = yamlCompiler(str)
closure.call()
I'm looking for a similar solution. We run hardened predefined pipelines for every project, but still want to allow dev teams to customise certain steps within the process —without allowing them the full power of a Jenkinsfile.
I'm also exploring the possibility of an —in your words— "opinionated YAML".
I've so far only found one example of such an implementation: Wolox-CI supports their own pre-defined build steps via YAML. You'll be able to see the steps they support here.
I'm thinking of parsing the YAML using Snake YAML. Here's an SO answer with an example on how to do it.
Two solutions:
create a shared library to abstract the actual pipeline and provide to your users some guidance on how to setup a shared library and a Jenkinsfile sample. Here is an example of embeded pipeline https://github.com/SAP/jenkins-library/blob/master/vars/piperPipeline.groovy
use another tool like https://drone.io/
If you're not an expert and don't want/have the time to become one, the second solution might be the best one.
Really? Is the only difference here when the plugin is executed?:
Write a plugin and consume YAML and dynamically create stage / steps.
Write a plugin to convert a YAML to Jenkins pipeline.
Forgive me, because I may be a little hardened, but abstracting a layer for the dynamic creation of a declarative, or scripted, Jenkinsfile written in the simple groovy lang syntax so that it can be pretty-printed in yml prevents users from updating your yml exactly how? It seems to me your abstraction only adds to the complexity with which you wish to implement usability.
One, all the current yml plugins for Jenkins do exactly that. Two, they don't actually have the full breadth of "features" (yes, I'm using that term loosely here) accessible by implementing the groovy/(java) classes already available in the Jenkins domain (referencing the DSL). Two solutions exist right now for this, and I've investigated both, and implemented both, extensively. One is wolox-ci, which is the better of the two, and the other is Pipeline-as-YAML. In my opinion, it's easy to use, but both lack the full breadth of implementation features simply using groovy provides. So why force it? Simply so your users can have a pretty-printed yml file, and not have to be concerned with simple syntax, which you claim hardens your infrastructure-as-code backend so that the same users can't screw it up? Sorry, I'm calling bull pucky on that assertion. What's to stop anyone from totally screwing up your builds by pushing a change to the yml file which breaks the integration with groovy, or worse, completely changes an algorithm you worked hard to customize?
Sorry, I just don't get it. Sure, making something more human readable is always a good thing. Doing it because of the reasons you've stipulated makes no sense, though. Also, unless you have a super simple defined algorithm in your CI/CD process, without any non-continuous-passing-style transform methods being implemented, then using the current iterations of the yml-as-Jenkinsfile-templates plugins is probably not the way you want to go.
Now, you could write your own plugin to do this, but what's the technical debt on that, versus just learning the groovy syntax? Also, it still doesn't prevent users from making code changes to your build infrastructure, then integrating those changes in a simple yml file.
Currently I am having one jenkinsfile which in process of development has grown to huge size. Mostly it contains some groovy methods used in pipeline, but I would like to place groovy scripts in separate files in other repository and than checkout every time when I need those scripts. Those works on a maps of settings which is commonly used in methods. How can I share those maps in script files? I can't use shared libraries because of security purposes.
An alternative to shared library is the load step, which loads and evaluates the given groovy script file.
settings.groovy:
[
answer: 42,
question: 'tbd'
]
main.groovy:
def settings = load 'settings.groovy'
echo "The answer is: ${settings.answer}"
Output:
The answer is: 42
Jenkins artifact URLs allow abstracting the "last successful build", so that instead of
http://myjenkins.local/job/MyJob/38/artifact/build/MyJob-v1.0.1.zip
we can say
http://myjenkins.local/job/MyJob/lastSuccessfulBuild/artifact/build/MyJob-v1.0.1.zip
Is it possible to abstract this further? My artifacts have their version number in their filename, which can change from build to build. Ideally I'd like to have a some kind of "alias" URL that looks like this:
http://myjenkins.local/job/MyJob/lastSuccessfulBuild/artifact/build/MyJob-latest.zip
MyJob-latest.zip would then resolve to MyJob-v1.0.1.zip.
If Jenkins itself can't do this, perhaps there's a plugin?
Never seen any such plugin, but Jenkins already has a similar functionality built-in.
You can use /*zip*/filename.zip in your artifact path, where filename is anything you choose. It will take all found artifacts, and download them in a zipfile (you may end up with a zip inside a zip, if your artifact is already a zip file)
In your case, it will be:
http://myjenkins.local/job/MyJob/lastSuccessfulBuild/artifact/build/*zip*/MyJob-latest.zip
This will get you the contents of /artifact/build/ returned in zipped archive with name MyJob-latest.zip. Note that if you have more than just that zip file in that directory, other files will be returned too.
You can use wildcards in the path. A single * for a regular wildcard, a double ** for skipping any number of preceding directories.
For example, to get any file that starts with MyJob, ends with .zip, and to look for it in any artifact directory, you could use:
/lastSuccessfulBuild/artifact/**/MyJob*.zip/*zip*/MyJob-latest.zip
Edit:
You cannot do something like this without some form of a container (a zip in this case). With the container, you are telling the system:
Get any possible [undetermined count] wildcard match and place into this container, then give me the container. This is logical and possible, as there is only one single container, whether it is empty or not.
But you cannot tell the system:
Give me a link to a specific single file, but I don't know which one or how many there are. The system can't guarantee that your wildcards will match one, more than one, or none. This is simply impossible from a logic perspective.
If you need it for some script automation, you can unzip the first level zip and be still left with your desired zipped artifact.
If you need to provide this link to someone else, you need an alternative solution.
Alternative 1:
After your build is complete, execute a post-build step that will take your artifact, and rename it to MyJob-latest.zip, but you are losing versioning in the filename. You can also chose to copy instead of rename, but you end up with double the space used for storing these artifacts.
Alternative 2 (recommended):
As a post-build action, upload the artifact to a central repository. It can be Artifactory, or even plain SVN. When you upload it, it will be renamed MyJob-latest.zip and the previous one would be overwritten. This way you have a static link that will always have the latest artifact from lastSuccessfulBuild
There is actually a plugin to assign aliases to build you've run, and I have found it pretty handy: the Build Alias Setter Plugin.
You can use it for instance to assign an alias in the form of your own version number for a build, instead (or rather in addition) to the internal Jenkins-assigned build number.
I found that it is usually most practical to use it in conjunction with the EnvInject plugin (or your favorite variant): you would export an env variable (e.g. MY_VAR=xyz) with a value to the target version or moniker, and then use the form ${ENV,var="myvar"} in the "Token Macro alias" config that the plugin provides in your job config.
You can also use it to assign aliases in the form of "lastSuccesful" if you have such a need, which allows you to distinguish between different types of successful (or other state) builds.
Wait thee's more! You can also use the /*zip*/ trick in conjunction with the alias setter as well.
I'm trying to use global variables within Jenkins on Windows to "automagically" retrieve the proper code base from our SCM system, but in each case that I've tried the variable substitution is not happening.
I've set up some global variables, with default values, within "Configure System" and have tried to access them with $VARIABLE, ${VARIABLE} and %VARIABLE% as part of the Branch field for the Surround SCM plugin with no success whatsoever.
I've also installed the Global Variable String Parameter plugin with the same success rate (0%). Using a literal value works just fine, but no type of variable substitution seems to work at all and I'm sure that someone has come upon this before and resolved it.
I've tried searching for something similar to this but nothing really approaches this usage of globals, instead it is normally discussed as a function within an external script, or parameter passed to a batch file, etc.
I've run "set" as the first step and can see that the variable is available, but the substitution is just not happening. If it means I will have to script something, then so be it, as I am trying to make this extremely flexible and as headache free as possible, but that isn't seeming to be the case in this case thus far.
My problem is eerily similar to this post: How are environment variables used in Jenkins with Windows Batch Command?, but again, I'm not looking to script this as it is a MUCH simpler solution to use the variable values directly.
from https://wiki.jenkins-ci.org/display/JENKINS/Surround+SCM+Plugin
Troubleshooting
Please contact Seapine support with questions about the integration or
to report bugs or feature requests.
Set your Jenkins project to be parameterized. Create a string parameter GIT_BRANCH that will be your branch variable (for example).
Under Source Control Management, use your branch variable in the form $GIT_BRANCH
That’s it. When you run your project, you will be prompted to enter a value for your GIT_BRANCH parameter.