Currently I am having one jenkinsfile which in process of development has grown to huge size. Mostly it contains some groovy methods used in pipeline, but I would like to place groovy scripts in separate files in other repository and than checkout every time when I need those scripts. Those works on a maps of settings which is commonly used in methods. How can I share those maps in script files? I can't use shared libraries because of security purposes.
An alternative to shared library is the load step, which loads and evaluates the given groovy script file.
settings.groovy:
[
answer: 42,
question: 'tbd'
]
main.groovy:
def settings = load 'settings.groovy'
echo "The answer is: ${settings.answer}"
Output:
The answer is: 42
Related
We have our shared libraries on gitlab called mainlibrary and it has a lot of groovy files.
Example in mainlibrary gitlab repo we have the following files.
startup_pipeline.groovy
cleanup_pipeline.groovy
In one of our Jenkins job we need to include multiple groovy files in the Jenkinsfile. Is this possible?
This is how the Jenkinsfile looks like:
#Library('mainlibrary')_
startup_pipeline(email:'example#example.com')
Can I include the second groovy function file into this Jenkinsfile like this?
#Library('mainlibrary')_
startup_pipeline(email:'example#example.com'),
cleanup_pipeline(email:'example#example.com')
Considering the cleanup_pipeline.groovy is located under the vars folder and your code already has a complete declaration inside, your example may work, and the second file can be included. The only modification is the extra comma:
#Library('mainlibrary')_
startup_pipeline(email:'example#example.com')
cleanup_pipeline(email:'example#example.com')
Or can be under src and imported, but I never used this approach.
Usually, I keep the main logic inside vars, and other complex things go to src.
Based on Jenkins documentation, see more in Directory structure and Defining custom steps.
Video step by step building a shared library, you can build and test something similar if you are not sure about the structure.
Jenkins declarative pipeline is too powerful for us, often users can abuse it. We are thinking to use an opinionated YAML to describe CI/CD pipeline. And it seems there are two choices.
Write a plugin and consume YAML and dynamically create stage / steps.
Write a plugin to convert a YAML to Jenkins pipeline.
I am not expert on Jenkins, so I hope some expert can give some guidance and maybe an example.
using official plugin pipeline-as-yaml, but it has a fixed grammar.
using or customization wolox-ci
create your own shared libaray. However, they are easy from beginning but full grammer design is required when used widely. Here is a psudo code based on curry.
// create a file named yamlCompiler.groovy in shared library,
def call(str){
def rawMap = readYaml(text: str)
// consume yaml and get a lambda function
return {
stage{
steps.each{it ->
it."$type"(it)
}
}
}
}
Use yamlCompiler in your jenkinsfile code block.
#Library('your libs name')
def str =
'''
steps:
- type: sh
script: ls -la
- type: echo
message: xxx
'''
Closure closure = yamlCompiler(str)
closure.call()
I'm looking for a similar solution. We run hardened predefined pipelines for every project, but still want to allow dev teams to customise certain steps within the process —without allowing them the full power of a Jenkinsfile.
I'm also exploring the possibility of an —in your words— "opinionated YAML".
I've so far only found one example of such an implementation: Wolox-CI supports their own pre-defined build steps via YAML. You'll be able to see the steps they support here.
I'm thinking of parsing the YAML using Snake YAML. Here's an SO answer with an example on how to do it.
Two solutions:
create a shared library to abstract the actual pipeline and provide to your users some guidance on how to setup a shared library and a Jenkinsfile sample. Here is an example of embeded pipeline https://github.com/SAP/jenkins-library/blob/master/vars/piperPipeline.groovy
use another tool like https://drone.io/
If you're not an expert and don't want/have the time to become one, the second solution might be the best one.
Really? Is the only difference here when the plugin is executed?:
Write a plugin and consume YAML and dynamically create stage / steps.
Write a plugin to convert a YAML to Jenkins pipeline.
Forgive me, because I may be a little hardened, but abstracting a layer for the dynamic creation of a declarative, or scripted, Jenkinsfile written in the simple groovy lang syntax so that it can be pretty-printed in yml prevents users from updating your yml exactly how? It seems to me your abstraction only adds to the complexity with which you wish to implement usability.
One, all the current yml plugins for Jenkins do exactly that. Two, they don't actually have the full breadth of "features" (yes, I'm using that term loosely here) accessible by implementing the groovy/(java) classes already available in the Jenkins domain (referencing the DSL). Two solutions exist right now for this, and I've investigated both, and implemented both, extensively. One is wolox-ci, which is the better of the two, and the other is Pipeline-as-YAML. In my opinion, it's easy to use, but both lack the full breadth of implementation features simply using groovy provides. So why force it? Simply so your users can have a pretty-printed yml file, and not have to be concerned with simple syntax, which you claim hardens your infrastructure-as-code backend so that the same users can't screw it up? Sorry, I'm calling bull pucky on that assertion. What's to stop anyone from totally screwing up your builds by pushing a change to the yml file which breaks the integration with groovy, or worse, completely changes an algorithm you worked hard to customize?
Sorry, I just don't get it. Sure, making something more human readable is always a good thing. Doing it because of the reasons you've stipulated makes no sense, though. Also, unless you have a super simple defined algorithm in your CI/CD process, without any non-continuous-passing-style transform methods being implemented, then using the current iterations of the yml-as-Jenkinsfile-templates plugins is probably not the way you want to go.
Now, you could write your own plugin to do this, but what's the technical debt on that, versus just learning the groovy syntax? Also, it still doesn't prevent users from making code changes to your build infrastructure, then integrating those changes in a simple yml file.
Context:
At my work most developers are free to write their own Jenkinsfile for their own team's projects.
As the Jenkins admin, I provide developers with a global shared library.
Most projects are using either v1 or v2 or v3 or another version of this library, using the idiom library("theSharedLib#v#").
Question: Is there a way for me to find out which Jenkinsfile is using which version of the shared library without having to actually lookup into all those Jenkinsfile files (50+ files in as much git repos)?
What I would see best is some mechanism that write up (into a file on the Jenkins master or in a DB) which project/Jenkinsfile is using which version at the time the library is loaded.
A possible solution would be to add some code to every function inside the library that will actually do this reporting. I could then see which function is used by who. Any better solution?
I wrote https://github.com/CiscoDevNet/es-logger to gather information such as this from Jenkins. It has a plugin that will run a regex against the console log of a completed job and can then post events to elastic search.
Jenkins helpfully posts library loads at the start of the log such as:
Loading library sharedLib#version
So a simple regex like
"^Loading library\S+(?P<library_name>.*?)#(?P<library_version>.*?)\S+$"
added to the console_log_events plugin would generate events in an elastic search for each usage and each version.
As far as I know, JMeter allows you to send multiple POST request with different parameters (e.g. { "value": "value1"}, {"value": "value2"}, ...) However, I'm more comfortable using a terminal-based interface similar to ab or siege. Basically, I need to load test a server simulating the case in which some requests are not previously cached.
Are there alternatives to JMeter for Linux that are able to use different parameters for a POST request?
UPDATE
As far as I can tell, JMeter requires the creation of a test plan (jmx file) in order to run via the command line. Unfortunately, this test plan needs to be built using the GUI, which is precisely what I want to avoid.
UPDATE 2
I will use JMeter because it offers dynamic parameters for POST requests and most alternatives depend on JMeter. However, if anyone knows of a standalone library that works exclusively from the terminal (similar to ab), please let me know.
you can use JMeter in terminal mode, it's called Non GUI mode.
To variabilize just use CsV dataset to load variables (varName for example )per thread, then use ${varName}
See :
http://jmeter.apache.org/usermanual/get-started.html#non_gui
http://jmeter.apache.org/usermanual/component_reference.html#CSV_Data_Set_Config
Nice report at end:
http://jmeter.apache.org/usermanual/generating-dashboard.html
If you don't want to use GUI even for building the test, then look at :
https://github.com/flood-io/ruby-jmeter
It allows you to generate the JMX from a DSL file.
Examples here:
https://github.com/flood-io/ruby-jmeter/tree/master/examples
DSL here:
https://github.com/flood-io/ruby-jmeter/blob/master/lib/ruby-jmeter/DSL.md
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
require 'ruby-jmeter'
test do
csv_data_set_config name:'MyCsv', filename: '/path to file', variableNames: 'myParam'
threads count: 10 do
visit name: 'Qwant Search', url: 'https://lite.qwant.com/?q=flood.io&t=web&p=${myParam}'
end
end.jmx(file: "path to your output plan")
Save file to ruby-jmeter-csv.rb
You can then generate the plan with:
ruby ruby-jmeter-csv.rb
And run it in non gui mode.
In fact JMeter GUI should be used for tests development and debugging only, when it comes to running the load test - it is recommended to run JMeter in command line mode, via Ant task or Maven plugin. Also there is a couple of more "geek" alternatives, i.e.:
JMeter .jmx scripts are basically XML files so you can use your favourite text editor to create or amend them
You can use JMeter API to create and kick off JMeter tests using Java language
If you're still looking for an alternative, here are few free and open source load testing tools
Grinder - you can write scripts in Jython
Gatling - you can write scripts in Scala-based DSL
Tsung - this guy exists for Linux and Unix-based platforms only, Erlang-based. Scripts are XML files.
Taurus - automation framework which supports all aforementioned tools (and some more), Python based, configuration files have simple YAML syntax.
See Open Source Load Testing Tools: Which One Should You Use? for more information on the above tools and comparison of them with JMeter
I am trying out the Jenkins Workflow plugin (https://github.com/jenkinsci/workflow-plugin) and using the Workflow Global Library (https://github.com/jenkinsci/workflow-plugin/tree/master/cps-global-lib) that comes with it.
I wanted to embed some script calls in my shared functions (bash/python). The obvious way to do this seems to be by using sh """...""". However this leads to some escaping being required ($ has to be escaped). Also its a bit messy to develop a script inside string quotes.
Is there a way to access resource files (e.g. a .sh or .py file) stored in the Global Library during a workflow execution?
So that I can do something like
sh getScript("script.sh")
''' syntax might be more helpful, since it reduces the need to escape. There are other Groovy syntaxes that are even better for long blocks, with various tradeoffs on the escape character.
But yes it would be helpful to be able to reference a static resource in the “classpath”. Feel free to file an RFE for this.