A custom plugin we wrote for an older version of Jenkins uses an EnvironmentContributingAction to provide environment variables to the execution so they could be used in future build steps and passed as parameters to downstream jobs.
While attempting to convert our build to workflow, I'm having trouble accessing these variables:
node {
// this step queries an API and puts the results in
// environment variables called FE1|BE1_INTERNAL_ADDRESS
step([$class: 'SomeClass', parameter: foo])
// this ends up echoing 'null and null'
echo "${env.FE1_INTERNAL_ADDRESS} and ${env.BE1_INTERNAL_ADDRESS}"
}
Is there a way to access the environment variable that was injected? Do I have to convert this functionality to a build wrapper instead?
EnvironmentContributingAction is currently limited to AbstractBuilds, which WorkflowRuns are not, so pending JENKINS-29537 which I just filed, your plugin would need to be modified somehow. Options include:
Have the builder add a plain Action instead, then register an EnvironmentContributor whose buildEnvironmentFor(Run, …) checks for its presence using Run.getAction(Class).
Switch to a SimpleBuildWrapper which defines the environment variables within a scope, then invoke it from Workflow using the wrap step.
Depend on workflow-step-api and define a custom Workflow Step with comparable functionality but directly returning a List<String> or whatever makes sense in your context. (code sample)
Since PR-2975 is merged, you are able to use new interface:
void buildEnvVars(#Nonnull Run<?, ?> run, #Nonnull EnvVars env, #CheckForNull Node node)
It will be used by old type of builds as well.
Related
In Jenkins declarative pipeline we can define build parameters like
pipeline {
…
parameters {
string(name: 'PARAMETER', defaultValue: 'INITIAL_DEFAULT')
choice(name: 'CHOICE', choices: ['THIS', 'THAT'])
}
…
}
However the parameter definitions of the job are only updated when the job runs after the build parameters dialog was already shown. That is, when I change the INITIAL_DEFAULT to something else, the next build will still default to INITIAL_DEFAULT and only the one after that will use the new value.
The same problem is with the choices, and there it is even more serious, because string default can be overwritten easily when starting the build, but if the new option isn't there, it cannot be selected at all.
So is there a way to define functions or expressions that will be executed before the parameter dialog to calculate current values (from files, variable in global settings or any other suitable external configuration)?
I remember using some plugins for this in the past with free-style jobs, but searching the plugin repository I can't find any that would mention how to use it with pipelines.
I don't care too much that the same problem applies to adding and removing parameters, because that occurs rarely. But we have some parameters where the default changes often and we need the next nightly to pick up the updated value.
It turns out the extended-choice-parameter does work with pipeline, and the configurations can be generated by the directive generator. It looks something like
extendedChoice(
name: 'PARAMETER',
type: 'PT_TEXTBOX',
defaultPropertyFile: '/var/lib/jenkins/something.properties',
defaultPropertyKey: 'parameter'
)
(there are many more options available in the generator)
Groovy script to get global environment variables can be had from this other answer.
I have created a Global choice Parameter using Extensible Choice Parameter plugin.
I am using this parameter list in one of my parametrized jenkins job.
Is there a way in jenkins, where I can execute the job with each of the parameters in the Global choice Parameter list?
I have had a look on Build Flow job in jenkins, as suggested in this answer, but it seems it accepts hardcoded parameters only, and not dynamic.
I finally managed to resolve this using the following steps (with great help from this post) -
As my parameters list is dynamic in nature, it could be added or modified according to other jobs, we have managed it in a text file.
Next, We have used Extensible Choice Parameter plugin to display the parameters, using the groovy script -
def list = [];
File file = new File("D:/JenkinJob/parameterList.txt")
file.eachLine { line ->
list.add("$line")
}
return list
Now I want to call this jenkins job for each of the parameter.
For this, I have installed, BuildFlow plugin, and crated a new jenkins job of BuildFlow type -
Next, get the Extended Choice Parameter plugin, and configure it as follows -
Now in the flow step of this job, write this script, where "Feature" is the parameter, that is just created above, and within call to "build" parameter, pass in the name of job which we want to call for each parameter -
def features = params['Features'].split(',')
for (feature in features ) {
build("JobYouWantToCall", JobParameter: feature,)
}
I have an Apache Beam job running on Google Cloud Dataflow, and as part of its initialization it needs to run some basic sanity/availability checks on services, pub/sub subscriptions, GCS blobs, etc. It's a streaming pipeline intended to run ad infinitum that processes hundreds of thousands of pub/sub messages.
Currently it needs a whole heap of required, variable parameters: which Google Cloud project it needs to run in, which bucket and directory prefix it's going to be storing files in, which pub/sub subscriptions it needs to read from, and so on. It does some work with these parameters before pipeline.run is called - validation, string splitting, and the like. In its current form in order to start a job we've been passing these parameters to to a PipelineOptionsFactory and issuing a new compile every single time, but it seems like there should be a better way. I've set up the parameters to be ValueProvider objects, but because they're being called outside of pipeline.run, Maven complains at compile time that ValueProvider.get() is being called outside of a runtime context (which, yes, it is.)
I've tried using NestedValueProviders as in the Google "Creating Templates" document, but my IDE complains if I try to use NestedValueProvider.of to return a string as shown in the document. The only way I've been able to get NestedValueProviders to compile is as follows:
NestedValueProvider<String, String> pid = NestedValueProvider.of(
pipelineOptions.getDataflowProjectId(),
(SerializableFunction<String, String>) s -> s
);
(String pid = NestedValueProvider.of(...) results in the following error: "incompatible types: no instance(s) of type variable(s) T,X exist so that org.apache.beam.sdk.options.ValueProvider.NestedValueProvider conforms to java.lang.String")
I have the following in my pipelineOptions:
ValueProvider<String> getDataflowProjectId();
void setDataflowProjectId(ValueProvider<String> value);
Because of the volume of messages we're going to be processing, adding these checks at the front of the pipeline for every message that comes through isn't really practical; we'll hit daily account administrative limits on some of these calls pretty quickly.
Are templates the right approach for what I want to do? How do I go about actually productionizing this? Should (can?) I compile with maven into a jar, then just run the jar on a local dev/qa/prod box with my parameters and just not bother with ValueProviders at all? Or is it possible to provide a default to a ValueProvider and override it as part of the options passed to the template?
Any advice on how to proceed would be most appreciated. Thanks!
The way templates are currently implemented there is no point to perform "post-template creation" but "pre-pipeline start" initialization/validation.
All of the existing validation executes during template creation. If the validation detects that there the values aren't available (due to being a ValueProvider) the validation is skipped.
In some cases it is possible to approximate validation by adding runtime checks either as part of initial splitting of a custom source or part of the #Setup method of a DoFn. In the latter case, the #Setup method will run once for each instance of the DoFn that is created. If the pipeline is Batch, after 4 failures for a specific instance it will fail the pipeline.
Another option for productionizing pipelines is to build the JAR that runs the pipeline, and have a production process that runs that JAR to initiate the pipeline.
Regarding the compile error you received -- the NestedValueProvider returns a ValueProvider -- it isn't possible to get a String out of that. You could, however, put the validation code into the SerializableFunction that is run within the NestedValueProvider.
Although I believe this will currently re-run the validation everytime the value is accessed, it wouldn't be unreasonable to have the NestedValueProvider cache the translated value.
I'm attempting to write a global function script that uses groovy.sql.SQL.
When adding the annotation #GrabConfig(systemClassLoader=true) I get an exception when using the global function in Jenkinsfile.
Here is the exception:
hudson.remoting.ProxyException: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
General error during conversion: No suitable ClassLoader found for grab
Here is my code:
#GrabResolver(name='nexus', root='http://internal.repo.com')
#GrabConfig(systemClassLoader=true)
#Grab('com.microsoft.sqlserver:sqljdbc4:4.0')
import groovy.sql.Sql
import com.microsoft.sqlserver.jdbc.SQLServerDriver
def call(name) {
echo "Hello world, ${name}"
Sql.newInstance("jdbc:sqlserver://ipaddress/dbname", "username","password", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
// sql.execute "select count(*) from TableName"
}
Ensure that the "Use groovy sandbox" checkbox is unticked (it's below the pipeline script text box).
As explained here, Pipeline "scripts" are not simple Groovy scripts, they are heavily transformed before running, some parts on master, some parts on slaves, with their state (variable values) serialized and passed to the next step. As such, not every Groovy feature is supported.
I'm not sure about #Grab support. It is discussed in JENKINS-26192 (which is declared as resolved, so maybe it works now).
Extract from a very interesting comment:
If you need to perform some complex or expensive tasks with
unrestricted Groovy physically running on a slave, it may be simplest
and most effective to simply write that code in a *.groovy file in
your workspace (for example, in an SCM checkout) and then use tool and
sh/bat to run Groovy as an external process; or even put this stuff
into a Gradle script, Groovy Maven plugin execution, etc. The workflow
script itself should be limited to simple and extremely lightweight
logical operations focused on orchestrating the overall flow of
control and interacting with other Jenkins features—slave allocation,
user input, and the like.
In short, if you can move that custom part that needs SQL to an external script and execute that in a separate process (called from your Pipeline script), that should work. But doing this in the Pipeline script itself is more complicated.
Does jenkins have any way to set global properties from a job? We have many such needs for this - but specifically - we have a number of slaves, across unix and windows, and various different permissions locations - so it's not easy to have a connected file system. We have various levels of maturity that we promote through - so for instance, we want to promote some build number to UAT - and then promote whatever number is in UAT to training and so on. So - really, in the "release to uat" - we want to store some idea of which build number was released - and read that from the "release to training" job. At the moment we are hacking it by restricting them to run from the same slave, and writing it to a file, which is very much not ideal.
I may not have totally understood your question but you can perform a lot of work with the built in groovy scripting function in jenkins, including reading parameters from other jobs, and rewriting or initializing the parameters in the current job. You can use parameters like this to record information that can be retrieved on demand by other jobs
For instance you can find the build number of the last successful build of a certain project:
import hudson.model.*
def hif = Hudson.instance
def a = hif.getItems(hudson.model.Project).find{it.displayName.toUpperCase()=='MY_PROJECTNAME'}.getBuilds().findAll{it.result==Result.SUCCESS }.first()
out.println a.number //build number
out.println a.buildVariableResolver.resolve('someVariable')// some parameter used to call a
(you could include any other criteria at this point)
If you want to save information to a parameter that can later be read by another bulid step or another job then you first create the parameter in the job config, then write to it in code like so:
import hudson.model.*
def hif = Hudson.instance
def buildMap = build.getBuildVariables()
buildMap['MySpecialVar']='SomeValue'
setBuildParameters(buildMap)
def setBuildParameters(map) {
def npl = new ArrayList<StringParameterValue>()
for (e in map) {
npl.add(new StringParameterValue(e.key.toString(), e.value.toString()))
}
def newPa = null
def oldPa = build.getAction(ParametersAction.class)
if (oldPa != null) {
build.actions.remove(oldPa)
newPa = oldPa.createUpdated(npl)
} else {
newPa = new ParametersAction(npl)
}
build.actions.add(newPa)
}
Combining these techniques you could for instance:
Save a bunch of information as 'output parameters' in job one
Find the most recent successful instance of job one and read its parameters
If necessary save those parameters to job2's parameter list so they are accessible from other build steps.
OR
If you are happy to use files then you may be able to use the archive plugin, where you would write to a file and then archive it as a post build action. The file would be saved to the master, and you could use the 'copy artifacts from another project' option in the second build to retrieve the file. You can use parameter filters and the techniques above to pick the right build.
Setting an environment variable permanently is entirely dependent on the underlying Operating System.
For example on Windows, the SetX command can be used, however note that SetX only takes affect after the next process is created by the system the inherits from global configuration. So, if you run SetX, and then run another job, it will not notice the change. However if you run SetX, and then restart Jenkins process (from which all child jobs inherit variables), then the other job will notice the change.
Not sure how to set permanent variables in Linux, but a quick Google search returns this answer: https://unix.stackexchange.com/questions/117467/how-to-permanently-set-environmental-variables