I am trying to automate testing Jenkins groovy dsl scripts, like here:
https://github.com/sheehan/job-dsl-gradle-example
The idea I think is very straight forward, what I'm having issues with is setting environment variables for the dummy Jenkins. I followed the instructions here:
https://wiki.jenkins-ci.org/display/JENKINS/Unit+Test
Specifically "How to set env variables" section and added the following to my test executor:
import hudson.slaves.EnvironmentVariablesNodeProperty
import hudson.EnvVars
/**
* Tests that all dsl scripts in the jobs directory will compile.
*/
class JobScriptsSpec extends Specification {
#Shared
#ClassRule
JenkinsRule jenkinsRule = new JenkinsRule()
EnvironmentVariablesNodeProperty prop = new EnvironmentVariablesNodeProperty();
EnvVars envVars = prop.getEnvVars();
#Unroll
void 'test script #file.name'(File file) {
given:
envVars.put("ENVS", "dev19");
jenkinsRule.jenkins.getGlobalNodeProperties().add(prop);
JobManagement jm = new JenkinsJobManagement(System.out, [:], new File('.'))
when:
new DslScriptLoader(jm).runScript(file.text)
then:
noExceptionThrown()
where:
file << jobFiles
}
However when I run the actual tests for one of the scripts, I still see the following:
Failed tests
test script Build.groovy
Expected no exception to be thrown, but got 'javaposse.jobdsl.dsl.DslScriptException'
at spock.lang.Specification.noExceptionThrown(Specification.java:118)
at com.dslexample.JobScriptsSpec.test script #file.name(JobScriptsSpec.groovy:40)
Caused by: javaposse.jobdsl.dsl.DslScriptException: (script, line 3) No such property: ENVS for class: script
The script Build.groovy uses the variable "${ENVS}" (as if it were provided by parameter in seed job of Jenkins), which works as expected when actually running in Jenkins... So any way to set these "parameters" or env variables in the test jenkins context?
Example of how I use the ENVS variable in the Build.groovy:
def envs = '-'
"${ENVS}".eachLine{
def env = it
envs+=env+'-'
}
envs.substring(0,envs.length()-1)
job('Build'+envs) {
...
}
The second argument of the JenkinsJobManagement constructor is a map of environment variables which will be available in the DSL scripts.
Map<String, String> envVars = [
FOO: 'BAR'
]
JobManagement jm = new JenkinsJobManagement(System.out, envVars, new File('.'))
Related
After the upgrade from Jenkins "1.651.3" to "2.346.3 LTS".
My job log contains this message many times:
ERROR: Not all environment variables could be successfully injected. Check for similarly-named environment variables.
There is no clue. How to find which variables could not be successfully injected?
I found that message goes from Environment Injector plugin
It goes from class EnvInjectEnvVarsContributor
And it seems that it is not an error. It is just a warning, that some property, already exists in environment variables.
Responsilbe commit from history
Map<String, String> result = jobPropertyInfo.getPropertiesContentMap(env);
if (result != null) {
int expectedEnvSize = env.size() + result.size();
env.putAll(result);
if (env.size() != expectedEnvSize) {
listener.error("Not all environment variables could be successfully injected. " +
"Check for similarly-named environment variables.");
}
}
To find which properties are confilicting I used this groovy script executed in the job like "Execute system Groovy script"
import hudson.model.*
import org.jenkinsci.plugins.envinject.EnvInjectJobProperty
import jenkins.model.Jenkins
println "FIND CONFLICTING PROPERTIES GROOVY SCRIPT---------------------------- START"
def job = Hudson.instance.getJob('my_job')
println job
EnvInjectJobProperty jobProperty = (EnvInjectJobProperty) job.getProperty(EnvInjectJobProperty.class);
def jobPropertyInfo = jobProperty.getInfo();
def env = Jenkins.instance.getGlobalNodeProperties()[0].getEnvVars()
Map<String, String> result = jobPropertyInfo.getPropertiesContentMap(env);
println "RESULT"
println result
println result.size()
println "ENV"
println env
println env.size()
println "FIND CONFLICTING PROPERTIES GROOVY SCRIPT---------------------------- END"
Maybe it will help to someone.
I searched a lot for this problem but couldn't find working solution anywhere. Can anybody please help me out? I want to get already existing env vars value through jenkins script console.
You need to distinguish:
build environment variables:
def myVar = build.getBuildVariables().get('myVar')
system environment variables:
System.getenv('MY_VARIABLE')
If you see
groovy.lang.MissingPropertyException: No such property: manager for class: Script1
Check this answer, and define build first:
import hudson.model.*
def build = Thread.currentThread().executable
def buildNumber = build.number
According to this answer, in order to access env vars from Jenkins script console, do as follows :
import jenkins.model.*;
import hudson.slaves.EnvironmentVariablesNodeProperty;
import hudson.EnvVars;
jenkins = Jenkins.instance;
EnvironmentVariablesNodeProperty prop = jenkins.getGlobalNodeProperties().get(EnvironmentVariablesNodeProperty.class)
EnvVars env = prop.getEnvVars()
def myVariable = env['MY_VAR']
The env vars listed in http://<JENKINS_URL>/env-vars.html are available for each build. In order to access these variables in the Jenkins script console you need to define first the build :
build = Jenkins.instance.getItemByFullName('JOB_NAME').getBuildByNumber(BUILD_NUMBER)
envvars = build.getEnvironment()
envvars.each{envvar ->
println envvar
}
How do I invoke Global environment variables in Jenkinsfile?
For example, if I have a variable -
name:credentialsId
value:xxxx-xxxx-xxxxx-xxxxxxxxx
How do I use it in the groovy script?
I tried ${credentialsId}, but it didn't work. It will just give error:
java.lang.NoSuchMethodError: No such DSL method '$' found among steps [ArtifactoryGradleBuild, ........
In a Jenkinsfile, you have the "Working with the Environment" which mentions:
The full list of environment variables accessible from within Jenkins Pipeline is documented at localhost:8080/pipeline-syntax/globals#env,
The syntax is ${env.xxx} as in:
node {
echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"
}
See also "Managing the Environment".
How can I pass the Global variables to the Jenkinsfile?
When I say Global variables - I mean in
Jenkins -> Manage Jenkins -> Configure System -> Global properties -> Environment variables
See "Setting environment variables"
Setting an environment variable within a Jenkins Pipeline can be done with the withEnv step, which allows overriding specified environment variables for a given block of Pipeline Script, for example:
Jenkinsfile (Pipeline Script)
node {
/* .. snip .. */
withEnv(["NAME=value"]) {
... your job
}
}
When referring to env in Groovy scope, simply use env.VARIABLE_NAME, for example to pass on BUILD_NUMBER of upstream job to a triggered job:
stage ('Starting job') {
build job: 'TriggerTest', parameters: [
[$class: 'StringParameterValue', name: 'upstream_build_number', value: env.BUILD_NUMBER]
]
}
Scripted pipeline
To read an environment variable whose name you know, use env.NAME
To read an environment variable whose name is not known until runtime use env.getProperty(name).
For example, a value from a YAML config file represents an environment variable name:
config.yaml (in workspace)
myconfig:
key: JOB_DISPLAY_URL
Jenkinsfile
node {
println("Running job ${env.JOB_NAME}")
def config = readYaml(file:'config.yaml')
def value = env.getProperty(config.myconfig.key)
println("Value of property ${config.myconfig.key} is ${value}")
}
For getting values all env.VAR, env['VAR'], env.getProperty('VAR') are fine.
For setting values the only safe way at the moment is withEnv. If you try to assign values to env.VAR it may not work in some cases like for parallel pipelines (like in JENKINS-59871).
Another syntax is $ENV:xxxx
node {
echo "Running $ENV.BUILD_ID on $ENV.JENKINS_URL" }
This worked for me
I am doing a simple pipeline:
Build -> Staging -> Production
I need different environment variables for staging and production, so i am trying to source variables.
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh'
But it returns Not found
[Stack Test] Running shell script
+ source /var/jenkins_home/.envvars/stacktest-staging.sh
/var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: 2: /var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: source: not found
The path is right, because i run the same command when i log via ssh, and it works fine.
Here is the pipeline idea:
node {
stage name: 'Build'
// git and gradle build OK
echo 'My build stage'
stage name: 'Staging'
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh' // PROBLEM HERE
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To staging
input message: "Does Staging server look good?"
stage name: 'Production'
sh 'source $JENKINS_HOME/.envvars/stacktest-production.sh'
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To production
sh './deploy.sh'
}
What should i do?
I was thinking about not using pipeline (but i will not be able to use my Jenkinsfile).
Or make different jobs for staging and production, using EnvInject Plugin (But i lose my stage view)
Or make withEnv (but the code gets big, because today i am working with 12 env vars)
One way you could load environment variables from a file is to load a Groovy file.
For example:
Let's say you have a groovy file in '$JENKINS_HOME/.envvars' called 'stacktest-staging.groovy'.
Inside this file, you define 2 environment variables you want to load
env.DB_URL="hello"
env.DB_URL2="hello2"
You can then load this in using
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
Then you can use them in subsequent echo/shell steps.
For example, here is a short pipeline script:
node {
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
echo "${env.DB_URL}"
echo "${env.DB_URL2}"
}
From the comments to the accepted answer
Don't use global 'env' but use 'withEnv' construct, eg see:
issue #9: don't set env vars with global env in top 10 best practices jenkins pipeline plugin
In the following example: VAR1 is a plain java string (no groovy variable expansion), VAR2 is a groovy string (so variable 'someGroovyVar' is expanded).
The passed script is a plain java string, so $VAR1 and $VAR2 are passed literally to the shell, and the echo's are accessing environment variables VAR1 and VAR2.
stage('build') {
def someGroovyVar = 'Hello world'
withEnv(['VAR1=VALUE ONE',
"VAR2=${someGroovyVar}"
]) {
def result = sh(script: 'echo $VAR1; echo $VAR2', returnStdout: true)
echo result
}
}
For secrets / passwords you can use credentials binding plugin
Example:
NOTE: CREDENTIALS_ID1 is a registered username/password secret on the Jenkins settings.
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
echo "User name: $USER"
echo "Password: $PASSWORD"
}
}
The jenkisn console log output hides the real values:
[Pipeline] echo
User name: ****
[Pipeline] echo
Password: ****
Jenkins and credentials is a big issue, probably see: credentials plugin
For completeness: Most of the time, we need the secrets in environment variables, as we use them from shell scripts, so we combine the withCredentials and withEnv like follows:
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
withEnv(["ENV_USERNAME=${USER}",
"ENV_PASSWORD=${PASSWORD}"
]) {
def result = sh(script: 'echo $ENV_USERNAME', returnStdout: true)
echo result
}
}
}
Another way to resolve this install 'Pipeline Utility Steps' plugin that provides us readProperties method ( for reference please go to the link https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#pipeline-utility-steps)
Here in the example we can see that they are storing the keys into an array and using the keys to retrieve the value.
But in that case the in production the problem will be like if we add any variable later into property file that variable needs to be added into the array of Jenkins file as well.
To get rid of this tight coupling, we can write code in such a way so that the Jenkins build environment can get information automatically about all the existing keys which presents currently in the Property file. Here is an example for the reference
def loadEnvironmentVariables(path){
def props = readProperties file: path
keys= props.keySet()
for(key in keys) {
value = props["${key}"]
env."${key}" = "${value}"
}
}
And the client code looks like
path = '\\ABS_Output\\EnvVars\\pic_env_vars.properties'
loadEnvironmentVariables(path)
With declarative pipeline, you can do it in one line ( change path by your value):
script {
readProperties(file: path).each {key, value -> env[key] = value }
}
Using withEnv() to pass environment variables from file splitted by new line and casted to List:
writeFile file: 'version.txt', text: 'version=6.22.0'
withEnv(readFile('version.txt').split('\n') as List) {
sh "echo ${version}"
}
If you are using Jenkins 2.0 you can load the property file (which consists of all required Environment variables along with their corresponding values) and read all the environment variables listed there automatically and inject it into the Jenkins provided env entity.
Here is a method which performs the above stated action.
def loadProperties(path) {
properties = new Properties()
File propertiesFile = new File(path)
properties.load(propertiesFile.newDataInputStream())
Set<Object> keys = properties.keySet();
for(Object k:keys){
String key = (String)k;
String value =(String) properties.getProperty(key)
env."${key}" = "${value}"
}
}
To call this method we need to pass the path of property file as a string variable For example, in our Jenkins file using groovy script we can call like
path = "${workspace}/pic_env_vars.properties"
loadProperties(path)
Please ask me if you have any doubt
Here is a complete example of externalizing environment variables and loading them in Jenkins pipeline execution. The pipeline is written in a declarative style.
stage('Reading environment variable defined in groovy file') {
steps {
script {
load "./pipeline/basics/extenvvariable/env.groovy"
echo "${env.env_var1}"
echo "${env.env_var2}"
}
}
}
Complete code example:
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/Jenkinsfile
Where variables are loaded from a groovy file placed with the pipeline code only.
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/env.groovy
This pattern comes very handy when you are creating a generic pipeline that could be used across teams.
You can externalize the dependent variable in such groovy file and each team can define their values according to their ecosystem.
Another solution is to use a custom method without allowing extra permissions such as for new Properties() which leads to this error before allowing:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new java.util.Properties
or adding extra plugin methods such as readProperties.
here is a method which reads a simple file named env_vars in this format:
FOO=bar
FOO2=bar
pipeline {
<... skipped lines ...>
script {
loadEnvironmentVariablesFromFile("env_vars")
echo "show time! ${BAR} ${BAR2}"
}
<... skipped lines ...>
}
private void loadEnvironmentVariablesFromFile(String path) {
def file = readFile(path)
file.split('\n').each { envLine ->
def (key, value) = envLine.tokenize('=')
env."${key}" = "${value}"
}
}
I am using the grails-cdn-asset-pipline plugin. I've gone through the installation and configuration steps on GitHub and I reach the usage section which says
Add this command to your build process (usually before war generation and deployment).
// If all the settings are defined in your Config.groovy
grails asset-cdn-push
// Or
grails asset-cdn-push --provider=S3 --directory=my-bucket --gzip=true --storage-path=some-prefix --expires=365 --region=eu-west-1 --access-key=$MY_S3_ACCESS_KEY --secret-key=$MY_S3_SECRET_KEY
Where in my project do I put this command?
Is it something that I can do within the context of my project, or do I need to keep it separate in another build process and run it in an environment like Jenkins?
In _Events.groovy, I tried to invoke the script in the eventCreateWarStart, but I am having no luck there. (Code taken from this question)
eventCreateWarStart = { warName, stagingDir ->
def pluginManager = PluginManagerHolder.pluginManager
def plugin = pluginManager.getGrailsPlugin("cdn-asset-pipline")
def pluginDir = plugin.descriptor.file.parentFile
Map<String, String> env = System.getenv()
final processBuilder = new ProcessBuilder()
processBuilder.directory(new File("${cdnAssetPipelinePluginDir}/scripts"))
processBuilder.command([env['GRAILS_HOME']+"/bin/grails","cdn-asset-push"])
println processBuilder.directory()
Process proc = processBuilder.start()
proc.consumeProcessOutput(out, err)
proc.waitFor()
}
This link explains the run-script functionality which was merged into Grails 1.3.6. But I ran into the same problem of not knowing where to run it automatically.