I am using jenkins scripted file.
I have .key file stored in jenkins files ( Where all env files are present ).
And I need to copy that file to code folder.
Like i want to store device.key in src/auth/keys.
Then will run test on code in pipeline.
I am using scripted Jenkinsfile. And i am unable to find any way to this.
node{
def GIT_COMMIT_HASH
stage('Checkout Source Code and Logging Into Registry') {
echo 'Logging Into the Private ECR Registry'
checkout scm
sh "git rev-parse --short HEAD > .git/commit-id"
GIT_COMMIT_HASH = readFile('.git/commit-id').trim()
# NEED TO COPY device.key to /src/auth/key
}
stage('TEST'){
nodejs(nodeJSInstallationName:'node'){
sh 'npm install'
sh 'npm test'
}
}
}
How I solved this:
I installed Config File Provider Plugin
I added the files as custom files for each environment
In the JenkinsFile I replace the configuration file from the project with the one comming from jenkins:
stage('Add Config files') {
steps {
configFileProvider([configFile(fileId: 'ID-of-Jenkins-stored-file', targetLocation: 'relative-path-to-destination-file-in-the-project')]) {
// some block , maybe a friendly echo for debugging
} } }
Please see the plugin doc as it is capable of replacing tokens in XML and json files and many others.
Want to use the below functionality(shown in image link) in Jenkins as code, but i'm failing to do, kindly help me replicate the functionality in the image to groovy script
stage ('Build Instance') {
sh '''
bash ./build.sh -Ddisable-rpm=false
'''
env "/fl/tar/ver.prop"
}
Jenkins GUI usage of Env Inject
Got a simple workaround :
script {
def props = readProperties file: '/fl/tar/ver.prop' //readProperties is a step in Pipeline Utility Steps plugin
env.WEATHER = props.WEATHER //assuming the key name is WEATHER in properties file
}
I want to parametrize my Jenkins pipeline with a simple properties config file
skip_tests=true
that I've added to Jenkins Config File Managment:
In my pipeline I'm importing this file and try to read from it using the Jenkins Pipeline Config File Plugin.
node('my-swarm') {
MY_CONFIG = '27206b95-d69b-4494-a430-0a23483a6408'
try {
stage('prepare') {
configFileProvider([configFile(fileId: "$MY_CONFIG", variable: 'skip_tests')]) {
echo $skip_tests
assert $skip_tests == 'true'
}
}
} catch (Exception e) {
currentBuild.result = 'FAILURE'
print e
}
}
This results in an error:
provisioning config files...
copy managed file [my.properties] to file:/home/jenkins/build/workspace/my-workspace#tmp/config7043792000148664559tmp
[Pipeline] {
[Pipeline] }
Deleting 1 temporary files
[Pipeline] // configFileProvider
[Pipeline] }
[Pipeline] // stage
[Pipeline] echo
groovy.lang.MissingPropertyException: No such property: $skip_tests for
class: groovy.lang.Binding
Any ideas what I'm doing wrong here?
With the help of the other answers and How to read properties file from Jenkins 2.0 pipeline script I found the following code to work:
configFileProvider([configFile(fileId: "$PBD1_CONFIG", variable: 'configFile')]) {
def props = readProperties file: "$configFile"
def skip_tests = props['skip_tests']
if (skip_tests == 'true') {
print 'skipping tests'
} else {
print 'running tests'
}
}
I had to use readProperties from Jenkins' Pipeline Utility Steps Plugin.
Since the file is in property format you can use it in a shell step:
sh """
source ${MY_CONFIG}
.
.
.
"""
You would need to export the properties that need to be available on programs that the shell calls (e.g. Maven)
You made a wrong usage of Groovy GString, you should wrap $skip_tests in " or use skip_tests directly.
configFileProvider([configFile(fileId: "$MY_CONFIG", variable: 'skip_tests')]) {
echo skip_tests
assert skip_tests == 'true'
echo "$skip_tests"
assert "$skip_tests" == 'true'
}
Note: the value of skip_tests is the file path of the config file which is copied from master to job's workspace. It's not the content of the config file.
I'm a using the jenkins pipeline. My usecase is that the developer are using a simple *.ini file that is parsed by a python script to add or remove stage within the jenkinsfile whenever they want. I don't want them to manually edit the jenkinsfile because they won't know how it works.
Expected behaviour is:
When a build is triggered I would like to first execute a python script which might write into the jenkinsfile to add/remove stage according to the *.ini file.
As far as I understand, when an event trigger a jenkins build, the first thing it does is opening the jenkinsfile. However I would like to know if it's possible to run some prebuild script before that ?
Thanks
Edit: here's a simple view of run of the pipeline (blue ocean UI)
The ini file might for example remove in the stage Compilation the step Building Plan C by removing the groovy code doing that in the jenkins file
Give an example for reference.
node {
git url: '', branch: '', credentialsId: ''
def parseStr = sh(script: 'python parser.py xxx.ini', returnStdout: true).trim()
// the python parser expect to return a JSON string like:
// {'run_stage1': false, 'run_stage2': true}
def parseObj = readJSON text: parseStr
stage('stage 1') {
if(parseObj.run_stage1) {
echo 'stage1'
...
}
}
stage('stage 1') {
if(parseObj.run_stage2) {
echo 'stage1'
....
}
}
}
Jenkins pipeline had supply apis: readJSON, readYaml, readProperties to read JSON, YAML and Properties files.
If you choose any of them to replace ini file, you can drop the python parser to make your pipeline more simple
I am doing a simple pipeline:
Build -> Staging -> Production
I need different environment variables for staging and production, so i am trying to source variables.
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh'
But it returns Not found
[Stack Test] Running shell script
+ source /var/jenkins_home/.envvars/stacktest-staging.sh
/var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: 2: /var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: source: not found
The path is right, because i run the same command when i log via ssh, and it works fine.
Here is the pipeline idea:
node {
stage name: 'Build'
// git and gradle build OK
echo 'My build stage'
stage name: 'Staging'
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh' // PROBLEM HERE
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To staging
input message: "Does Staging server look good?"
stage name: 'Production'
sh 'source $JENKINS_HOME/.envvars/stacktest-production.sh'
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To production
sh './deploy.sh'
}
What should i do?
I was thinking about not using pipeline (but i will not be able to use my Jenkinsfile).
Or make different jobs for staging and production, using EnvInject Plugin (But i lose my stage view)
Or make withEnv (but the code gets big, because today i am working with 12 env vars)
One way you could load environment variables from a file is to load a Groovy file.
For example:
Let's say you have a groovy file in '$JENKINS_HOME/.envvars' called 'stacktest-staging.groovy'.
Inside this file, you define 2 environment variables you want to load
env.DB_URL="hello"
env.DB_URL2="hello2"
You can then load this in using
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
Then you can use them in subsequent echo/shell steps.
For example, here is a short pipeline script:
node {
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
echo "${env.DB_URL}"
echo "${env.DB_URL2}"
}
From the comments to the accepted answer
Don't use global 'env' but use 'withEnv' construct, eg see:
issue #9: don't set env vars with global env in top 10 best practices jenkins pipeline plugin
In the following example: VAR1 is a plain java string (no groovy variable expansion), VAR2 is a groovy string (so variable 'someGroovyVar' is expanded).
The passed script is a plain java string, so $VAR1 and $VAR2 are passed literally to the shell, and the echo's are accessing environment variables VAR1 and VAR2.
stage('build') {
def someGroovyVar = 'Hello world'
withEnv(['VAR1=VALUE ONE',
"VAR2=${someGroovyVar}"
]) {
def result = sh(script: 'echo $VAR1; echo $VAR2', returnStdout: true)
echo result
}
}
For secrets / passwords you can use credentials binding plugin
Example:
NOTE: CREDENTIALS_ID1 is a registered username/password secret on the Jenkins settings.
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
echo "User name: $USER"
echo "Password: $PASSWORD"
}
}
The jenkisn console log output hides the real values:
[Pipeline] echo
User name: ****
[Pipeline] echo
Password: ****
Jenkins and credentials is a big issue, probably see: credentials plugin
For completeness: Most of the time, we need the secrets in environment variables, as we use them from shell scripts, so we combine the withCredentials and withEnv like follows:
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
withEnv(["ENV_USERNAME=${USER}",
"ENV_PASSWORD=${PASSWORD}"
]) {
def result = sh(script: 'echo $ENV_USERNAME', returnStdout: true)
echo result
}
}
}
Another way to resolve this install 'Pipeline Utility Steps' plugin that provides us readProperties method ( for reference please go to the link https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#pipeline-utility-steps)
Here in the example we can see that they are storing the keys into an array and using the keys to retrieve the value.
But in that case the in production the problem will be like if we add any variable later into property file that variable needs to be added into the array of Jenkins file as well.
To get rid of this tight coupling, we can write code in such a way so that the Jenkins build environment can get information automatically about all the existing keys which presents currently in the Property file. Here is an example for the reference
def loadEnvironmentVariables(path){
def props = readProperties file: path
keys= props.keySet()
for(key in keys) {
value = props["${key}"]
env."${key}" = "${value}"
}
}
And the client code looks like
path = '\\ABS_Output\\EnvVars\\pic_env_vars.properties'
loadEnvironmentVariables(path)
With declarative pipeline, you can do it in one line ( change path by your value):
script {
readProperties(file: path).each {key, value -> env[key] = value }
}
Using withEnv() to pass environment variables from file splitted by new line and casted to List:
writeFile file: 'version.txt', text: 'version=6.22.0'
withEnv(readFile('version.txt').split('\n') as List) {
sh "echo ${version}"
}
If you are using Jenkins 2.0 you can load the property file (which consists of all required Environment variables along with their corresponding values) and read all the environment variables listed there automatically and inject it into the Jenkins provided env entity.
Here is a method which performs the above stated action.
def loadProperties(path) {
properties = new Properties()
File propertiesFile = new File(path)
properties.load(propertiesFile.newDataInputStream())
Set<Object> keys = properties.keySet();
for(Object k:keys){
String key = (String)k;
String value =(String) properties.getProperty(key)
env."${key}" = "${value}"
}
}
To call this method we need to pass the path of property file as a string variable For example, in our Jenkins file using groovy script we can call like
path = "${workspace}/pic_env_vars.properties"
loadProperties(path)
Please ask me if you have any doubt
Here is a complete example of externalizing environment variables and loading them in Jenkins pipeline execution. The pipeline is written in a declarative style.
stage('Reading environment variable defined in groovy file') {
steps {
script {
load "./pipeline/basics/extenvvariable/env.groovy"
echo "${env.env_var1}"
echo "${env.env_var2}"
}
}
}
Complete code example:
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/Jenkinsfile
Where variables are loaded from a groovy file placed with the pipeline code only.
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/env.groovy
This pattern comes very handy when you are creating a generic pipeline that could be used across teams.
You can externalize the dependent variable in such groovy file and each team can define their values according to their ecosystem.
Another solution is to use a custom method without allowing extra permissions such as for new Properties() which leads to this error before allowing:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new java.util.Properties
or adding extra plugin methods such as readProperties.
here is a method which reads a simple file named env_vars in this format:
FOO=bar
FOO2=bar
pipeline {
<... skipped lines ...>
script {
loadEnvironmentVariablesFromFile("env_vars")
echo "show time! ${BAR} ${BAR2}"
}
<... skipped lines ...>
}
private void loadEnvironmentVariablesFromFile(String path) {
def file = readFile(path)
file.split('\n').each { envLine ->
def (key, value) = envLine.tokenize('=')
env."${key}" = "${value}"
}
}