I need to take env vars for different apis from a repo that contains all the .env files(one per api). For example:
Inside /env_file directory:
foo.env
bar.env
Is it possible to use a .env file located in a repo to get the env vars instead of having them set manually in environment {}? In case it is, how could be referenced each file in each api stage?
Thanks in advance! Never worked in Jenkinsfile like this before. Sorry if this is a weird question.
You can inject environment into Jenkins pipeline's global variable: env which is a Map and the wirepuller of environment {}, but it has a limitation:
Allow insert new environment, NOT allow override existing environment
If your those .env files has same environment and you need to load them together, you will run into 'override existing environment' issue.
If you're not in that case, you can do as following:
pipeline {
stages {
stage('Test endpoint A') {
script {
def props = readProperties file: 'A.env'
for (p in props) {
env[p.key] = p.value
}
}
// other steps
}
stage('Test endpoint B') {
script {
def props = readProperties file: 'B.env'
for (p in props) {
env[p.key] = p.value
}
}
// other steps
}
}
}
Related
Hello Trying to update jenkins Env variables from Powershell script written in stage
pipeline {
agent { label 'master' }
environment {
def var1 = "default_value"
def var2 = "default var 2"
}
stages {
stage ('step-1') {
steps {
script {
powershell (returnStdout: true, script: '''
${env:var1} = "changed from powershell"
${env:var2} = "var2 changed from powershell"
''').trim()
}
}
}
stage ("step-2"){
steps {
echo var1
echo var2
}
}
}
}
I can access values but cannot change the same, Does Scope is limited to Powershell only?
By design, a subshell cannot modify the environment of its parent shell. Every powershell, sh, or bat step called within a Jenkinsfile spawns a new subshell to run the child processes. Therefore, any environment variable declared inside these steps are lost as soon as the child processes end and are not available to the pipeline Groovy script.
One way to do this currently in a Jenkinsfile is to assign the stdout of your subshell to a variable in the Groovy script itself.
script {
env.var1 = powershell(returnStdout: true, script: '<single command>').trim()
}
Evidently, this approach has serious limitations as you can run only one command at a time and that too, only if it would return an appropriate output to be used as a variable.
Another way is to write the values to a file and then read these values using Groovy script to set the environment variables downstream in your pipeline.
Edit
Based on your comment, if your stdout returns two distinct values as a string, you can try parsing them into separate values. For example:
script {
def output = powershell(returnStdout: true, script: '''
<command1>
<command2>
''')
println(output)
/*Prints:
value1
value2
*/
env.var1 = output.tokenize('\n')[0].trim() \\value1
env.var2 = output.tokenize('\n')[1].trim() \\value2
}
I have configfile, which is a JSON file. I want to be able to read it before any steps, as it provides variables I need to execute them. However, I don't know where do I put that. To contain config file provider call, I tried creating a separate node before pipeline, to no avail, also tried to set up script in stages, stage (also as post).
I did a simple practice on my jenkins as following.
def config;
node(){
configFileProvider([configFile(fileId: '<your config file id>', targetLocation: 'myConfig')]) {
config = readJSON file: 'myConfig'
}
}
pipeline {
agent any
stages {
stage('Build') {
steps {
echo config.myKey // or config['myKey']
}
}
}
}
I've implemented loading a groovy file for env variables by following this post.
env.groovy:
env.DB_USER = 'testuser'
env.DB_PASS = credentials('DB_PASS')
Jenkinsfile:
stages {
stage ("print") {
steps {
load "${WORKSPACE}/env.groovy"
echo "${env.DB_USER}"
echo "${env.DB_PASS}"
}
}
}
Output:
[Pipeline] echo
testuser
[Pipeline] echo
#credentials(<anonymous>=DB_PASS)
Is accessing the Jenkins credential store possible for DB_PASS when loading a groovy env variable file?
Note: I know I can access the environment variables in the environment { } block of my Jenkinsfile. But since I have so many env variables, I was wondering if I could reference them all in a separate groovy file instead.
For the time being, try to do it like this inside your jenkinsfile.
environment {
DB_PASS = credentials('DB_PASS')
}
However you still can't echo your DB_PASS environment variable since credentials type variables retain the property of secrecy. And also keep in mind that variables have scope in jenkinsfile.
If you have
environment {
DB_PASS = credentials('DB_PASS')
}
If you really would like to see the contents of that
you could do
sh """
echo "$DB_PASS" | base64
"""
That will show you the secret, hope this helps.
In Jenkins scripted pipeline you can set PATH env variable like this :
node {
git url: 'https://github.com/jglick/simple-maven-project-with-tests.git'
withEnv(["PATH+MAVEN=${tool 'M3'}/bin"]) {
sh 'mvn -B verify'
}
}
Notice the PATH+MAVEN as explained here https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-withenv-code-set-environment-variables :
A list of environment variables to set, each in the form
VARIABLE=value or VARIABLE= to unset variables otherwise defined. You
may also use the syntax PATH+WHATEVER=/something to prepend /something
to $PATH.
But I didn't find how to do it in declarative pipeline using environment syntax (as explained here : https://jenkins.io/doc/pipeline/tour/environment).
environment {
DISABLE_AUTH = 'true'
DB_ENGINE = 'sqlite'
}
Ideally I would like to update the PATH to use custom tools for all my stages.
It is possible with environment section:
pipeline {
agent { label 'docker' }
environment {
PATH = "/hot/new/bin:${env.PATH}"
}
stages {
stage ('build') {
steps {
echo "PATH is: ${env.PATH}"
}
}
}
}
See this answer for info.
As a workaround, you can define an environment variable and use it in the sh step:
pipeline {
environment {
MAVEN_HOME = tool('M3')
}
stages {
stage(Maven') {
sh '${MAVEN_HOME}/bin/mvn -B verify'
}
}
}
Check the following link, this explains how to configure your tools.
Using the declarative pipeline things become a bit different but overall it is easier to understand.
declarative-maven-project
Using the tool section in pipeline is only allowed for pre-installed Global Tools. Some tools are provided by plugins, but if it not exists I'am afraid you cannot use the environment setup via pipeline tool declaration.
I hope to be wrong!
I am doing a simple pipeline:
Build -> Staging -> Production
I need different environment variables for staging and production, so i am trying to source variables.
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh'
But it returns Not found
[Stack Test] Running shell script
+ source /var/jenkins_home/.envvars/stacktest-staging.sh
/var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: 2: /var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: source: not found
The path is right, because i run the same command when i log via ssh, and it works fine.
Here is the pipeline idea:
node {
stage name: 'Build'
// git and gradle build OK
echo 'My build stage'
stage name: 'Staging'
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh' // PROBLEM HERE
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To staging
input message: "Does Staging server look good?"
stage name: 'Production'
sh 'source $JENKINS_HOME/.envvars/stacktest-production.sh'
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To production
sh './deploy.sh'
}
What should i do?
I was thinking about not using pipeline (but i will not be able to use my Jenkinsfile).
Or make different jobs for staging and production, using EnvInject Plugin (But i lose my stage view)
Or make withEnv (but the code gets big, because today i am working with 12 env vars)
One way you could load environment variables from a file is to load a Groovy file.
For example:
Let's say you have a groovy file in '$JENKINS_HOME/.envvars' called 'stacktest-staging.groovy'.
Inside this file, you define 2 environment variables you want to load
env.DB_URL="hello"
env.DB_URL2="hello2"
You can then load this in using
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
Then you can use them in subsequent echo/shell steps.
For example, here is a short pipeline script:
node {
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
echo "${env.DB_URL}"
echo "${env.DB_URL2}"
}
From the comments to the accepted answer
Don't use global 'env' but use 'withEnv' construct, eg see:
issue #9: don't set env vars with global env in top 10 best practices jenkins pipeline plugin
In the following example: VAR1 is a plain java string (no groovy variable expansion), VAR2 is a groovy string (so variable 'someGroovyVar' is expanded).
The passed script is a plain java string, so $VAR1 and $VAR2 are passed literally to the shell, and the echo's are accessing environment variables VAR1 and VAR2.
stage('build') {
def someGroovyVar = 'Hello world'
withEnv(['VAR1=VALUE ONE',
"VAR2=${someGroovyVar}"
]) {
def result = sh(script: 'echo $VAR1; echo $VAR2', returnStdout: true)
echo result
}
}
For secrets / passwords you can use credentials binding plugin
Example:
NOTE: CREDENTIALS_ID1 is a registered username/password secret on the Jenkins settings.
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
echo "User name: $USER"
echo "Password: $PASSWORD"
}
}
The jenkisn console log output hides the real values:
[Pipeline] echo
User name: ****
[Pipeline] echo
Password: ****
Jenkins and credentials is a big issue, probably see: credentials plugin
For completeness: Most of the time, we need the secrets in environment variables, as we use them from shell scripts, so we combine the withCredentials and withEnv like follows:
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
withEnv(["ENV_USERNAME=${USER}",
"ENV_PASSWORD=${PASSWORD}"
]) {
def result = sh(script: 'echo $ENV_USERNAME', returnStdout: true)
echo result
}
}
}
Another way to resolve this install 'Pipeline Utility Steps' plugin that provides us readProperties method ( for reference please go to the link https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#pipeline-utility-steps)
Here in the example we can see that they are storing the keys into an array and using the keys to retrieve the value.
But in that case the in production the problem will be like if we add any variable later into property file that variable needs to be added into the array of Jenkins file as well.
To get rid of this tight coupling, we can write code in such a way so that the Jenkins build environment can get information automatically about all the existing keys which presents currently in the Property file. Here is an example for the reference
def loadEnvironmentVariables(path){
def props = readProperties file: path
keys= props.keySet()
for(key in keys) {
value = props["${key}"]
env."${key}" = "${value}"
}
}
And the client code looks like
path = '\\ABS_Output\\EnvVars\\pic_env_vars.properties'
loadEnvironmentVariables(path)
With declarative pipeline, you can do it in one line ( change path by your value):
script {
readProperties(file: path).each {key, value -> env[key] = value }
}
Using withEnv() to pass environment variables from file splitted by new line and casted to List:
writeFile file: 'version.txt', text: 'version=6.22.0'
withEnv(readFile('version.txt').split('\n') as List) {
sh "echo ${version}"
}
If you are using Jenkins 2.0 you can load the property file (which consists of all required Environment variables along with their corresponding values) and read all the environment variables listed there automatically and inject it into the Jenkins provided env entity.
Here is a method which performs the above stated action.
def loadProperties(path) {
properties = new Properties()
File propertiesFile = new File(path)
properties.load(propertiesFile.newDataInputStream())
Set<Object> keys = properties.keySet();
for(Object k:keys){
String key = (String)k;
String value =(String) properties.getProperty(key)
env."${key}" = "${value}"
}
}
To call this method we need to pass the path of property file as a string variable For example, in our Jenkins file using groovy script we can call like
path = "${workspace}/pic_env_vars.properties"
loadProperties(path)
Please ask me if you have any doubt
Here is a complete example of externalizing environment variables and loading them in Jenkins pipeline execution. The pipeline is written in a declarative style.
stage('Reading environment variable defined in groovy file') {
steps {
script {
load "./pipeline/basics/extenvvariable/env.groovy"
echo "${env.env_var1}"
echo "${env.env_var2}"
}
}
}
Complete code example:
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/Jenkinsfile
Where variables are loaded from a groovy file placed with the pipeline code only.
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/env.groovy
This pattern comes very handy when you are creating a generic pipeline that could be used across teams.
You can externalize the dependent variable in such groovy file and each team can define their values according to their ecosystem.
Another solution is to use a custom method without allowing extra permissions such as for new Properties() which leads to this error before allowing:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new java.util.Properties
or adding extra plugin methods such as readProperties.
here is a method which reads a simple file named env_vars in this format:
FOO=bar
FOO2=bar
pipeline {
<... skipped lines ...>
script {
loadEnvironmentVariablesFromFile("env_vars")
echo "show time! ${BAR} ${BAR2}"
}
<... skipped lines ...>
}
private void loadEnvironmentVariablesFromFile(String path) {
def file = readFile(path)
file.split('\n').each { envLine ->
def (key, value) = envLine.tokenize('=')
env."${key}" = "${value}"
}
}