Hiding passwords in Jenkins Pipeline log output without using WithCredentials - jenkins

I have a parametrized Jenkins pipeline based on a Jenkinsfile. Some of the parameters contain sensitive passwords that I don't want to appear in the job's build logs.
So my question is: can I somehow register a String within the Jenkinsfile that is then replaced - by let's say ********** - whenever it appears in the log output?
I am aware of the withCredentials step, but I can't use it, since the credentials are not stored in the Jenkins credentials store (but provided as parameters at runtime).
I found this answer here https://stackoverflow.com/a/42372859/1549950 and tried it like this:
def secrets = [
[password: firstPassword, var: 'SECRET'],
[password: secondPassword, var: 'SECRET'],
[password: thirdPassword, var: 'SECRET']
]
node() {
wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: secrets]) {
// my stages containing steps...
}
}
Where firstPassword, secondPassword, thirdPassword are variables containing my passwords. But still I get the content of firstPassword... displayed plain text in the log output.
I have the Mask Password plugin installed on my Jenkins in version 2.12.0.
Basically I am searching for something like this: https://issues.jenkins-ci.org/browse/JENKINS-27486 - ticket is resolved, but no sample snippet of final implementation is given.

Actually I don't know why this didn't work in the first place, but here is the solution to the problem.
Define an array with secrets that you want to hide like this:
def splunkPassword = 'verySecretPa55w0rd'
def basicAuthPassword = 'my8asicAuthPa55w0rd'
def getSecrets() {
[
[password: splunkPassword, var: 'SECRET'],
[password: basicAuthPassword, var: 'SECRET']
]
}
Disclaimer: I don't know whether the SECRET value has an important role, copy and pasted it from some snippet and it works as expected :)
Afterwards, you can wrap any calls in your scripted pipeline like this:
node {
wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: getSecrets()]) {
stage 'First Stage' { ... }
stage 'Second Stage' { ... }
}
}
All passwords provided in the getSecrets() array will then be masked like this in your build output:
SPLUNK_PASSWORD: ********
BASIC_AUTH_ADMIN_PASSWORD: ********

I think you are looking for JENKINS-36007?

Update 26 May 2020
The workaround below stopped working for me recently. My guess is that something changed in a recent Jenkins update. I was trying to avoid installing another plugin, but I eventually gave up and installed the Mask Passwords plugin.
I used the following syntax for use with parameters:
parameters {
string(name: 'USERNAME', defaultValue: '', description: 'Username')
password(name: 'PASSWORD', defaultValue: '', description: 'Password')
}
Then in the build stage:
steps {
script {
wrap([$class: 'MaskPasswordsBuildWrapper',
varPasswordPairs: [
[password: "${USERNAME}", var: 'USR'],
[password: "${PASSWORD}", var: 'PSW']
]
]) {
sh '''
echo "Username: ${USERNAME}"
echo "Password: ${PASSWORD}"
'''
}
}
}
The original workaround is below, in case anyone else tries to go down the same path.
I've discovered a workaround that is a bit of a hack, but seems to work well. The trick is to use withCredentials, but override the variable with a parameter.
Here's an example which uses the environment directive's credentials() helper method to populate an environment variable, then overrides the two additional environment variables that are automatically defined (and masked in the logs).
First, create a dummy Username with password Credentials. The Username and Password values don't matter, we just need a Credential to use as a placeholder. Enter an ID such as dummy-credentials.
Then define an environment variable using the dummy credentials, and override the automatically defined variables with the parameters (MYUSERNAME and MYPASSWORD in this example):
environment {
MY_CREDS = credentials('dummy-credentials')
MY_CREDS_USR = "${params.MYUSERNAME}"
MY_CREDS_PSW = "${params.MYPASSWORD}"
}
Use the MY_CREDS_USR and MY_CREDS_PSW environment variables wherever you need to reference the secrets. Their contents will be masked in the console log.
sh '''
echo "Username: ${MY_CREDS_USR}"
echo "Password: ${MY_CREDS_PSW}"
'''

You might have a look at https://github.com/jenkinsci/log-file-filter-plugin
This plugin allows filtering Jenkins' console output by means of regular expressions. If some pattern matches the matched string is replaced by a string that can be specified for each pattern in the configuration.
Currently the plugin doesn't support adding filter-patterns from a jenkinsfile but only from the Jenkins global settings.

Highly brutish workaround.
Write a simple script, e.g. bash, and echo the parameter credentials into some file of arbitrary format, down to your echoing approach.
E.g. basic shell script:
$ cat executor/obfuscate.sh
#!/bin/bash
echo "PASSWORD: ${AWX_PW}" > ./executor/credential.yml
In your pipeline then:
stages {
stage('Placing') {
steps {
**sh './executor/obfuscate.sh'** }
[...]
< something reading credential.yml>
}
}
Outcome, nothing showing up in console:

Related

Jenkins - Hide sensitive data from pipeline steps page (flowGraphTable)

I am writing a sensitive data to a .yml file as a string in my Jenkins groovy file. I have attached the .groovy file which would show what I am trying to achieve but it reveals the data in Pipeline steps which is also attached. Nothing shows in the console log. Value from AWS Parameter store is retrieved and passed as a string to a .yml file and this string value is shown in the pipeline steps. I am looking for a way to hide this from being shown in the Pipeline steps. I have tried using set +x but it is not hiding the string value in the pipeline steps. Any thoughts on how this can be achieved would be of great help.Screenshot
#!/usr/bin/env groovy
properties([
parameters([
string(name:'NODE_NAME', defaultValue: 'node', description: 'Node Name')
])
])
nodeName = params.NODE_NAME
node("${nodeName}") {
def AWS_REGION = "eu-west-2"
def paramPath = "path"
def paramName = "PASSWORD"
stage("Data from SSM") {
param = sh (script: "aws ssm get-parameter --name ${paramPath} --with-decryption --region ${AWS_REGION} | /var/jenkins_home/tools/jq .Parameter.Value | tr -d '\"'", returnStdout: true).trim()
sh('#!/bin/sh -e\n' + "echo '${paramName}: ${param}' >> vars.yml")
}
}
You can make use of credentials in jenkins. Go to Jenkins -> credentials -> global -> add credentials
Select either 'Username with password'/'secret text' depending on your need
This will generate a secret in jenkins
Then in your groovy file you can use the credentials as env variables which will not be printed by jenkins
withCredentials([usernamePassword(credentialsId: 'Your-Credential-ID', passwordVariable: 'pwdToken', usernameVariable: 'userToken')]){
//Use pwdToken as a variable here which will not be printed
}

Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods write java.io.File java.lang.String

I'm trying to create vault-deployment using Jenkins. Here's a link to my repo.
When running the script I'm getting
"Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods write java.io.File java.lang.String. Administrators can decide whether to approve or reject this signature." issue.
I got this issue after adding a stage "Generate Vars".
If I remove this stage in the code the other stages works, but they don't complete the job. This is because it needs to get token for vault deployment and it needs to get it from .tfvars file.
It's not a good idea to share my variables on GitHub, that's why I`m trying to create vault.tfvars through Jenkins and provide any token before running a pipeline job.
Does anyone know how to fix this???
If some part is not clear please feel free to ask questions!
If I find the solution for this issue I will share it here with the link to my GitHub.
Thanks
Here is my code Jenkinsfile.groovy
node('master') {
properties([parameters([
string(defaultValue: 'plan', description: 'Please provide what action you want? (plan,apply,destroy)', name: 'terraformPlan', trim: true),
string(defaultValue: 'default_token_add_here', description: 'Please provide a token for vault', name: 'vault_token', trim: true)
]
)])
checkout scm
stage('Generate Vars') {
def file = new File("${WORKSPACE}/vaultDeployment/vault.tfvars")
file.write """
vault_token = "${vault_token}"
"""
}
stage("Terraform init") {
dir("${workspace}/vaultDeployment/") {
sh 'ls'
sh 'pwd'
sh "terraform init"
}
stage("Terraform Plan/Apply/Destroy"){
if (params.terraformPlan.toLowerCase() == 'plan') {
dir("${workspace}/vaultDeployment/") {
sh "terraform plan -var-file=variables.tfvars"
}
}
if (params.terraformPlan.toLowerCase() == 'apply') {
dir("${workspace}/vaultDeployment/") {
sh "terraform apply --auto-approve"
}
}
if (params.terraformPlan.toLowerCase() == 'destroy') {
dir("${workspace}/vaultDeployment/") {
sh "terraform destroy --auto-approve"
}
}
}
}
}
Generally, we choose pipeline to execute in Groovy sandbox which has restriction in some aspects for security considering. Like using new keyword, using static method.
But you need Jenkins admin to add the restriction to whitelist in jenkins > Manage jenkins > In-process Script Approval
To write file, Jenkins pipeline supply alternative writeFile which has no such restriction.
writeFile file: '<file path>', text: """
vault_token = "${vault_token}"
"""
As #yong already pointed out the right way to achieve this and avoid eventual restrictions in environments where we don't have admin control is to use writeFile
i.e.:
writeFile file: 'tmp/query.sql', text: "SELECT * FROM table"
Advantage of this is that migrating from fully managed to restricted environment will be painless.
Subfolders, like 'tmp' in example, will be automatically created and code itself is pretty verbose

Access Jenkins credential store secrets with env groovy file

I've implemented loading a groovy file for env variables by following this post.
env.groovy:
env.DB_USER = 'testuser'
env.DB_PASS = credentials('DB_PASS')
Jenkinsfile:
stages {
stage ("print") {
steps {
load "${WORKSPACE}/env.groovy"
echo "${env.DB_USER}"
echo "${env.DB_PASS}"
}
}
}
Output:
[Pipeline] echo
testuser
[Pipeline] echo
#credentials(<anonymous>=DB_PASS)
Is accessing the Jenkins credential store possible for DB_PASS when loading a groovy env variable file?
Note: I know I can access the environment variables in the environment { } block of my Jenkinsfile. But since I have so many env variables, I was wondering if I could reference them all in a separate groovy file instead.
For the time being, try to do it like this inside your jenkinsfile.
environment {
DB_PASS = credentials('DB_PASS')
}
However you still can't echo your DB_PASS environment variable since credentials type variables retain the property of secrecy. And also keep in mind that variables have scope in jenkinsfile.
If you have
environment {
DB_PASS = credentials('DB_PASS')
}
If you really would like to see the contents of that
you could do
sh """
echo "$DB_PASS" | base64
"""
That will show you the secret, hope this helps.

How to get environment variable in Jenkins email notification?

In Jenkins job configuration I have written a bat script in the command window of build section. In one of the script commands I set an environment variable as a system environment variable in the server machine as so:
setx Analysis_URL "http://analysis_url/analysis/%analysis_id%.html
My task now is to get this environment variable value back to Jenkins and include it in my post build notification email content. Is there a simple way to do it ?
In my research I have come across the plugin envInject but I think it is used for setting environment variables, is that right ?
UPDATE 1 :
It turned out that the variable could be accessed by a simple $Analysis_URL in the email content, however, that raised another issue as my environment variable changes its value after each job build, but as Jenkins only takes a copy of the system environment variables I keep getting the same variable value after each build in my email content, it only changes after restarting Jenkins. Is there a way to get the updated system environment variables to Jenkins ?
UPDATE 2 :
EnvInject plugin did the job I wanted. These are the steps that I performed:
Build step "batch command window": added command :
echo ANALYSIS_URL=$ANALYSIS_URL > my.properties
Build step "Inject environment variables": in field "Properies File Path"
$WORKSPACE/my.properties
Post-Build Actions: "Editable Email Notification", Field "Default Content":
Current analysis url: $ANALYSIS_URL
Have you tried the env.Analysis_URL to access to your environment variable ?
Printing Variables in Post Build Email:
Add this to build: execute shell section:
echo count=$count > count.txt
echo distinctcount=$distinctcount > distinctcount.txt
Add this to Inject Environmental Variables section under "Properties File Path":
${WORKSPACE}/count.txt
and do the same also for the second file ${WORKSPACE}/distinctcount.txt (as I have two variables)
Note 1: ${WORKSPACE} is pwd for jenkins and lists the location)
Note 2: You can actually put both variables in just one file by using echo distinctcount=$distinctcount >> count.txt
Call the variable in the "default content" section of "editable email notification":
Count total:$count Distinct Count: $distinctcount
NOTE
Be sure you have email setup as "always" under "advanced settings" at the bottom of the email settings, otherwise it only emails upon first success or any failure:
You can use environment variables and global variables inside script with sh. For example:
pipeline {
environment {
APP_VERSION = "1.2.1"
}
post {
success {
script {
sh "sed -i 's#%BUILD_URL#$BUILD_URL#g' .jenkins/email.html"
sh "sed -i 's#%APP_VERSION#${APP_VERSION}#g' .jenkins/email.html"
emailext attachLog: true, mimeType: 'text/html', body: '${FILE, path=".jenkins/email.html"}', subject: '[$BUILD_STATUS] $PROJECT_NAME', to: 'youremail#mail.com'
}
}
}
}
But the secret is here:
' single quote parses global variables like $BUILD_STATUS $BUILD_URL ...
" double quote parses env variables like ${APP_VERSION}
So, you can use:
subject: '$BUILD_STATUS'
or
subject: "${APP_VERSION}"
Final solution
Transform all global variables you need to environment variables:
pipeline {
environment {
APP_VERSION = "1.2.1"
}
post {
success {
script {
BUILD_STATUS = '$BUILD_STATUS'
PROJECT_NAME = '$PROJECT_NAME'
sh "sed -i 's#%BUILD_URL#$BUILD_URL#g' .jenkins/email.html"
sh "sed -i 's#%APP_VERSION#${APP_VERSION}#g' .jenkins/email.html"
emailext attachLog: true, mimeType: 'text/html', body: '${FILE, path=".jenkins/email.html"}', subject: "${BUILD_STATUS} ${PROJECT_NAME} ${APP_VERSION}", to: 'youremail#mail.com'
}
}
}
}
You can just use: ${yourDefinedVriableName}

Load file with environment variables Jenkins Pipeline

I am doing a simple pipeline:
Build -> Staging -> Production
I need different environment variables for staging and production, so i am trying to source variables.
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh'
But it returns Not found
[Stack Test] Running shell script
+ source /var/jenkins_home/.envvars/stacktest-staging.sh
/var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: 2: /var/jenkins_home/workspace/Stack Test#tmp/durable-bcbe1515/script.sh: source: not found
The path is right, because i run the same command when i log via ssh, and it works fine.
Here is the pipeline idea:
node {
stage name: 'Build'
// git and gradle build OK
echo 'My build stage'
stage name: 'Staging'
sh 'source $JENKINS_HOME/.envvars/stacktest-staging.sh' // PROBLEM HERE
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To staging
input message: "Does Staging server look good?"
stage name: 'Production'
sh 'source $JENKINS_HOME/.envvars/stacktest-production.sh'
echo '$DB_URL' // Expects http://production_url/my_db
sh 'gradle flywayMigrate' // To production
sh './deploy.sh'
}
What should i do?
I was thinking about not using pipeline (but i will not be able to use my Jenkinsfile).
Or make different jobs for staging and production, using EnvInject Plugin (But i lose my stage view)
Or make withEnv (but the code gets big, because today i am working with 12 env vars)
One way you could load environment variables from a file is to load a Groovy file.
For example:
Let's say you have a groovy file in '$JENKINS_HOME/.envvars' called 'stacktest-staging.groovy'.
Inside this file, you define 2 environment variables you want to load
env.DB_URL="hello"
env.DB_URL2="hello2"
You can then load this in using
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
Then you can use them in subsequent echo/shell steps.
For example, here is a short pipeline script:
node {
load "$JENKINS_HOME/.envvars/stacktest-staging.groovy"
echo "${env.DB_URL}"
echo "${env.DB_URL2}"
}
From the comments to the accepted answer
Don't use global 'env' but use 'withEnv' construct, eg see:
issue #9: don't set env vars with global env in top 10 best practices jenkins pipeline plugin
In the following example: VAR1 is a plain java string (no groovy variable expansion), VAR2 is a groovy string (so variable 'someGroovyVar' is expanded).
The passed script is a plain java string, so $VAR1 and $VAR2 are passed literally to the shell, and the echo's are accessing environment variables VAR1 and VAR2.
stage('build') {
def someGroovyVar = 'Hello world'
withEnv(['VAR1=VALUE ONE',
"VAR2=${someGroovyVar}"
]) {
def result = sh(script: 'echo $VAR1; echo $VAR2', returnStdout: true)
echo result
}
}
For secrets / passwords you can use credentials binding plugin
Example:
NOTE: CREDENTIALS_ID1 is a registered username/password secret on the Jenkins settings.
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
echo "User name: $USER"
echo "Password: $PASSWORD"
}
}
The jenkisn console log output hides the real values:
[Pipeline] echo
User name: ****
[Pipeline] echo
Password: ****
Jenkins and credentials is a big issue, probably see: credentials plugin
For completeness: Most of the time, we need the secrets in environment variables, as we use them from shell scripts, so we combine the withCredentials and withEnv like follows:
stage('Push') {
withCredentials([usernamePassword(
credentialsId: 'CREDENTIALS_ID1',
passwordVariable: 'PASSWORD',
usernameVariable: 'USER')]) {
withEnv(["ENV_USERNAME=${USER}",
"ENV_PASSWORD=${PASSWORD}"
]) {
def result = sh(script: 'echo $ENV_USERNAME', returnStdout: true)
echo result
}
}
}
Another way to resolve this install 'Pipeline Utility Steps' plugin that provides us readProperties method ( for reference please go to the link https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#pipeline-utility-steps)
Here in the example we can see that they are storing the keys into an array and using the keys to retrieve the value.
But in that case the in production the problem will be like if we add any variable later into property file that variable needs to be added into the array of Jenkins file as well.
To get rid of this tight coupling, we can write code in such a way so that the Jenkins build environment can get information automatically about all the existing keys which presents currently in the Property file. Here is an example for the reference
def loadEnvironmentVariables(path){
def props = readProperties file: path
keys= props.keySet()
for(key in keys) {
value = props["${key}"]
env."${key}" = "${value}"
}
}
And the client code looks like
path = '\\ABS_Output\\EnvVars\\pic_env_vars.properties'
loadEnvironmentVariables(path)
With declarative pipeline, you can do it in one line ( change path by your value):
script {
readProperties(file: path).each {key, value -> env[key] = value }
}
Using withEnv() to pass environment variables from file splitted by new line and casted to List:
writeFile file: 'version.txt', text: 'version=6.22.0'
withEnv(readFile('version.txt').split('\n') as List) {
sh "echo ${version}"
}
If you are using Jenkins 2.0 you can load the property file (which consists of all required Environment variables along with their corresponding values) and read all the environment variables listed there automatically and inject it into the Jenkins provided env entity.
Here is a method which performs the above stated action.
def loadProperties(path) {
properties = new Properties()
File propertiesFile = new File(path)
properties.load(propertiesFile.newDataInputStream())
Set<Object> keys = properties.keySet();
for(Object k:keys){
String key = (String)k;
String value =(String) properties.getProperty(key)
env."${key}" = "${value}"
}
}
To call this method we need to pass the path of property file as a string variable For example, in our Jenkins file using groovy script we can call like
path = "${workspace}/pic_env_vars.properties"
loadProperties(path)
Please ask me if you have any doubt
Here is a complete example of externalizing environment variables and loading them in Jenkins pipeline execution. The pipeline is written in a declarative style.
stage('Reading environment variable defined in groovy file') {
steps {
script {
load "./pipeline/basics/extenvvariable/env.groovy"
echo "${env.env_var1}"
echo "${env.env_var2}"
}
}
}
Complete code example:
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/Jenkinsfile
Where variables are loaded from a groovy file placed with the pipeline code only.
https://github.com/dhruv-bansal/jenkins-pipeline-exploration/blob/master/pipeline/basics/extenvvariable/env.groovy
This pattern comes very handy when you are creating a generic pipeline that could be used across teams.
You can externalize the dependent variable in such groovy file and each team can define their values according to their ecosystem.
Another solution is to use a custom method without allowing extra permissions such as for new Properties() which leads to this error before allowing:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new java.util.Properties
or adding extra plugin methods such as readProperties.
here is a method which reads a simple file named env_vars in this format:
FOO=bar
FOO2=bar
pipeline {
<... skipped lines ...>
script {
loadEnvironmentVariablesFromFile("env_vars")
echo "show time! ${BAR} ${BAR2}"
}
<... skipped lines ...>
}
private void loadEnvironmentVariablesFromFile(String path) {
def file = readFile(path)
file.split('\n').each { envLine ->
def (key, value) = envLine.tokenize('=')
env."${key}" = "${value}"
}
}

Resources