Automate Setting Jenkins Credentials and Global Variables - jenkins

I have Terraform+Ansible script that automates the deployment of a Jenkins server. However, I'd also like to pre-load that server with "Global" environment variables (Found in "Manage Jenkins" > "Configure System" > "Global Properties" > "Environment Variables"). I'm also looking to automatically set a few Jenkins credentials (found in "Manage Jenkins" > "Manage Credentials").
From what I understand, the credentials are stored encrypted in a file called /var/lib/jenkins/credentials.xml. And the global properties might be stored in config.xml in the same Jenkins directory, though the variables don't look very structured. I might be able to parse out the config.xml and add some values. But I'm wondering how I can create an encrypted secret that can be added to the credentials.xml.
Are there any tools or strategies for automating the creation of secrets and environment variables in Jenkins? I'm hoping that there's a better way to automate this other than parsing the xml documents. Any advice is much appreciated.

You shouldn't be manually altering these files. You can use one of the following options to automate Credential and Global variable creation.
Jenkins API.
Jenkins CLI.
Python-jenkinsapi. (Python wrapper for Jenkins API)
How to use Jenkins API to create credentials
Following commands need curl and jq. Execute in the same session.
# Change the following appropriately
JENKINS_URL="http://localhost:8080"
JENKINS_USER=admin
JENKINS_USER_PASS=admin
Get the Crumb
JENKINS_CRUMB=$(curl -u "$JENKINS_USER:$JENKINS_USER_PASS" -s --cookie-jar /tmp/cookies $JENKINS_URL'/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
Get the Access token
ACCESS_TOKEN=$(curl -u "$JENKINS_USER:$JENKINS_USER_PASS" -H $JENKINS_CRUMB -s \
--cookie /tmp/cookies $JENKINS_URL'/me/descriptorByName/jenkins.security.ApiTokenProperty/generateNewToken' \
--data 'newTokenName=GlobalToken' | jq -r '.data.tokenValue')
Create Credentials
curl -u $JENKINS_USER:$ACCESS_TOKEN \
-H $JENKINS_CRUMB \
-H 'content-type:application/xml' \
"$JENKINS_URL/credentials/store/system/domain/_/createCredentials" \
-d '<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<id>TestCredentials</id>
<description>This is sample</description>
<username>admin2</username>
<password>admin2</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>'
Note: The credential create URL format is /credentials/store/CREDENTIALS_STORE_NAME/domain/DOMAIN_NAME/ Change this appropriately if you want to create the Credential in a custom location. The easiest way to get this URL is by navigating to a existing credential from the UI and copying the URL.
How to create Credentials from Jenkins CLI
# Change the following appropriately
JENKINS_URL="http://localhost:8080"
JENKINS_USER=admin
JENKINS_USER_PASS=admin
Download the Jenkins CLI Jar
wget $JENKINS_URL/jnlpJars/jenkins-cli.jar
Create a file named cred.xml with the following content.
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>PassID</id>
<username>Username</username>
<password>password</password>
<description>Description</description>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
Create credentials
java -jar jenkins-cli.jar -s $JENKINS_URL -auth $JENKINS_USER:$JENKINS_USER_PASS create-credentials-by-xml system::system::jenkins _ < cred.xml
How to create credentials with Jenkins Python API
Following is a simple sample to create Global credentials using Jenkins Python API.
from api4jenkins import Jenkins
j = Jenkins('http://localhost:8080', auth=('admin', 'admin'))
xmlPayload = '''<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<id>user-id</id>
<username>user</username>
<password>upassword</password>
<description>user id for testing</description>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>'''
j.credentials.create(xmlPayload)

Related

How can I use a linux system variable in my Jenkinsfile?

I want to use a curl with a username and password that I set in the bashrc. ie:
curl -u $jenkuser:$jenkpass foobar.org
but this isn't working for me. So what is a good way to set secret credentials that I don't want in my repo/Jenkinsfile
Create a Jenkins project with Execute shell build step. In that shell you can run curl command and to set credentials, there is one option named This build is parameterized, where you can create Password Parameter. These parameters can be used in shell with curl command. Here is screenshot of my test project.
This way is secure because password is stored in encrypted format.
.

No Valid crumb is included in that request

I have one jenkins server A where I am trying to create a scripted pipeline but I have to call another job (Job1) on another jenkins server B.
In order to do that I am using REST API with crumb in Header. I retrieve my crumb by running the following command on my browser.
http://myhudson.com/crumbIssuer/api/json?xpath=concat(//crumbRequestField,":",//crumb)"
I tried different commands mentioned below but there is no luck.Please advise some thing. I do have access or permission for triggering build on both of the servers. I am executing below commands from Jenkins server A with details of Server B.
1. curl -v -u Username:<API_TOKEN> -X POST http://UsedrName:<API_TOKEN>#myjenkins.com/job/Test_job/build?token=<API_TOKEN> -H Jenkins-Crumb:<Crumb number>
2. curl -v -X POST http://UsedrName:<API_TOKEN>#myjenkins.com/job/Test_job/build?token=<API_TOKEN> -H Jenkins-Crumb:<Crumb number>
3. curl -v -u Username:<API_TOKEN> -X POST http://UsedrName:<API_TOKEN>#myjenkins.com/job/Test_job/build?token=<API_TOKEN> -H .crumb:<Crumb number>
did jenkins API work for you, https://wiki.jenkins.io/display/JENKINS/Remote+access+API you can trigger remote jobs
You need to use Build With Parameters Plugin for it . you can write a shell script in build option to call the another jenkins job on other server.
shell script snippet :curl -X POST -u userid of other server:API token of other server {jenkins server url of B job /job/jobname}
To get rid of this error i got Jenkins-Crumb by using wget command.
Trigger parameterized build with curl and crumb
I was able to execute remote parameter job.
I faced the same issue and it was because on that port something else is running and jenkins is giving Error as "No Valid crumb is included in that request" thus changed the port in server.xml and things worked.

How to export credentials from one jenkins instance to another?

I am using the credentials plugin in Jenkins to manage credentials for git and database access for my team's builds. I would like to copy the credentials from one jenkins instance to another, independent jenkins instance. How would I go about doing this?
UPDATE: TL;DR Follow the link provided below in a comment by Filip Stachowiak it is the easiest way to do it. In case it doesn't work for you go on reading.
Copying the $HUDSON_HOME/credentials.xml is not the solution because Jenkins encrypts paswords and these can't be decrypted by another instance unless both share a common key.
So, either you use the same encription keys in both Jenkins instances (Where's the encryption key stored in Jenkins? ) or what you can do is:
Create the same user/password, you need to share, in the 2nd Jenkins instance so that a valid password is generated
What is really important is that user ids in both credentials.xml are the same. For that (see the credentials.xml example below) for user: Jenkins the identifier <id>c4855f57-5107-4b69-97fd-298e56a9977d</id> must be the same in both credentials.xml
<com.cloudbees.plugins.credentials.SystemCredentialsProvider plugin="credentials#1.22">
<domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash">
<entry>
<com.cloudbees.plugins.credentials.domains.Domain>
<specifications/>
</com.cloudbees.plugins.credentials.domains.Domain>
<java.util.concurrent.CopyOnWriteArrayList>
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>c4855f57-5107-4b69-97fd-298e56a9977d</id>
<description>Para SVN</description>
<username>jenkins</username>
<password>J1ztA2vSXHbm60k5PjLl5jg70ZooSFKF+kRAo08UVts=
</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
</java.util.concurrent.CopyOnWriteArrayList>
</entry>
</domainCredentialsMap>
</com.cloudbees.plugins.credentials.SystemCredentialsProvider>
I was also facing the same problem. What worked for me is I copied the credentials.xml, config.xml and the secrets folder from existing jenkins to the new instance. After the restart of jenkins things worked fine.
This is what worked for me.
Create a job in Jenkins that takes the credentials and writes them to output. If Jenkins replaces the password in the output with ****, just obfuscate it first (add a space between each character, reverse the characters, base64 encode it, etc.)
I used a Powershell job to base64 encode it:
[convert]::ToBase64String([text.encoding]::Default.GetBytes($mysecret))
And then used Powershell to convert the base64 string back to a regular string:
[text.encoding]::Default.GetString([convert]::FromBase64String("bXlzZWNyZXQ="))
After trying quite a few things for several days this is the best solution I found for migrating my secrets from a Jenkins 2.176 to a new clean Jenkins 2.249.1 jenkins-cli was the best approach for me.
The process is quite simple just dump the credentials from the old instance to a local machine, or Docker pod with java installed, as a XML file (unencrypted) and then uploaded to the new instance.
Before starting you should verify the following:
Access to the credentials section on both Jenkins instances
Download the jenkins-ccli.jar from one of the instances (https://www.your-jenkins-url.com/cli/)
Have User and Password/Token at hand.
Notice: In case your jenkins uses an oAuth service you will need to
create a token for your user. Once logged into jenkins at the top
right if you click your profile you can verify both username and
generate password.
Now for the special sauce, you have to execute both parts from the same machine/pod:
Notice: If your instances are using valid Certificates and you want to
secure your connection you must remove the -noCertificateCheck
flag from both commands.
# OLD JENKINS DUMP # 
export USER=madox#example.com
export TOKEN=f561banana6ead83b587a4a8799c12c307
export SERVER=https://old-jenkins-url.com/
java -jar jenkins-cli.jar -noCertificateCheck -s $SERVER -auth $USER:$TOKEN list-credentials-as-xml "system::system::jenkins" > /tmp/jenkins_credentials.xml
# NEW JENKINS IMPORT # 
export USER=admin
export TOKEN=admin
export SERVER=https://new-jenkins-url.com/
java -jar jenkins-cli.jar -noCertificateCheck -s $SERVER -auth $USER:$TOKEN import-credentials-as-xml "system::system::jenkins" < /tmp/jenkins_credentials.xml
If you have the credentials.xml available and the old Jenkins instance still running, there is a way to decrypt individual credentials so you can enter them in the new Jenkins instance via the UI.
The approach is described over at the DevOps stackexchange by kenorb.
This does not convert all the credentials for an easy, automated migration, but helps when you have only few credentials to migrate (manually).
To summarize, you visit the /script page over at the old Jenkins instance, and use the encrypted credential from the credentials.xml file in the following line:
println(hudson.util.Secret.decrypt("{EncryptedCredentialFromCredentialsXml=}"))
To migrate all credentials to a new server, from Jenkins: Migrating credentials:
Stop Jenkins on new server.
new-server # /etc/init.d/jenkins stop
Remove the identity.key.enc file on new server:
new-server # rm identity.key.enc
Copy secret* and credentials.xml to new server.
current-server # cd /var/lib/jenkins
current-server # tar czvf /tmp/credentials.tgz secret* credentials.xml
current-server # scp credentials.tgz $user#$new-server:/tmp/
new-server # cd /var/lib/jenkins
new-server # tar xzvf /tmp/credentials.tgz -C ./
Start Jenkins.
new-server # /etc/init.d/jenkins start
Migrating users from a Jenkins instance to another Jenkins on a new server -
I tried following https://stackoverflow.com/a/35603191 which lead to https://itsecureadmin.com/2018/03/26/jenkins-migrating-credentials/. However I did not succeed in following these steps.
Further, I experimented exporting /var/lib/jenkins/users (or {JENKINS_HOME}/users) directory to the new instance on new server. After restarting the Jenkins on new server - it looks like all the user credentials are available on new server.
Additionally, I cross-checked if the users can log in to the new Jenkins instance. It works for now.
PS: This code is for redhat servers
Old server:
cd /var/lib/jeknins
or cd into wherever your Jenkins home is
tar cvzf users.tgz ./users
New server:
cd /var/lib/jeknins
scp <user>#<oldserver>:/var/lib/jenkins/user.tgz ~/var/lib/jenkins/.
sudo tar xvzf users.tgz
systemctl restart jenkins
Did you try to copy the $JENKINS_HOME/users folder and the $JENKINS_HOME/credentials.xml file to the other Jenkins instance?

Using createItem remote API with CloudBees folder plugin to create nested jobs or subdirectories

I am attempting to create nested directories and jobs mirroring the structure of my SVN repository using the CloudBees Folder plugin and Jenkins remote API.
For a subdirectory (assume I've copied the config.xml to my CWD):
wget --no-proxy --auth-no-challenge --http-user=username --ask-password --header="Content-Type: application/xml" --post-file="dconfig.xml" 'http://ci.mycompany.com/jenkins/createItem?name=foo'
works. When the command completes, I have a newly created "foo" folder in the top level of my dashboard. However when I try:
wget --no-proxy --auth-no-challenge --http-user=username --ask-password --header="Content-Type: application/xml" --post-file="dconfig.xml" 'http://ci.mycompany.com/jenkins/createItem?name=foo/bar'
I get a "ERROR 400: Bad Request". I've also tried substituting "name=/foo/bar", "name=%2Ffoo%2Fbar", and "name=foo%2Fbar", all with the same results.
Likewise, I get a similar result when using the createItem/from/mode/name API to copy internal folders:
wget --no-proxy --auth-no-challenge --http-user=username --ask-password --header="Content-Type: application/xml" --post-file="dconfig.xml" 'http://ci.mycompany.com/jenkins/createItem?name=foo/bar&mode=copy&from=foo'
I find this very unusual since - through the web interface - I can easily move, copy, and create nested directories and jobs.
I am using Jenkins 1.569 with the CloudBees Folders 4.7 plugin.
name may only be a simple name. The endpoint you are looking for is on the folder, not at root level. Thus
wget --no-proxy --auth-no-challenge --http-user=username --ask-password --header="Content-Type: application/xml" --post-file="dconfig.xml" 'http://ci.mycompany.com/jenkins/job/foo/createItem?name=bar'

How to add accounts to Jenkins without the web interface?

I want to automate the entire installation of Jenkins, given a list of user names I want to be able to create user accounts for each. The only method I've read to set up user accounts is here:
https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup
Tried seeing if there was an option to configure with command line at:
https://localhost:8080/cli/
But does not seem to the be case.
Is it possible to add user accounts without using the web interface? More specifically a method that is scriptable.
My last resort is to do raw post requests but hoping there is a nicer way.
Yes of course, it is possible to script provisioning for jenkins. But not with the cli tool alone.
I guess you want to use "Jenkins own user database" with Project Matrix Authorization Strategy.
Steps to prepare provisioning:
Configure your Jenkins manually (enable security, add rolls and at least one user)
Shutdown your jenkins (to let him write all in-memory changes to disk)
Copy $JENKINS_HOME/config.xml to your provisioning script (as as seed data)
Copy $JENKINS_HOME/users/ (as seed data)
Get the cli tool: cd /tmp; wget -nv http://localhost:8080/jnlpJars/jenkins-cli.jar
If you do not want to have static seed data (one config.xml for each user) you can generate a (users/username/)config.xml using a bash script or a more advanced tool. But for simplicity sake you can take users/username1/config.xml as a template. Replace relevant data with a placeholder e.g. "PLACEHOLDER_FULLNAME" for full user name.
e.g.:
change
"<fullName>sample full username</fullName>"
to
"<fullName>PLACEHOLDER_FULLNAME</fullName>"
In your provisioning script, iterate over all users. For each user, replace each placeholder with the correct value.
e.g.
cp $SEED_DATA/templates/user/config.xml /tmp/config.xml
sed -e "s/\${PLACEHOLDER_USERNAME}/1/" -e "s/\${ChuckNorris}/dog/" /tmp/config.xml
sed -e "s/\${PLACEHOLDER_EMAIL}/1/" -e "s/\${he#findsyou.com}/dog/" /tmp/config.xml
...
mkdir -p $SEED_DATA/users/$USERNAME/
cp /tmp/config.xml $SEED_DATA/users/$USERNAME/config.xml
When you want to use generated users config.xml please generate for each user some permission settings in $JENKINS_HOME/config.xml:
<authorizationStrategy class="hudson.security.ProjectMatrixAuthorizationStrategy">
...
<permission>hudson.model.View.Create:username1</permission>
<permission>hudson.model.View.Delete:username1</permission>
<permission>hudson.model.View.Read:username1</permission>
...
<permission>hudson.model.View.Create:username2</permission>
<permission>hudson.model.View.Delete:username2</permission>
<permission>hudson.model.View.Read:username2</permission>
...
</authorizationStrategy>
Provisioning steps:
Install jenkins as you did before & maybe dynamic config generator (see above)
cp $SEED_DATA/config.xml $JENKINS_HOME/
cp -R $SEED_DATA/users/ $JENKINS_HOME/
chown -R "jenkins:jenkins" $JENKINS_HOME/users/ (maybe optional)
cd /tmp; java -jar jenkins-cli.jar -s http://localhost:8080/ reload-configuration

Resources