Jenkins lockable resource - lock without unlocking - jenkins

I'm using the "Locable Resources Plugin" together with the Jenkins multipipeline functionality:
lock('my-resource-name') {
}
I have a situation where I need to lock a resource in one pipeline and unlock it in another. I have workarounds, but I would prefer using the lock plugin, if possible.
I imagagine something like this
lock("my-resource-name")
.....
unlock("my-resource-name)
The why of it
I'm implementing canary releases, and there are three different paths through my pipeline (I considered other solutions, like pausing the pipeline with the input plugin - providing me with more is out of scope for this question).
I would like to lock/halt the pipeline while the canary is evaluated, and then, when the promotion or rollback is done, unlock it again.
Omitting the body for the lock statement just gives me java.lang.IllegalStateException: There is no body to invoke

One way of doing it is this:
curl -XGET https://your-jenkins/lockable-resources/reserve?resource=myresource
curl -XGET https://your-jenkins/lockable-resources/unreserve?resource=myresource
The GET is not a mistake, it really is GET.
To wait for a lock
lock("my-resource-name") {}
I also know Jenkins support something called "milestones" apparently is something used to communicate between builds. But I have no idea if it can be used to solve this problem in any meaningful way.

The lock can be done this way
import org.jenkins.plugins.lockableresources.LockableResource
import org.jenkins.plugins.lockableresources.LockableResourcesManager
...
LockableResourcesManager lockableResourcesManager = LockableResourcesManager.get()
LockableResource lockableResource = lockableResourcesManager.fromName(resourceName)
// if the resource doesn't exists, create it
if (lockableResource == null) {
lockableResourcesManager.createResource(resourceName)
lockableResource = lockableResourcesManager.fromName(resourceName)
}
// wait until lock is successful
waitUntil { lockableResourcesManager.lock([lockableResource].toSet(), currentBuild.rawBuild, null) }
The unlock is much simpler because you know the resource exists (if it doesn't there is no reason to try to unlock it)
LockableResourcesManager lockableResourcesManager = LockableResourcesManager.get()
LockableResource lockableResource = lockableResourcesManager.fromName(resourceName)
// do not pass currentBuild.rawBuild as in your case is not the build that locked this resource
// so I am passing null instead
lockableResourcesManager.unlock([lockableResource], null)
Note that I used to test this org.6wind.jenkins:lockable-resources:2.7, the implementation details may vary depending on the version used.
<dependency>
<groupId>org.6wind.jenkins</groupId>
<artifactId>lockable-resources</artifactId>
<version>2.7</version>
<scope>provided</scope>
</dependency>

from LockableResourcesManager get each lock, then
foreach lock do if lock.getName matches then do lock.reset() to release the lock
e.g. some groovy to find locked locks not owned by any builds and clean them up:
print "START\n"
def all_lockable_resources = org.jenkins.plugins.lockableresources.LockableResourcesManager.get().resources
all_lockable_resources.each { r->
if (r.isLocked() || r.isReserved()) {
println "Lock " + r + " is locked or reserved by " + r.getBuild() + " B CARSE " + r.getLockCause()
b = r.getBuild()
if (b) {
if (b.isBuilding()) { println "build:" + b + " is building" }
if (b.getResult().equals(null)) { println "build:" + b + " result is not in yet" }
if ( ! b.isBuilding() && ! b.getResult().equals(null)) {
println "build:" + b + " is not building and result is " + b.getResult() + " yet the lock " + r + " is locked."
println "ACTION RELEASE LOCK " + r
println "getLockCause:" + r.getLockCause()
println "getDescription:" + r.getDescription()
println "getReservedBy:" + r.getReservedBy()
println "isReserved:" + r.isReserved()
println "isLocked:" + r.isLocked()
println "isQueued:" + r.isQueued()
//release the lock
r.reset()
println "getLockCause:" + r.getLockCause()
println "getDescription:" + r.getDescription()
println "getReservedBy:" + r.getReservedBy()
println "isReserved:" + r.isReserved()
println "isLocked:" + r.isLocked()
println "isQueued:" + r.isQueued()
}
}
}
}
API: http://javadoc.jenkins.io/plugin/lockable-resources/org/jenkins/plugins/lockableresources/LockableResource.html

or navigate to https://jenkins-host/lockable-resources/ page. here we can find list of all the lockable resources. we can manually lock or reserve these.

Related

From deprecated cliOnline to OnlineNodeCommand

So I created this groovy script but read that the cliOnline() command is deprecated.
But I can't seem to figure out how to actually change my code to use hudson.model.Hudson.instance.OnlineNodeCommand()?
deprecated cliOnline
def nodes_checklist = ["PD100069", "PD100070", "PD100090", "PD10756"]; // List of nodes which should be turned online
def jenkinsNodes = jenkins.model.Jenkins.instance.getNodes() // Get all existing nodes on this Jenkins URL
for(def node_checker: nodes_checklist) {
for(def node: jenkinsNodes) {
if(node.getNodeName().contains(node_checker)) {
println "The node " + node.getNodeName() + "'s offline status: " + node.toComputer().isOffline()
if (node.toComputer().isOffline()){
println "Turning " + node.getNodeName() + " online"
node.toComputer().cliOnline() // If node is offline, turn it online
println node.getNodeName() + "'s online status: " node.toComputer().isOnline()
}
}
}
}
Does anyone know how to rewrite this to use the non-deprecated version?
If you look at this depricated method, it simply calls a non depricated method setTemporarilyOffline(boolean temporarilyOffline, OfflineCause cause). So not sure why this was depricated. Anyway instead of using cliOnline() you can use setTemporarilyOffline. Check the following.
node.getComputer().setTemporarilyOffline(false, null)
Some proper code with a proper cause. Cause is not really needed when setting the node online though.
import hudson.slaves.OfflineCause.UserCause
def jenkinsNodes = Jenkins.instance.getNodes()
for(def node: jenkinsNodes) {
if (node.getComputer().isTemporarilyOffline()){
node.getComputer().setTemporarilyOffline(false, null)
}
}
Setting to temporarily offline
UserCause cause = new UserCause(User.current(), "This is a automated process!!")
node.getComputer().setTemporarilyOffline(true, cause)

How to use `StaplerResponse rsp` in `AsyncResourceDisposer.doStopTracking()`

I'm trying to remove (stop tracking) trackig item from Jenkins AsyncResourceDisposer (${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer) via groovy scripts (${JENKINS_URL}/script).
According to the Javadoc and source code
// Javadoc
#Restricted(value=org.kohsuke.accmod.restrictions.DoNotUse.class)
public org.kohsuke.stapler.HttpResponse doStopTracking(#QueryParameter
int id,
org.kohsuke.stapler.StaplerResponse rsp
)
// source code
#Restricted(DoNotUse.class)
#RequirePOST
public HttpResponse doStopTracking(#QueryParameter int id, StaplerResponse rsp) {
...
}
I'd like to know how to add org.kohsuke.stapler.StaplerResponse rsp in doStopTracking(int id, org.kohsuke.stapler.StaplerResponse rsp):
import org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer
AsyncResourceDisposer disposer = AsyncResourceDisposer.get()
disposer.backlog.each {
disposer.doStopTracking( it.id, <what should I put here> )
}
Current I can get the item id, and the other informaitons like below:
import org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer
AsyncResourceDisposer disposer = AsyncResourceDisposer.get()
String url = Jenkins.instance.rootUrl + disposer.url
disposer.getBacklog().each { item ->
println "\n${item.id} : \t${url}/stopTracking/?id=${item.id} : \t${item.class.simpleName} : \n" +
"\t${item.getLastState().getDisplayName()} : \n" +
"\t${item.getDisposable().node} : ${item.getDisposable().path}\n" +
"\t${item.toString()}"
}
If I'm go to the url "${url}/stopTracking/?id=${item.id}" in browser (login first), the item can be removed after click RETRY USING POST (as below)
So... I'm using the API call curl -H <crumbIssues> -X POST <url> by passed the disposer.doStopTracking(int, org.kohsuke.stapler.StaplerResponse) (still really wants know how to use it)
Before running the following script, Strict Crumb Issuers Plugin is necessary to be installed and configured (or setup -Dhudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID=true) due to SECURITY-626 : Improved CSRF protection since:
obtain a crumb using the /crumbIssuer/api URL will now fail to perform actions protected from CSRF unless the scripts retain the web session ID in subsequent requests.
Here is details:
import org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer
import org.jenkinsci.plugins.strictcrumbissuer.StrictCrumbIssuer
AsyncResourceDisposer disposer = AsyncResourceDisposer.get()
StrictCrumbIssuer issuer = jenkins.model.Jenkins.instance.crumbIssuer
String jenkinsCrumb = "${issuer.crumbRequestField}:${issuer.crumb}"
String url = Jenkins.instance.rootUrl + disposer.url
disposer.getBacklog().each { item ->
println "\n ~~> removeing ${item.id} : "
[ 'bash', '-c', 'curl -s ' +
'-u <user>:<token> ' +
'-X POST ' +
"-H \"Content-Type: application/json\" " +
"-H \"Accept: application/json\" " +
"-H \"${jenkinsCrumb}\" " +
"${url}/stopTracking/?id=${item.id} "
].execute().with{
def stdout = new StringBuffer()
def stderr = new StringBuffer()
it.waitForProcessOutput( stdout, stderr )
println "EXIT CODE: ${it.exitValue()}"
println "ERROR: ${stderr}"
println "OUTPUT: ${stdout}"
}
}
Although, I still have a question... As we know that if the groovy script running in ${JENKINS_URL}/script, which means the "runner" is the administrator, so, how I can remove the specific user authorication '-u <user>:<token>' (by using the jenkins administrator authorication) in curl ?

function lines after httpRequest are not executed in groovy jenkins pipeline

None of the lines after making httpRequest are getting executed. Everything else works fine in this function. What could be going wrong here?
However, network request is going fine and I am able to see the response in the console. httpRequest is being made via plugin
I've even tried CURL - but lines after curl are not executed.
#NonCPS
def doPRCommentBasedTesting() {
def causes = currentBuild.rawBuild.getCauses()
def commentURL
for(cause in causes) {
if (cause.class.toString().contains("GitHubPullRequestCommentCause")) {
commentURL = cause.getCommentUrl()
commentURL = commentURL.substring(commentURL.lastIndexOf("-") + 1)
println "This job was caused by job " + commentURL
def url1 = "https://<git_url>/api/v3/repos/<owner>/<repo>/issues/comments/" + commentURL
def commentText = httpRequest authentication: '<auth_cred>', url: url1, consoleLogResponseBody: true
println commentText
println commentText.getClass()
println "hello world, how are you doing today?"
}
else {
println "Root cause : " + cause.toString()
}
}
println "==============================="
return 0
}
A non cps function does not have the ability to pause in between because it runs in a go. You need to put network call into a different function that is not marked as nonCPS and then it will work. In general the nonCPS block should be very small and limited to code that cannot be serialised

Remove user from Jenkins via groovy script when role based strategy is used

I am trying to automate Jenkins user deletion. Groovy script which I tried is as below.
import hudson.model.User
User u = User.get('rh54')
u.delete()
Mentioned groovy script does not delete when user is LDAP integrated. Also Security realm which is being used is Role-based Strategy.
Please suggest
The below groovy script should create a new instance of RoleBasedAuthorizationStrategy excluding the given user from all roles.
Please note that I've not tested it thoroughly and make sure to take a backup of your Jenkins instance before testing this script.
import hudson.model.*
import com.michelin.cio.hudson.plugins.rolestrategy.*
String removeRolesFromUser="test"
def roleBasedAuthorizationStrategy = Jenkins.getInstance().getAuthorizationStrategy()
if(roleBasedAuthorizationStrategy instanceof RoleBasedAuthorizationStrategy){
boolean changed=false
Map<String, RoleMap> roleMaps = roleBasedAuthorizationStrategy.getRoleMaps()
RoleBasedAuthorizationStrategy updatedRoleBasedAuthorizationStrategy = new RoleBasedAuthorizationStrategy()
for(Map.Entry<String, RoleMap> roleMapEntry : roleMaps.entrySet()){
Set<Role> roleSet = roleMapEntry.getValue().getRoles()
for(Role role : roleSet){
updatedRoleBasedAuthorizationStrategy.addRole(roleMapEntry.getKey(),role);
for(String addUserAgain : roleMapEntry.getValue().getSidsForRole(role.getName())){
if(!addUserAgain.equals(removeRolesFromUser)){
updatedRoleBasedAuthorizationStrategy.assignRole(roleMapEntry.getKey(),role,addUserAgain)
}else{
println("User : " + removeRolesFromUser + " excluded from role : " + role.getName())
changed=true
}
}
}
}
if(changed){
Jenkins.getInstance().setAuthorizationStrategy(updatedRoleBasedAuthorizationStrategy)
println("INFO: Authorization strategy updated, user " + removeRolesFromUser + " removed from all roles.")
Jenkins.getInstance().save();
println("INFO: Authorization strategy saved to disk.")
}else{
println("INFO: No changes made to Authorization strategy since user : " + removeRolesFromUser + " does not belong to any role!")
}
}else{
println("ERROR : This script works only for RoleBasedAuthorizationStrategy!")
}

Timeout Notification for Asynchronous Request

I am sending SPARQL queries as asynchronous requests to a SPARQL endpoint, currently DBpedia using the dotNetRDF library. While simpler queries usually work, more complex queries sometimes result in timeouts.
I am looking for a way to handle the timeouts by capturing some event when they occur.
I am sending my queries by using one of the asynchronous QueryWithResultSet overloads of the SparqlRemoteEndpoint class.
As described for SparqlResultsCallback, the state object will be replaced with an AsyncError instance if the asynchronous request failed. This does indicate that there was a timeout, however it seems that it only does so 10 minutes after the request was sent. When my timeout is, for example, 30 seconds, I would like to know 30 seconds later whether the request was successful. (35 seconds are ok, too, but you get the idea.)
Here is a sample application that sends two requests, the first of which is very simple and likely to succeed within the timeout (here set to 120 seconds), while the second one is rather complex and may easily fail on DBpedia:
using System;
using System.Collections.Concurrent;
using VDS.RDF;
using VDS.RDF.Query;
public class TestTimeout
{
private static string FormatResults(SparqlResultSet results, object state)
{
var result = new System.Text.StringBuilder();
result.AppendLine(DateTime.Now.ToLongTimeString());
var asyncError = state as AsyncError;
if (asyncError != null) {
result.AppendLine(asyncError.State.ToString());
result.AppendLine(asyncError.Error.ToString());
} else {
result.AppendLine(state.ToString());
}
if (results == null) {
result.AppendLine("results == null");
} else {
result.AppendLine("results.Count == " + results.Count.ToString());
}
return result.ToString();
}
public static void Main(string[] args)
{
Console.WriteLine("Launched ...");
Console.WriteLine(DateTime.Now.ToLongTimeString());
var output = new BlockingCollection<string>();
var ep = new SparqlRemoteEndpoint(new Uri("http://dbpedia.org/sparql"));
ep.Timeout = 120;
Console.WriteLine("Server == " + ep.Uri.AbsoluteUri);
Console.WriteLine("HTTP Method == " + ep.HttpMode);
Console.WriteLine("Timeout == " + ep.Timeout.ToString());
string query = "SELECT DISTINCT ?a\n"
+ "WHERE {\n"
+ " ?a <http://www.w3.org/2000/01/rdf-schema#label> ?b.\n"
+ "}\n"
+ "LIMIT 10\n";
ep.QueryWithResultSet(query,
(results, state) => {
output.Add(FormatResults(results, state));
},
"Query 1");
query = "SELECT DISTINCT ?v5 ?v8\n"
+ "WHERE {\n"
+ " {\n"
+ " SELECT DISTINCT ?v5\n"
+ " WHERE {\n"
+ " ?v6 ?v5 ?v7.\n"
+ " FILTER(regex(str(?v5), \"[/#]c[^/#]*$\", \"i\")).\n"
+ " }\n"
+ " OFFSET 0\n"
+ " LIMIT 20\n"
+ " }.\n"
+ " OPTIONAL {\n"
+ " ?v5 <http://www.w3.org/2000/01/rdf-schema#label> ?v8.\n"
+ " FILTER(lang(?v8) = \"en\").\n"
+ " }.\n"
+ "}\n"
+ "ORDER BY str(?v5)\n";
ep.QueryWithResultSet(query,
(results, state) => {
output.Add(FormatResults(results, state));
},
"Query 2");
Console.WriteLine("Queries sent.");
Console.WriteLine(DateTime.Now.ToLongTimeString());
Console.WriteLine();
string result = output.Take();
Console.WriteLine(result);
result = output.Take();
Console.WriteLine(result);
Console.ReadLine();
}
}
When I run this, I reproducibly get an output like the following:
13:13:23
Server == http://dbpedia.org/sparql
HTTP Method == GET
Timeout == 120
Queries sent.
13:13:25
13:13:25
Query 1
results.Count == 10
13:23:25
Query 2
VDS.RDF.Query.RdfQueryException: A HTTP error occurred while making an asynchron
ous query, see inner exception for details ---> System.Net.WebException: Der Rem
oteserver hat einen Fehler zurückgegeben: (504) Gatewaytimeout.
bei System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
bei VDS.RDF.Query.SparqlRemoteEndpoint.<>c__DisplayClass13.<QueryWithResultSe
t>b__11(IAsyncResult innerResult)
--- Ende der internen Ausnahmestapelüberwachung ---
results == null
Obviously, the exact times will be different, but the crucial point is that the error message based on the second query is received approximately 10 minutes after the request was sent, nowhere near the 2 minutes set for the timeout.
Am I using dotNetRDF incorrectly here, or is it intentional that I have to run an additional timer to measure the timeout myself and react on my own unless any response has been received meanwhile?
No you are not using dotNetRDF incorrectly rather there appears to be a bug that the timeouts set on an endpoint don't get honoured when running queries asynchronously. This has been filed as CORE-393
By the way even with this bug fixed you won't necessarily get a hard timeout at the set timeout. Essentially the value you set for the Timeout property of the SparqlRemoteEndpoint instance that value is used to set the Timeout property of the .Net HttpWebRequest. The documentation for HttpWebRequest.Timeout states the following:
Gets or sets the time-out value in milliseconds for the GetResponse
and GetRequestStream methods.
So you could wait up to the time-out to make the connection to POST the query and then up to the time-out again to start receiving a response. Once you start receiving a response the timeout becomes irrelevant and is not respected by the code that processes the response.
Therefore if you want a hard timeout you are better off implementing it yourself, longer term this may be something we can add to dotNetRDF but this is more complex to implement that simply fixing the bug about the timeout not getting honoured for the HTTP request.

Resources