Im trying to use the TFS 2015 Client API to get the build details of the new builder system.
I already got pretty far,
Using the GetBuildsAsync i get the currently running builds, but not all properties are filled.
var buildClient = new BuildHttpClient(new Uri(m_myCollectionUrl, UriKind.Absolute), cred);
Build[] buildsInProgress = (buildClient.GetBuildsAsync(projectName, statusFilter: Microsoft.TeamFoundation.Build.WebApi.BuildStatus.InProgress)).Result.ToArray();
the Build.Demands list is always null even if the build definition has one or more default Demands, also when specifying another one at build queue time, it is not showing up using the API. The list is always null.
I have tried in a few ways now
buildClient.GetFullDefinitionsAsync - results in a BuildDefinition where demands = null
buildClient.GetDefinitionAsync - results in a BuildDefinition where demands is filled with the default demands from the defintion, but only the ones we defined ourselves, not the system ones.
buildClient.GetBuildsAsync - results in a Build where demands is null So i still haven't found a call where i can get the Demands that have been used to find that BuildAgent. To test above, i am running a test build with 2 "exists" demands and i specify another at queue time, which i cannot find back.
I am not able to find much around this online. What should i do to get more details on the builds.
Thanks.
The demands is a new feature in vNext build system. With using Rest API to get either a build detail or build definition, there is also not any related default demands info in response json.
Afraid this is due to some default demand is setting exists, and the value is null. If you manually add a demand with equals something, this will return accordingly value.
From the response json in rest api of getting build definition, will get something as below:
"demands": [
"agentname -equals test" ],
Suggest you to use client API to get the build definition info, if there are equals setting, this will return some value of the demand.
public static DefinitionReference[] GetBuildDefinitions(this BuildHttpClient buildServer, string collectionName)
{
var buildDefinitions = buildServer.GetDefinitionsAsync(project: collectionName).Result.ToList();
return buildDefinitions.ToArray();
}
Update
Afraid you could not use Client API to get the default system demands list of the build definition and the agent queue position. It's TFS sever based. It's not able to use a Client API to get TFS sever side info.
Related
I apologize for duplicating my post from sonarsource, but sometimes this gets a different audience.
We’re using SonarQube 7.9.2.
Our Jenkins builds use the pipeline steps “withSonarQubeEnv” and “waitForQualityGate”, and in between we use “mvn sonar:sonar” to run the scan. At the end of the latter, it prints the task id it’s going to be waiting for in “waitForQualityGate”. It also shows that task id in the results of that step.
What WebApi call(s) can I perform in between “mvn sonar:sonar” and “waitForQualityGate” that will let me store into a variable the task id that is going to be polled for? I know the project key at that point. I’ve inspected all of the environment variables in scope at that point.
I know how to find the WebApi documentation, and I’ve scanned through what I think are the relevant operations, but I can’t figure out which operation I need for this particular “task”.
When you run the mvn sonar:sonar, report-task.txt will be created in the workspace folder .scannerwork.
You will get the ceTaskUrl and ceTaskId in report-task.txt. Now, you can use that ceTaskUrl to get the analysisId.
You can use the below api to get the quality gate status using analysisId.
https://localhost:9000/sonarqube/api/qualitygates/project_status?analysisId=$ANALYSIS_ID"
I've configured a service hook in VSTS to connect to Jenkins. I'm able to use the Jenkins extension to trigger a Jenkins job if it's not in a subfolder. E.g. http://myhost/job/Always%20succeed/
In that case, I can simply connect like this and run my job:
If my job is nested, however, I can't figure out how to connect. Here's an example: http://myhost/view/Production/job/Automation/job/Test/job/My%20Job
I've tried using just the name (e.g. "My Job"), the whole url, and a dot notation (Production.Automation.Test.My Job). How can I make this run and where can I find more documentation?
It's pretty nuanced and one could argue buggy. First off, I can reach the same job with two urls:
http://myhost/view/Production/job/Automation/job/Test/job/My%20Job
http://myhost/job/Automation/job/Test/job/My%20Job
Turns out the latter is the way to go.
I tried the following name, and it tried reaching the corresponding endpoint:
Automation/job/Test/job/My%20Job <- name used in VSTS "Job name" field
/job/Automation/job/job/Test/job/My%20Job/build <- url attempted, failed (404)
Note the double job/. Then I tried the following with better results:
Automation/Test/job/My%20Job <- name used
/job/Automation/job/Test/job/My%20Job/build <- url tried, success
It's a bit concerning that the pattern isn't consistent regarding the double "job/" part but whatever.
I have an Apache Beam job running on Google Cloud Dataflow, and as part of its initialization it needs to run some basic sanity/availability checks on services, pub/sub subscriptions, GCS blobs, etc. It's a streaming pipeline intended to run ad infinitum that processes hundreds of thousands of pub/sub messages.
Currently it needs a whole heap of required, variable parameters: which Google Cloud project it needs to run in, which bucket and directory prefix it's going to be storing files in, which pub/sub subscriptions it needs to read from, and so on. It does some work with these parameters before pipeline.run is called - validation, string splitting, and the like. In its current form in order to start a job we've been passing these parameters to to a PipelineOptionsFactory and issuing a new compile every single time, but it seems like there should be a better way. I've set up the parameters to be ValueProvider objects, but because they're being called outside of pipeline.run, Maven complains at compile time that ValueProvider.get() is being called outside of a runtime context (which, yes, it is.)
I've tried using NestedValueProviders as in the Google "Creating Templates" document, but my IDE complains if I try to use NestedValueProvider.of to return a string as shown in the document. The only way I've been able to get NestedValueProviders to compile is as follows:
NestedValueProvider<String, String> pid = NestedValueProvider.of(
pipelineOptions.getDataflowProjectId(),
(SerializableFunction<String, String>) s -> s
);
(String pid = NestedValueProvider.of(...) results in the following error: "incompatible types: no instance(s) of type variable(s) T,X exist so that org.apache.beam.sdk.options.ValueProvider.NestedValueProvider conforms to java.lang.String")
I have the following in my pipelineOptions:
ValueProvider<String> getDataflowProjectId();
void setDataflowProjectId(ValueProvider<String> value);
Because of the volume of messages we're going to be processing, adding these checks at the front of the pipeline for every message that comes through isn't really practical; we'll hit daily account administrative limits on some of these calls pretty quickly.
Are templates the right approach for what I want to do? How do I go about actually productionizing this? Should (can?) I compile with maven into a jar, then just run the jar on a local dev/qa/prod box with my parameters and just not bother with ValueProviders at all? Or is it possible to provide a default to a ValueProvider and override it as part of the options passed to the template?
Any advice on how to proceed would be most appreciated. Thanks!
The way templates are currently implemented there is no point to perform "post-template creation" but "pre-pipeline start" initialization/validation.
All of the existing validation executes during template creation. If the validation detects that there the values aren't available (due to being a ValueProvider) the validation is skipped.
In some cases it is possible to approximate validation by adding runtime checks either as part of initial splitting of a custom source or part of the #Setup method of a DoFn. In the latter case, the #Setup method will run once for each instance of the DoFn that is created. If the pipeline is Batch, after 4 failures for a specific instance it will fail the pipeline.
Another option for productionizing pipelines is to build the JAR that runs the pipeline, and have a production process that runs that JAR to initiate the pipeline.
Regarding the compile error you received -- the NestedValueProvider returns a ValueProvider -- it isn't possible to get a String out of that. You could, however, put the validation code into the SerializableFunction that is run within the NestedValueProvider.
Although I believe this will currently re-run the validation everytime the value is accessed, it wouldn't be unreasonable to have the NestedValueProvider cache the translated value.
I am using Visual Studio Online for Source Control and Build processes. I created a new build definition using the TfvcContinuousDeploymentTemplate.12.xaml. When Queuing a new build from within VS I have the chance to change some parameters, but I can't change the Build number format. What determines what shows up on that parameter list and how can I make sure the Build number format appears there?
My suggestion is to investigate modifying the build template to:
1) Take a custom input value, which I believe you can change on each invocation of the build; and
2) Finding the appropriate step to interrogate the existing Build Number and modify it based on this input.
This should get you started:
http://msdn.microsoft.com/en-us/library/dd647551.aspx
Alternatively, you could remove the Build Number Activity in its entirety and substitute your own – but I don’t think these measures are warranted in this case. This would get you started down that trail:
http://blogs.msdn.com/b/willbar/archive/2010/01/21/generating-custom-build-number-in-tfs-build-2010.aspx
HTH –
jlo
To show the property you have to edit the template, expand the arguments, search for Metadata and click on the ellipse towards the right of the row. Find the property you want, in my case it is BuildNumberFormat and change the View this parameter when: Always show the parameter
I have been scouring the internet for days, I have a problem similar to this.
I need to retrieve the console output in raw (plain) text. But if I can get it in HTML that is fine too, I can always parse it. The only thing is that I need to get it during the build step, which is a problem since the location where it should be available is truncated...
I have tried retrieving the console output from the following URL's (relative to the job):
/consoleText
/logText/progressiveText
/logText/progressiveHTML
The two text ones are plain text and would be perfect if not for the truncation, same goes for the HTML one... exactly what I need - only its truncated....
I am sure it is possible to retrieve this information somehow, since when viewing /consoleFull there is a real-time update of the console, without truncating or buffering.
However, upon examining that web page, instead of finding the content I desired, I found this code where it should have been (I did not include the full pages code, since it would be mostly irrelevant, and I believe those answering would be able to find out and know what should be there on their own)
new Ajax.Request(href,{
method: "post",
parameters: {"start":e.fetchedBytes},
requestHeaders: headers,
onComplete: function(rsp,_) {
var stickToBottom = scroller.isSticking();
var text = rsp.responseText;
if(text!="") {
var p = document.createElement("DIV");
e.appendChild(p); // Needs to be first for IE
// Use "outerHTML" for IE; workaround for:
// http://www.quirksmode.org/bugreports/archives/2004/11/innerhtml_and_t.html
if (p.outerHTML) {
p.outerHTML = '<pre>'+text+'</pre>';
p = e.lastChild;
}
else p.innerHTML = text;
Behaviour.applySubtree(p);
if(stickToBottom) scroller.scrollToBottom();
}
e.fetchedBytes = rsp.getResponseHeader("X-Text-Size");
e.consoleAnnotator = rsp.getResponseHeader("X-ConsoleAnnotator");
if(rsp.getResponseHeader("X-More-Data")=="true")
setTimeout(function(){fetchNext(e,href);},1000);
else
$("spinner").style.display = "none";
}
});
Specifically, I am hoping there is a way for me to get the content from text whatever it may be. I am not familiar with this language and so am not sure how I might be able to get the content I want. Plugins won't help since I want to retrieve this content as part of my script during the build step
You did pretty much good investigation already. I can only add the following: all console related plug-ins I know are designed as a post build actions.
The Log Trigger plugin provides a post-build action that allows Hudson
builds to search their console log for a given regular expression and
if found, trigger additional downstream jobs.
So it looks like there is no straightforward solution to your problem. I can see the following options:
1. Use tee or something similar (applicable to shell build steps only)
This solution is far from being universal, but it can provide quick access to the latest console output, produced by a command or set of command.
tee - read from standard input and write to standard output and files
Using synonyms on the system level other Jenkins build steps can modified in order to produce console output. File with console output can be referenced through Jenkins or using any other way.
2. Modify Jenkins code
You can just do a quick fix for internal usage or provide a patch introducing specific system-wide setting.
3. Mimic /console behavior
Code in your example is used to request updates from the Jenkins server. As you may expect the server side can return piece of information starting with some offset. Let me show.
Periodically console page sends requests to the server:
Parameters are straightforward:
Response is a chunk of information to be added:
Another request with updated offset (start) value
You can easily understand there is no data by analyzing Content-Length
So the answer is: use url/job-name/build-number/logText/progressiveHtml, specify start offset, send request and receive console update.
I had a similar issue, the last part of my Jenkinsfile build script needs to parse the ConsoleLog for particular error messages to put in an email build report.
First attempt: http request.
It felt like a hack, it mostly worked, but ran into issues when we locked down access to the Jenkins server & my build nodes could no longer perform annon http gets on the page
Second attempt: use the APIs to enumerate the log lines.
It felt like the right thing to do, but it failed horribly as my nodes would take 30 minutes to get through the 100 meg log files. My presumption is that the Jenkins server was not caching the file, so each request involved a re-reading of the entire file up until the point of the last read.
Third and most successful solution: run grep on the server.
node('master') {
sh 'grep some_criteria $JENKINS_HOME/workspace/path/to/job/console.log'
}
it was fast, reliable, and it didn't matter how big the log files were.
Yes, this required trust of the Jenkins admin and knowledge of the directory paths on the Jenkins server - but since I was the admin, I trusted myself to do the right thing. Your mileage may vary.
To add some insight: when the Jenkins build was in progress, the response for the .../consoleText URL maxed out at 10000 lines, exactly.
I was using 'requests()' package in Python. I have tried the same URL with curl and again received only the first 10K lines.
Only after the build has finished both methods returned the full log (>22K lines in my case).
I will research further and hope to report back.
[2015-08-18] Update: It seems that this is a known issue (see here) and it's fixed in Jenkins 1.618 and later. I am still running 1.615 so I cannot verify.
Amir