TFS Build 2015 not publishing test results - tfs

UPDATE (I have restructured this question with a new and basic solution and build as the same problem exists there)
I'm using the new build system in Tfs 2015 and I'm using Update 1. I have noticed that on the build summary screen, no test results or Code Coverage results are submitted:
I have a test step which runs and contains results and I have a .trx and a .coverage file but the test step log contains the warning:
##[warning]Failed to publish test results: The value for the Build.Id property is not within the permissible values for it.
The Build is very basic, it contains 2 steps and Code Coverage is Enabled on the test step:
Here is the build def as retrieved by the TEST REST API:
{
"build":[
{
"enabled":true,
"continueOnError":false,
"alwaysRun":false,
"displayName":"Build solution $/IsdDev/SQTestBuild/Main/TestBuildSystem.sln",
"task":{
"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda",
"versionSpec":"*"
},
"inputs":{
"solution":"$/IsdDev/SQTestBuild/Main/TestBuildSystem.sln",
"msbuildArgs":"",
"platform":"$(BuildPlatform)",
"configuration":"$(BuildConfiguration)",
"clean":"false",
"restoreNugetPackages":"true",
"vsVersion":"14.0",
"msbuildArchitecture":"x86",
"logProjectEvents":"true"
}
},
{
"enabled":true,
"continueOnError":false,
"alwaysRun":false,
"displayName":"Test Assemblies **\\$(BuildConfiguration)\\*test*.dll;-:**\\obj\\**",
"task":{
"id":"ef087383-ee5e-42c7-9a53-ab56c98420f9",
"versionSpec":"*"
},
"inputs":{
"testAssembly":"**\\$(BuildConfiguration)\\*test*.dll;-:**\\obj\\**",
"testFiltercriteria":"",
"platform":"$(BuildPlatform)",
"configuration":"$(BuildConfiguration)",
"runSettingsFile":"",
"overrideTestrunParameters":"",
"codeCoverageEnabled":"true",
"vsTestVersion":"14.0",
"pathtoCustomTestAdapters":"",
"otherConsoleOptions":""
}
}
],
"options":[
{
"enabled":false,
"definition":{
"id":"7c555368-ca64-4199-add6-9ebaf0b0137d"
},
"inputs":{
"multipliers":"[]",
"parallel":"false",
"continueOnError":"true"
}
}
],
"variables":{
"BuildConfiguration":{
"value":"debug",
"allowOverride":true
},
"BuildPlatform":{
"value":"any cpu",
"allowOverride":true
}
},
"retentionRules":[
{
"branches":[
"+refs/heads/*"
],
"artifacts":[
"build.SourceLabel"
],
"daysToKeep":10,
"deleteBuildRecord":true,
"deleteTestResults":true
}
],
"_links":{
"self":{
"href":"http://removed/343974ee-60a8-40ee-a103-37fa5db37c41/_apis/build/Definitions/71"
},
"web":{
"href":"http://removed/_permalink/_build/index?collectionId=808a695f-85b6-49db-a2df-ca4aaf8bbb9f&projectId=343974ee-60a8-40ee-a103-37fa5db37c41&definitionId=71"
}
},
"buildNumberFormat":"$(date:yyyyMMdd)$(rev:.r)",
"createdDate":"2015-11-19T07:43:27.977Z",
"jobAuthorizationScope":"projectCollection",
"jobTimeoutInMinutes":60,
"repository":{
"properties":{
"labelSources":"0",
"labelSourcesFormat":"$(build.buildNumber)",
"tfvcMapping":"{\"mappings\":[{\"serverPath\":\"$/IsdDev\",\"mappingType\":\"map\",\"localPath\":\"\\\\\"},{\"serverPath\":\"$/IsdDev/Drops\",\"mappingType\":\"cloak\",\"localPath\":\"\"}]}"
},
"id":"$/",
"type":"TfsVersionControl",
"name":"IsdDev",
"url":"http://removed/",
"defaultBranch":"$/IsdDev",
"rootFolder":"$/IsdDev",
"clean":"undefined",
"checkoutSubmodules":false
},
"quality":"definition",
"authoredBy":{
"id":"7d24caba-6c09-4b78-8ff1-2404e0b0ec64",
"displayName":"Rod Hall",
"uniqueName":"ISD-NR\\Rod.Hall",
"url":"http://removed/_apis/Identities/7d24caba-6c09-4b78-8ff1-2404e0b0ec64",
"imageUrl":"http://removed/_api/_common/identityImage?id=7d24caba-6c09-4b78-8ff1-2404e0b0ec64"
},
"queue":{
"pool":{
"id":2,
"name":"Back Office Pool"
},
"id":2,
"name":"Back Office Pool"
},
"uri":"vstfs:///Build/Definition/71",
"type":"build",
"revision":1,
"id":71,
"name":"SQ Test",
"url":"http://removed/343974ee-60a8-40ee-a103-37fa5db37c41/_apis/build/Definitions/71",
"project":{
"id":"343974ee-60a8-40ee-a103-37fa5db37c41",
"name":"IsdDev",
"url":"http://removed/_apis/projects/343974ee-60a8-40ee-a103-37fa5db37c41",
"state":"wellFormed",
"revision":7
}
}
What is the Build.Id that the warning refers to, how can I view it and set it?
This could even be a problem with TFS 2015 Update 1...
UPDATE I'm not sure if it matters but when I go to the Build section of the TFS Admin Console it appears to not be configured:
I have my application tier on 1 server and then I have 3 Agent Pools each with 3 Agents. There are 3 build servers each of which have 1 agent from each pool installed for a total of 9 agents. I set these up using PowerShell and I downloaded the agent from the TFS page. My builds still work Ok, it's just this tests publishing

Check the attached vNext build configuration, with which I can get the Test result and Code coverage. Make sure you have specified the solution in Visual Studio Build and select Code Coverage Enabled in Visual Studio Test.

Credit to Jesse Houwing for helping me on this one. I had to detach and reattach the Team Project Collection using the TFS Admin Console on the App Server. It seems there is an issue with Update 1 (or perhaps one of the Release Candidates).
For those with the same problem, it will obviously take TFS offline but it took less than 5 minutes.

I had a similar issue yesterday: Visual Studio Test using Test Agent step was failing with the "The value for the Build.Id property is not within the permissible values for it" error message when it was trying to execute Coded UI tests on the remote test machine. Detaching project collection and re-attaching it back fixed the issue.

Related

Custom build task not showing in YAML task list

We wrote a custom Azure DevOps build task, but we can't find it in the YAML editor task list. It doesn't even show up in search.
This is my task.json:
{
"id": "17813657-13c6-4cd8-b245-8d8b3b0cf210",
"name": "ApplitoolsBuildTask",
"friendlyName": "Applitools Build Task",
"description": "Add the Applitools dashboard as a tab in the Azure DevOps build results page.",
"categories": [ "Build" ],
"category": "Build",
"author": "Applitools",
"version": {
"Major": 0,
"Minor": 44,
"Patch": 0
},
"instanceNameFormat": "Applitools Build Task $(version)",
"execution": {
"Node": {
"target": "dist/index.js"
}
}
}
I also tried with only categories property, and it still didn't show in the search.
I then tried downloading Augurk locally and examined its content (also available in GitHub: https://github.com/Augurk/vsts-extension/tree/master/src), and I saw in AugurkCLI it doesn't even have categories property, as it has a typo: categorues, and for some reason it still shows up. This leads me to think there's no relation between that property and the task list.
I also tried examining the XML file and saw it has <PublisherDetails> section, which my .vsix file doesn't have. What should I put in my vss-extension.json file to have it? And will it help getting my extension to show up in the Task List?
Note that in the Classic editor (the one with the UI) I see it just fine, in the right categories (if I have the "category" property), and if I don't have it then it still shows up when I search. The problem I have is to get my build-task to show up in the YAML editing Task List.
Indeed, our team is fixing this issues now. The issue caused by the YAML assistant panel doesn't allow tasks without input parameters. But worked in classic editor.
Before our fixed release deployed, you can use this workaround to achieve your customize task appeared in the YAML editor task list:
Change your script to accept an input parameter. And then the task will appeared in YAML editor task list.
You can reference this ticket we received recently. We will inform you here once we deployed the fixed release and the issue be fixed.
Well, it seems it's a bug on Microsoft side. I don't have any input fields in the build task, and the Azure DevOps YAML editor task list filters out any task that doesn't have input fields.
They told me they had fixed it:
https://developercommunity.visualstudio.com/content/problem/576169/our-custom-azure-devops-build-task-doesnt-show-in.html
The fix should be available in a few weeks.

Unable to get the result from Google machine learning Cloud REST API

I am trying to run a job using Google Cloud Machine Learning REST-API ml.jobs.project.create
The latest job that I submitted has job id 'drivermonitoring20180109335'. Here on completion of the job, message 'job completed successfully' is displayed but I cannot see any desired output file in the specified location. Output logs can be seen in fig1
Also I would like to keep in-front of you my few observations while running this job id:
i) Running the job took very less time in comparison to any other job that I executed before.
ii) While running jobs before, every job earlier was executed via two different tasks viz a)master-replica-0 and b)service (refer fig2) but this job didn't have master-replica-0 task(refer fig3) I tried to Google the issue, but was unable to find any solution related to the issue.
So I can infer that the task that I was trying to run is being scheduled but the python script that I am trying to run is never scheduled to be executed.
Kindly let me know if you require more screenshots or if you want to have a look at the project structure to help with the issue.
Thanks in advance.
EDIT 1: Added JSON while making API call
POST https://ml.googleapis.com/v1/projects/drivermonitoringsystem/jobs?key={YOUR_API_KEY}
{
"trainingInput": {
"pythonModule": "trainer.retrain",
"args": [
"--bottleneck_dir=ModelTraining/tf_files/bottlenecks \
--model_dir=ModelTraining/tf_files/models/ \
--architecture=mobilenet_0.50_224 \
--output_graph=gs://<BUCKET_NAME>/tf_files/retrained_graph.pb \
--output_labels=gs://<BUCKET_NAME>/tf_files/retrained_labels.txt \
--image_dir=gs://<BUCKET_NAME>/dataset224x224/"
],
"region": "us-central1",
"packageUris": [
"gs://<BUCKET_NAME>/ModelTraining4.tar.gz"
],
"jobDir": "gs://<BUCKET_NAME>/tf_files/",
"runtimeVersion": "1.4"
},
"jobId": "job_id201801101535"
}
I have just run myself some sample jobs using both the gcloud command and the REST API, and everything has just worked fine in both of the cases. It looks like, in your case, the job was never executed, as there is no cluster created for processing the job itself (that is why master-replica-0 is missing).
The jobs that you had run previously and which had worked were launched also using the REST API, or instead with gcloud or a Client Library?
Here I share an example JSON I used when making the API call to ml.projects.jobs.create through the API Explorer link you shared, I suggest you try adapting it to your requirements and check if you got any missing field:
POST https://ml.googleapis.com/v1/projects/<YOUR_PROJECT>/jobs?key={YOUR_API_KEY}
{
"jobId": "<JOB_ID>",
"trainingInput": {
"jobDir": "gs://<LOCATION_TO_STORE_OUTPUTS>",
"runtimeVersion": "1.4",
"region": "<REGION>",
"packageUris": [
"gs://<PATH_TO_YOUR_TRAINER>/trainer-0.0.0.tar.gz"
],
"pythonModule": "<PYTHON_MODULE_TO_RUN>",
"args": [
"--train-files",
"gs://<PATH_TO_YOUR_TRAINING_DATA>/data.csv",
"--eval-files",
"gs://<PATH_TO_YOUR_TEST_DATA>/test.csv",
"--train-steps",
"100",
"--eval-steps",
"10",
"--verbosity",
"DEBUG"
]
}
}
Change TrainingInput to PredictionInput (and the appropriate child fields) if you are trying to run a prediction job instead of a training one, as in this example.

How to associate a test result posted to VSTS with a testcase residing in VSTS?

We have an automation test framework written in Java.
At present, I can post a new test run to VSTS.
I can then post a new test result to the test run, eg:
{'testCaseTitle':'vstsPostTest',
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
However, when viewing the test result entry in VSTS, the testCaseTitle always seems to be overridden with value: 'myTestCaseTitle'.
Thinking this may be because I haven't associated the posted result with a testcase that resides in VSTS, I have added the testCase id parameter, and specified the ID of a random existing testCase in VSTS, eg,
{'testCaseTitle':'is title now necessary?',
'testCase'{'id':283},
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
My posted result still has the generic title 'myTestCaseTitle' - and I have seen nothing yet in VSTS which illustrates a link between the posted result and the specified testcase which resides in VSTS.
What else is necessary to:
a) Ensure that a posted result uses the specified testCaseTitle value (or receives it from the ID of the associated testCase)
b) See that either a testCase or story in VSTS is flagged as failing/passed, based on the latest testrun results. (ie, identify where stories are failing as a result of automated regression testing)
Many thanks
Try to specify test point ID:
POST: https://[team project url]/_apis/test/runs/[test run id]/results?api-version=3.0-preview
[
{
"testPoint":{
"id":XX
},
"priority": 1,
"outcome": "Passed"
}
]
You can get test point id by using Test points REST API
With thanks to Starains help. As commented: I ended up carefully following this MS post: social.msdn.microsoft.com/Forums/sqlserver/en-US/… To others reading: after a VSTS test plan, suite, case is created: post a new test run (specify plan ID), then post a new test result (specify testPoint ID, outcome, etc) then get the testResult ID and update that test result to state: Completed. This is the minimum to affect the outcome of a test case in VSTS.

inconsistency in outcome status when execute test method between TFS2017 & TFS2015

I created a test method that uses "Assert.Inconclusive(skipMessage)" function. On TFS2015, the executing outcome is 'done', but on TFS2017 the executing outcome is 'not executed'.
Could you please help me to make the outcome is 'done' when using "Assert.Inconclusive(skipMessage)" on TFS2017?
In both TFS 2015 and TFS 2017, the Skipper test methods are both shown like 'not executed' not 'Done'.
This settings in TFS can't be changed. But if you want to change/edit the outcome of a specify test result, you could use TFS REST API to achieve this.
HTTP Method: PATCH
URL: http://SERVERNAME:8080/tfs/DefaultCollection/teamProject/_apis/test/runs/testrunId/results?api-version=3.0-preview
Json object:
{
"id": 100000, --this is test result id
"outcome": "None"
}
You could change the outcome from NotExecuted to None, Passed, Failed, Inconclusive, Timeout, Aborted, Blocked, Warning, Error, NotApplicable, Paused or InProgress.

MS Build TestCategory Filter not working

Environment
TFS2013.Update 2 12.0.30324.0
C#
Visual Studio 2013
Visual Studio Test Runner
Gated Check-In
Problem
As shown here I decorated my TestMethod like
[TestCategory("Gated"), TestMethod()]
As shown here I filtered, like
TestCategory=Gated
When I build without any filter, the TestRun run successfully with all unit tests. But when I tried to filter over TestCaterory, no TestRun started at all.
I tried without success ...
... to put the filter in quotation:
"TestCategory=Gated"
... to separate the attributes:
[TestMethod]
[TestCategory("Gated")]
... a similar question has been posted here, but I can't revert back to using the old default.xaml.
... I found just one warning in the build log under "Run Visual Studio Test Runner for Test Sources": "Dependent Assembly Newtonsoft.Json, Version=4.5.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed of C:\Builds2\56\614\bin\External\Microsoft.TeamFoundation.TestManagement.Common.dll not found. Can be ignored if not a NUnit project."
-> I added the reference to Newtonsoft.Json in every single NUnit-Project in all the solutions, that were build during the gated check-in - also without success.
The problem was not the TestCategory, but another pretty simple one:
The unittest-project in which the “broken” testmethod resides was not part of ”projects to Build” of the Build Definition.

Resources