I tried to show test result from automation test on dashboard but I wasn't able to do it.
In release I have environment which is responsible for running automation tests, I can see test results in release details and in tests runs. In test runs there are a lot of charts that I want to pin to dashboard, unfortunately there is no option in test run. I tried also creating test chart widget etc. directly from dashboard but they are only working with tests running with build.
Is there way to pin test result from release environment to tfs dashboard?
I am using tfs 2017
Unfortunately the feature is not supported.
For now, we can only Add a release summary chart to dashboard, then you can naviagte to the specific release from the summary chart to see the test results.
Please see Add charts to a dashboard to check which chart/graph are available to be added to a dashboard.
Related
Since MTM lab management is deprecated in TFS 2018 and we are setting up test agents as part of build definition ,How can we see test cases results in real time ?
For example if user wants to know how many test cases are in progress ,passed,failed.
In MTM we can see while running test cases using TCM.exe till tfs 2017
Could any one please help
Thanks
Unfortunately real-time output for tests is not supported for now.
There are some similar user voices submitted to suggest the feature:
Provide VStest.console.exe \ Test Explorer Real Time Test Execution
Status DURING Test Execution
That I could see the reports that I see in the MTM tool, from a
Confluence Page (real time updated info on testing status).
You can go and vote them up or submit a new user voice to suggest the feature again to achieve the feature in future release.
Similar threads for your reference:
Real time status/logs of test execution not available on VSTS
console
Real time logs/stats of unit test execution on VSTS console
We are running TFS version 2018 RTW (16.122.27102.1). I am trying to view the complete run history for my each of my test cases. I am only seeing the last couple of test runs when I go to the Build -> Tests -> Click on a test -> Right panel shows a graph with the link "View History" below it. The graph is only showing the last couple of test runs:
The Build definition I am working with has a retention policy of 15 days and a minimum to keep of 3. TFS seems to be only showing history for builds it has not yet deleted. However, the box "Automated test results" is not checked - I am under the impression that not checking this box should allow me to see the completed history of my test runs.
How can I view the completed history of my test runs without having to retain all my builds?
Additional Details
My build specific retention policy is listed first. This means it will override the global policy if I understand correctly.
I can see all the test runs for my deleted builds. However the test history for each individual test only shows 3 bars. I was expecting to see 8 bars (the current 3 plus the 5 test runs from the deleted builds).
There are not only the build retention policy in your build definition, but also a Global build retention policy
Global build retention policy settings can be managed from the Build and Release settings of your account or team project collection:
TFS 2018: https://{your_server}/tfs/DefaultCollection/_admin/_buildQueue
You could not be able to change the delete test results value in the policy there. So if your builds fit the global policy, the test result will still be deleted.
Besides, you could use the REST API to get the test case and Test Run list.
However, it's not so easy to get the test run history for a specific test case, unless you run only the specific test case each time. In fact, usually we will run multiple test cases in a test run.
REST API to get the test case,
GET http://SERVER:8080/tfs/DefaultCollection/2017ScrumProjectFromVS/_apis/test/plans/105/suites/106/testcases/107?api-version=1.0
REST API to get the test run list :
GET http://server:8080/tfs/DefaultCollection/2017ScrumProjectFromVS/_apis/test/Runs/
Actually the best way to retrieve test cases run status history is
using MTM (Microsoft Test Manager):
Connect TFS server -->> Select the Team Project -->> Select Test
tab >> Select the test plan -->> Select the specific test case -->>
View Results -->> At the end of the opened page you will see the result history (All test runs for the specific test case).
We also found that the ability to view easily view a test cases' result history was lost when we moved from MTM.
There is a feature request uservoice here https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/2037641-provide-more-visibility-for-a-test-case-s-result-s
However, if you have got MTM handy, you can use the the following workaround
1) In TFS, click on "Test" from the top menu and select the test suite where your test case is. Select the test case that you are interested in. Then Click pass or fail button. This will generate a manual test run for the given test.
2) Go to MTM -> Test --> Analyze Test Run. Select option "Manual Runs" in the View option.
3) Open the test run. Right click on test and click "View Results"
4) The list of results will show you the manual run as well as automated runs, which is what you are looking for.
Hope it helps!!
In the old TFS 2013 days, if you wire up your automated test cases with MTM and run them via MTM, you can view all historical test results for each of those test cases in the Test Plan.
With TFS 2015 Build/Release + Run Functional Tests tasks, in MTM, you can only see the latest result. How can you view all historical runs of a particular test case? In the web portal, you can see "Test Runs", but there's no way to find just the runs pertaining to the particular test case in question.
I bumped into a workaround for this by accident. If I manually mark the test as Pass/Fail, then when I go back to MTM, and "View Results" again for this test case, it will all of a sudden display everything in its history, including the automated test runs generated by builds or releases.
It seems like a bug in the MTM UI, but this works for now!
Using the default build template on TFS 2013, we use the Post-test script to collect trx files from protractor tests, in order to include those in the build report summary.
We use vs2015 MsTest:
MSTest.exe
/publishresultsfile:...\TRXTestResult\CHROME-54.0.2840.71-results.trx
/publish:https://.../tfs/defaultcollection
/teamproject:...
/publishbuild:...
/platform:AnyCpu
/flavor:Release
Result:
Waiting to publish...
Publishing results of test run (test name) to
https://.../tfs/defaultcollection ...
Publish completed successfully.
However, the published test result wont show up in the build report (only the default from unit test).
This can't be achieved by a simple test script. You need to create a custom activity to handle this.
Inside this activity you can handle everything that is necessary to run the protractor tests and publish the test results. With the custom activity you are also able to integrate the automated protractor tests very easy into different build templates. This improves the reusability in a great way.
Then you need to create the custom build template and integration into a TFS Build. More details please refer the tutorial from this blog: Automated UI testing of AngularJS applications using TFS
I'm trying to setup TFS 2010 Lab Management based Automated Testing system and I have a "Test suite" (Created in TFS Test Manager) with following tests:
1) Start and Login
2) Create group
3) Create user and add to the group
These have order specified properly in Testing Center->Plan->(Test suite)->Order column
However, when they are executed (on TFS Lab Management Build) by Test Agent they run out of order: 2nd then 1st and then 3rd.
Is there a way to make Test Agent run these tests in order?
Thanks.
Create an Ordered Test, it's a simple list of which test methods should run and in what order.
I also had the same issue. I solved it by installing VS2010 Service Pack and TFS 2010 Service Pack in both the Test controller machine and Test agent machines. Please go through the following link for more details
http://social.msdn.microsoft.com/Forums/en-US/vsmantest/thread/29ac3c7b-b0cd-40f3-9992-f5f3c2285331
Unfortunately, There's a blog regarding ordering test cases which says:
Please note this only applies to the manual tests but not automated
tests. For automated tests, the order you set here will not be
respected during test execution.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/11/21/run-your-tests-in-the-order-you-want.aspx
I don't know if it's a new feature but you can create Ordered tests. Just right click on your project in Solution Explorer and choose Add -> Ordered Test. Select you tests in order.
In Test Explorer, run you ordered test.
Details are in the link below but all I had to do is what I'd said above. I am using VS 2015 Enterprise.
https://msdn.microsoft.com/en-us/library/ms182631.aspx