I have a TFS build in VS2010. Following the build unit tests are executed.
In the Build summary it tells me that "1 test run(s) completed - 100% average pass rate" but below this it states "No tests were impacted".
I guess Impacted Tests relate to functionality providing the ability to only run tests that were impacted by code checked in?
Or is there a way that i can run only tests which where impacted based on result of Test Impact Analysis.
I have Set "Analyze Test Impact" to True but still no result coming and its executing all test cases in test projects.
The following thing worked for me.
Microsoft.TeamFoundation.TestImpact
http://scrumdod.blogspot.in/2011/03/tfs-2010-build-only-run-impacted-tests.html
Related
im currently facing the issue that i am not able to test "only the things that have changed" in a change-set using vnext with tfs 2017.
when using the "run functional test" step, i can only choose a test assembly but it will always test the changes with the solution i've picked and all tests within the test assembly.
i've tried to split test assemblys into more test categories but running all 2000 tests against a change on 1 file seems little bit too much.
is there a way to only run tests against source code that has been changed?
we want to decrease test time.
You can set multiple test assemblies and separate by semicolon, such as you can specify **\commontests\*test*.dll; **\frontendtests\*test*.dll as Test Assembly for run functional tests task.
If changes come from your feature project, you should test all assemblies. If changes come from part of your test projects, you can only test the changed test assemblies. You can achieve it by two build definition: first build definition with power shell task, second build definition is your current build.
Power shell task in first build (detect what changes and queue second buils):
If the feature project changed, specify your second build with **\*test*.dll for test assembly, and then queue your second build.
If part of test projects changed, specify your second build with **\*project1test*.dll; **\*project1test*.dll etc for test assembly, and then queue your second build.
How do I label flaky tests in junit xml syntax that jenkins uses to give us a report? jenkins gives me a nice report of tests that succeeeded and failed. I would like to know which tests are known to be flaky.
Certainly a flaky test is something that needs to be avoided and fixed. Tests should always pass and if not they need to be worked on.
However we don't live in a perfect world and thus it might be helpful to identify tests that have failed every now and then in the past.
Since Jenkins keeps track of test results you can step through the history of a particular test case (using the "Previous/Next build" links).
Additionally there are two Jenkins plugins that might be helpful:
Test Results Analyzer Plugin:
this plugin lets you see the history of a particular test case (or test suite) at one glance (plus adds nice charts)
Flaky Test Handler Plugin: this plugin has deeper support for failing tests (i.e. rerunning them). It's a bit restricted to maven and git though.
You label such tests not helpful. And your primary goal is to eliminate the "flakiness" of such tests - by identifying the root cause of the problem and either fixing production or test code; or both; or worst case by deleting or #Ignore'ing those tests.
Disclaimer of course: jenkins can't tell you about flaky testcases. How could it?!
If you assume that you have flaky testcases, you could spent some time to create a jenkins setup were all tests are run 5, 10, 50 times against the same build output; to then compare statistics. But this is nothing that would come for free - you will have to implement that yourself.
We recently added Coded UI Tests to our solution. The tests complete successfully when ran through Test Explorer but when the code is checked in and a build is triggered, all the CUIT's fail (error message is below).
I have gone to each of the links in the error message. The first one details how to set up a test agent to run the tests. We don't really need tests on all our environments as we are setting up a lab environment to run them and adding the test agent would require me submitting a ticket which would probably take months to get a response back, not to mention walking the person through what needs to be done.
The second link is dead and I'm pretty sure I don't want to change the build to be interactive.
I am hoping there is an easy way to change the build definition (which I have access to) so that it can ignore all Coded UI Tests but still run the Unit Tests. Is this possible? Is there an easier way of going about this? Each set of tests have their own project file, all Coded UI Tests in one project and all Unit Tests in another project.
Thanks in advance.
Here is the error that is on all of the coded UI tests:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735).
One thing you can do is to group all the coded UI tests into a particular test category and exclude running those in the build. We do this with our "integration" tests and only choose to run our true unit tests for speed reasons.
See below link for how to use test categories:
https://msdn.microsoft.com/en-us/library/dd286683.aspx
Also see the below for how to edit your build definition to only run the tests you want:
Excluding tests from tfs build
You can use use some naming patterns and combine that with the Test case Filter option in the execute test step:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
Something like:
FullyQualifiedName ~ .Ui.
This would require the Ui tests to have .Ui in the namespace
How to make specific test cases run when changes are made to a specific component in TFS?
For example- I have 10 test cases for component A and 10 test cases for component B. When a developer merges a code reated to component A, I want only the test cases related to component A to run
What you need is Test Impact Analysis. Described as this MSDN article,
Test Impact Analysis (TIA) helps in analysis of impact of development on existing tests. Using TIA, developers know exactly which tests need to be verified as a result of their code change.
To enable test impact analysis in a build process, you need to:
1). Configure test impact analysis in a test settings file. Check the "Enabling Test Impact Collection" part of this article.
2). Specify to use the testsetting in the build definition.
3). Set Analyze Test Impact to be true. Check the Enable Test Impact Analysis part of this article for the details.
Additionally, it is possible for you customize your build process template to only run impacted tests: http://blog.robmaherconsulting.co.nz/2011/03/tfs-2010-build-only-run-impacted-tests.html
When creating a test suite in selenium ide it is possible to let all test cases in a test suite run in a continuous manner and see results when finished. I'm looking into creating test suites in Microsoft test manager and possibly automating with the code with cuit, my question is, is it possible to run the tests one after another with no manual interaction, as from what I've seen so far, it seems you have to manually verify the test results in each step for MTM tests and manually verify the pass or fail status at the end of the test?
You can create a test case and tie an automated test case (Selenium/CUIT) to it in Visual Studio. This flips a flag in the test case work item to "automated", and allows you to automatically execute those test cases on test agents.
https://msdn.microsoft.com/en-us/library/dd380741.aspx