Using the Microsoft.TeamFoundation.Build.Activities.RunAgileTestRunner activity that shipped with VS2013RTM, I have unit tests set up like this:
When I run these builds, the tests run fine, however there are different results in the count being reported every time. Not only are the counts different between builds, but the Build Summary page for a given build run will also consistently shows different counts of tests than the actual test results (.trx) file.
I also tried the Microsoft.TeamFoundation.Build.Workflow.Activities.RunTests activity, and I get the same issues with the test results described above. However, I am able to use its RunTestsTestCompletedEventArgs to write a build message of the name of the test followed by the result, and when I do this I see the same number of tests run each time, which means something is wrong with the summary report and/or the published results (right? RIGHT???):
Also, I have two assemblies with 'tests' in the name which get picked up by the filter, I know that this aspect is working because the bin folder on the build box will have same size on disk among all of the different runs which produce different results.
So, what is going on with these numbers? What am I doing wrong in my build definition or on the build box that is causing this?
Related
I setup my jmeter to run on jenkins daily. Everything works fine except the past jenkins build automatically flip to fail status the next day when a new schedule run kicks in and it also removed the past build results. I attached a screenshot of the build history. For sep 12, they were all green.
Anyone experience this issue or know which area I can look into?
My goal is to try to compare with past build result and send an email if the performance didn't pass criteria.
I am happy to share more details.
Unfortunately we're not telepathic enough to guess the reason by looking at a single screenshot.
Double check your job configuration, especially location of the JMeter's jtl result files as it might be the case you're storing them in current job folder instead of workspace
Double check your Performance Plugin configuration, especially "thresholds" section
Inspect Console Output for the previous/latest builds
Check jenkins.log file for any suspicious entries
We recently added Coded UI Tests to our solution. The tests complete successfully when ran through Test Explorer but when the code is checked in and a build is triggered, all the CUIT's fail (error message is below).
I have gone to each of the links in the error message. The first one details how to set up a test agent to run the tests. We don't really need tests on all our environments as we are setting up a lab environment to run them and adding the test agent would require me submitting a ticket which would probably take months to get a response back, not to mention walking the person through what needs to be done.
The second link is dead and I'm pretty sure I don't want to change the build to be interactive.
I am hoping there is an easy way to change the build definition (which I have access to) so that it can ignore all Coded UI Tests but still run the Unit Tests. Is this possible? Is there an easier way of going about this? Each set of tests have their own project file, all Coded UI Tests in one project and all Unit Tests in another project.
Thanks in advance.
Here is the error that is on all of the coded UI tests:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735).
One thing you can do is to group all the coded UI tests into a particular test category and exclude running those in the build. We do this with our "integration" tests and only choose to run our true unit tests for speed reasons.
See below link for how to use test categories:
https://msdn.microsoft.com/en-us/library/dd286683.aspx
Also see the below for how to edit your build definition to only run the tests you want:
Excluding tests from tfs build
You can use use some naming patterns and combine that with the Test case Filter option in the execute test step:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
Something like:
FullyQualifiedName ~ .Ui.
This would require the Ui tests to have .Ui in the namespace
I am trying to run UI tests on multiple identical instances of the web application. For example, let's say the identical version of the application is available at 3 places:
https://some1.app.com
https://some2.app.com
https://some3.app.com
The intended system should check which instance is available and run a test (that is not already run) on it. It should be able to run 3 tests on the 3 instances simultaneously in the Jenkins environment.
I have explored the Jenkins Matrix Configuration, but that appears to run all tests on all possible combinations in the matrix. My intention is to divide and load balance the tests, not run on all combinations. Any ideas on how this can be done?
I am using JUnit4 with Ant for running the tests on Jenkins.
One solution would be a Matrix Project Plugin. You could configure your url as parameter a bit like in here: Building a matrix project
I am using the combination of Jenkins, python, unittest, nosetests to run test suites. I publish the results in nosetests.xml to Jenkins using Junit plugin.
My question: How can I run the same test suites with different browsers (Chrome, FF, IE,…,etc) and publish all the results in the same Jenkins job and still be able to distinguish each test result with respect to browser it used?
I am thinking about running the test suite many time, each with different browser and rename the tests in each nosetest report, nosetests.xml, before publishing to Jenkins but I don’t think that it is an elegant solution.
Thanks for your help
This is what multi-configuration jobs (or matrix jobs) are designed for in Jenkins.
You specify your job configuration once, but add one or more variables that should change each time, building a matrix of combinations (in your case, the matrix has one dimension: browser).
Jenkins then runs one main build with multiple sub-builds — one for each combination in the matrix. You can then clearly see the results for each combination.
This requires that your test job can be parameterised, i.e. you can choose at runtime which browser should be run, rather than running all tests together in a single job.
The Jenkins wiki has minimal documentation on this feature, but there are a few good blog posts (and Stack Overflow questions) out there on how to set it up.
Long story short,
I was wondering if anyone ever felt the need for (and knows of any implementation of) the possibility of "instantiating" (OO terminology) a parametrized build.
What I mean is treating a parametrized build as a template, from which many "instances" can be generated.
Each instance is supposed to define a different combination of values for the parameters.
The final goal is twofold:
DRY (which is given simply by the parametrized build concept)
having separate build histories / test reports for each instance (otherwise it would be a mess)
the instances would be schedulable directly in jenkins UI (while a parametrized build is not)
The template would then be used only for:
manual builds
changing the config for all of the instances at once
Now, time for some context, as I may be missing something in my overall approach.
You are welcome to point me in the right direction :)
I have a maven project with a suite of selenium tests that I want jenkins to run.
The suite is parametrized: browser, OS, test environment.
So, I can run it e.g. with mvn test -Dbrowser=chrome -Dplatform=win [..].
I want a separate test report for each combination of my parameters.
As a newbie, my first solution was "Copy existing job".
Quick and dirty. But effective.
As you will know, problems arise when you need to make a change to the configuration of the job, and you want to keep in sync all of these copy&pasted jobs.
Then I found the parametrized build feature.
It's very cool (code reuse/maintainability++), but the test report and the build history is shared among all of the actual builds, therefore I can not rely on them for a tidy reporting like "this test is always failing on IE; but it isn't on chrome", and so on.
Thank you very much in advance
I think what you are describing is the matrix project
There are also selenium plugins, I put one together to work with matrix jobs https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Axis+Plugin
One lack I can see: you can't build a single combination, as the build btn is present only at the "top level".
Have you tried the Matrix Combination plugin
https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Combinations+Plugin