One team I am working on uses Jenkins + PHPUnit for CI. I am looking for a good way to indicate a particular person is addressing a particular test which may be failing the build. The Claim plugin seems great for claiming an entire failing build but I'm looking for some way to make it more granular. It would be particularly useful to make a "sticky" per-test claim so if the same test fails next build the Claimer is retained. Can Claim do this (I thought it could, but I must be missing something if so)? Can something else do this?
This is provided by the Claim plugin, as indicated by MrsTang. However, you need to activate this per job.
In the Job configuration, under the Post-Build Action Publish Test Results you will find a check box Allow claiming of failed tests. That will allow you to claimed failed tests. We found that feature to work quite fine.
Related
We use SpecFlow and NUnit in Visual Studio at work. Very useful, etc, etc. Once we've finished development the checked in code goes to a build server which uses Jenkins to build and run all the tests. Very helpful, etc.
However, there is an annoying hole: if you a/ use a step in SpecFlow that you don't define, a default step definition is used which marks the test as Inconclusive, and then b/ NUnit ignores Inconclusive tests (apparently MSTest fails them instead), and then c/ Jenkins doesn't detect a problem and passes the build. This means our build servers can pass tests that aren't even defined correctly.
So, can anyone EITHER:
- tell me how get SpecFlow to throw errors on missing steps? (can't find an option for it or anything on the web) OR:
- tell me how to get NUnit to treat Inconclusive tests as failing? (once more, can't find an option for it or anything on the web) OR:
- somehow get Jenkins to pick up the Inconclusive results and treat that as failing?
All suggestions will be considered, left-field ones included! Thank you.
There is a configuration option to change this behavior.
See here for the documentation. The configuration we're interested in here is missingOrPendingStepsOutcome.
The default setting is:
missingOrPendingStepsOutcome="Inconclusive"
We simply need to change (or add the setting) as follows:
missingOrPendingStepsOutcome="Error"
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
Long story short,
I was wondering if anyone ever felt the need for (and knows of any implementation of) the possibility of "instantiating" (OO terminology) a parametrized build.
What I mean is treating a parametrized build as a template, from which many "instances" can be generated.
Each instance is supposed to define a different combination of values for the parameters.
The final goal is twofold:
DRY (which is given simply by the parametrized build concept)
having separate build histories / test reports for each instance (otherwise it would be a mess)
the instances would be schedulable directly in jenkins UI (while a parametrized build is not)
The template would then be used only for:
manual builds
changing the config for all of the instances at once
Now, time for some context, as I may be missing something in my overall approach.
You are welcome to point me in the right direction :)
I have a maven project with a suite of selenium tests that I want jenkins to run.
The suite is parametrized: browser, OS, test environment.
So, I can run it e.g. with mvn test -Dbrowser=chrome -Dplatform=win [..].
I want a separate test report for each combination of my parameters.
As a newbie, my first solution was "Copy existing job".
Quick and dirty. But effective.
As you will know, problems arise when you need to make a change to the configuration of the job, and you want to keep in sync all of these copy&pasted jobs.
Then I found the parametrized build feature.
It's very cool (code reuse/maintainability++), but the test report and the build history is shared among all of the actual builds, therefore I can not rely on them for a tidy reporting like "this test is always failing on IE; but it isn't on chrome", and so on.
Thank you very much in advance
I think what you are describing is the matrix project
There are also selenium plugins, I put one together to work with matrix jobs https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Axis+Plugin
One lack I can see: you can't build a single combination, as the build btn is present only at the "top level".
Have you tried the Matrix Combination plugin
https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Combinations+Plugin
Jenkins says a build succeeded or failed, but can it identify the exact commit (and author!) that caused a build to fail?
This issue would seem to indicate no.
Edit: From my exchange with Pace:
What I see is "include culprits", which is everyone since the last
build. I don't want that. I want THE culprit, with Jenkins doing the
binary search. If Jenkins does two builds 10 commits apart, I don't
want 10 possible culprits, I want it to find the one.
I haven't yet heard how to do that.
That page was talking about the "find bugs" plugin, not the normal build cycle. Depending on how things are setup Jenkins can identify the exact commit and author that caused a failure. If Jenkins has the appropriate source control plugins installed and is configured to know about the repository the build is tied to then for every build it will list the changes since the last build.
In addition, Jenkins has the capability in many of its reporting plugins to blame the faulty committer. It can, for example, send an e-mail notification on a failed build to the developer that made the faulty commit.
However, many setups make it difficult for Jenkins to know. For example, if Jenkins is configured for daily builds then there are likely many commits which could have caused the issue. It's also possible that Jenkins isn't configured to know about the source control repository, or there is no source control repository. All of these issues could cause Jenkins to be unable to identify the build breaker.
Specifically for e-mailing faulty committers you can use the email-ext plugin which has options to send e-mails to everyone that committed since the last successful build.
For a humorous take on this subject check out this approach.
I think what you're asking for is impossible in some cases. Determining who the culprit is requires insight into conflict resolution that only a human can decide. Even still, sometimes a manager has to be involved in order to arbitrate. Say for instance you get 3 commits (A,B,C) that depend on a preexisting definition. However, another commit (D) modifies the behavior of that function. Which do you revert? Perhaps it's the business plan to keep A,B,C as is and return D to its original state. The opposite, modifying A,B,C to adapt to the changes of D, is also possible.
In the cases where a machine can handle the arbitration, it is the responsibility of unit tests, and static analyzers, to determine the culprit (although still imperfect). Static analyzers sometimes have built in features that email the person who committed a violation. Unit tests can be written that notify teams or team members responsible for a failed test. Both could work in the same way that identifies who was the last committer on a particular line that failed. Still, if it is a problem with linking, then perhaps some members should be associated with the particular makefile.
If I create a bug in TFS, is it possible to associate it with a unit test and be notified if this bug is fixed/broken?
Not exactly, but the workflow can go the other way: instead of creating a bug up-front, you can let Team Foundation Server handle the work item creation for you. If you create a continuous integration build that runs your unit tests, you can create a bug assigned to the user who submitted the changeset if that build fails (including because the unit tests fail.)
A good article on configuring continuous integration to do this is in MSDN Magazine.
A new feature in TFS 2010 is Gated Check-in, which will prevent this failing code from getting checked-in by requiring the build and test steps to complete successfully before they get committed.
There is, currently, no mechanism to do what you desire. You can force users to check in changes tagged with a work item, which you can then manually examine. Provided the user either alters a test class or creates a new one, and checks in with the fix, you can see whether it is done or not. But, there is no automagic "if this test is changed, alert me".
If you need some type of "change and alert" functionality, I would examine CodePlex and see if there are any tools of this type. It could be facilitated at the database layer, as well, but I am not sure what kind of monkeying that would take. I have not looked at the service endpoints for the latest TFS, so there may be a way to poll to see if there are changes.
Wish I had a better answer, but this is an interesting ASK, so I would consider going to Microsoft Connect for TFS and requesting the feature. It could be included in a future rev if enough people are interested in this feature.