I have a weird situation with Jenkins... We've just started using Gradle for a project at my job and when I run the tests locally with JUnit everything is fine. But when these tests are run by jenkins for the builds of branch "A", only one test fails because of an assert(always the same test).
org.junit.ComparisonFailure: expected: "E[ZZ0530]Z" but was:"E[SY5654]Z"
It looks like the mock is not injected or the mock is ignoring the "when" mocking statement.
Here is the test :
#Test
public void testEvent() {
Date eventDateTime = TimeUtils.parseDate("2013-05-30 00:00:00");
event.setEventDatetime(eventDateTime);
//Mocking the prefix return
Mockito.when(eventCodeHelperMock.getEventCodePrefixFromEvent(event)).thenReturn("EZZ");
//Tested methode
eventWrapper.setSuffix("Z");
// Event code = prefix + date + suffix
assertEquals("EZZ0530Z", event.getEventCode());
}
What is a lot stranger is that when I create a branch "B" from branch "A" all the tests succeeds when the build is created on jenkins.
I've made some research and tried to force an other build, wipe out the current workspace and recreating the job but it didn't work.
Thanks for your help!
I have had similar problems in the past and it has been due to the order in which the junits tests are run. For example, one test modifies the state of an object but you dont see the effects of this till the tests run in a different order, and tests unexpectedly fail. There is not sufficient code in the question you have posted to tell whether this is definitely the case, but I would recommend checking the order in which the tests are being run, and also look at the objects that you are using to determine if there is a problem with the state of those objects being 'dirtied'.
Related
Hello i am new to jbehave and thucydides and the issue i am facing is all the steps are executed in the .java file but it only skips the #when step due to which my test gets skipped. I tried several options but it always marks when as pending when i run the test.
After executing tests case, check your console or report file for story/step error annotation.
Tests that contain no steps are considered to be pending. If one of steps (the "given-when-then" structure) gets PENDING while executing, then whole test gets labelled as SKIPPED. http://www.wakaleo.com/thucydides-one-page/thucydides.html#_defining_high_level_tests_in_junit - 6.2.1
From my experience, most pending steps ("given-when-then") are from bad spelling of a step name/title. Step from .story file and implementation file of a story (your_story.java depending on language) are different. Like "xx yy" =/= "xx yv"
I have two projects and with dependencies so that project A is started, it updates files from git and then runs a multi-configuration project B, which:
has three axes: "foo", "bar" and "baz" with 11 x 4 x 2 items
I'm going to call the values like fooN for item N from axis foo, etc.
has a configuration filter, ruling out the last axis by running only when
baz=="baz1" (maybe in a later phase we'll want to run also tests with baz2
for baz)
runs a shell script that only cds and calls python interpreter with a script
cd /path/to/scripts
python test_${bar}.py
So when the project is run, I expect 44 configurations to be tested. But only 43 are.
It's always the same configuration (which happens to be the last one triggered, as Jenkins seems to remember the order(?)) that does never run at all:
in the final matrix looks as a gray dot with "Disabled" tooltip
in Console output, after saying "Triggering bazN,barN,fooN" for all 44 combinations,
then "bazN,barN,fooN completed with result SUCCESS" for all except the last one, but
the last one seems to be always cancelled/aborted:
baz1,bar7,foo3 appears to be cancelled
baz1,bar7,foo3 completed with result ABORTED
Console output for the single combination is not available---it looks like it never
has been built
Jenkins log does not show anything interesting about "baz1,bar7,foo3"
What does this mean? Any other pointers how to troubleshoot this?
Edit: I tried adding a "HTTP ping" script to the repo and called it from above script,
just before the python test_${bar}.py part. Which proved that for the affected
configuration, Jenkins does not even run those lines.
Without knowing how you got here to begin with (probably a bug):
Append configure to the URL of the disabled configuration, and in the resulting form, uncheck Disabled and Save.
Not really an answer, but as a workaround to the problem, cloning whole project to a new one helped: with the new project, all configurations ran normally.
This is a solved Jenkins issue:
https://issues.jenkins-ci.org/browse/JENKINS-19179
By the Matrix Project plug-in, version 1.4:
https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Project+Plugin
From time to time I run into the issue that Grails integration tests the name of which ends in "IntegrationTests" don't work coming up with exceptions that show that GORM methods have not been added to domain classes. After renaming those tests to "*IntegrationTest" (no s at the end) they work fine.
A short example:
class MyIntegrationTests {
#Test
void myTest() {
assert MyDomainClass.count() == 0
}
}
Will fail with the following exception:
Failure: myTest(de.myproject.MyIntegrationTests)
groovy.lang.MissingMethodException: No signature of method: de.myproject.MyDomainClass.count() is applicable for argument types: () values: []
Possible solutions: count(), ident(), print(java.io.PrintWriter), print(java.lang.Object), getCount(), wait()
at de.myproject.MyIntegrationTests.myTest(MyIntegrationTests.groovy:9)
After renaming MyIntegrationTests to MyIntegrationTest the test passes.
Is there some kind of magic happening according to the test's name? All I found in Grails documentation is: "Tests can also use the suffix of Test instead of Tests. " Any ideas?
I eventually found the cause for the different behaviour of "*Test" and "*Tests" myself: Different postfixes change the order in which the tests are being run. To make things worse, the exact order is platform-dependent. Thus, my tests ran locally (OSX) in a different order than on my CI machine (Linux), and thereby produced different results.
Why the exception occurrs in some order is a totally different problem, though, which I haven't figured out (yet).
This should work how you had it originally as long as the file is in the integration folder. Are you sure you didn't have it in the unit test folder and then move it into the integration folder on the rename? Or possibly that you're using intellij and you did a "junit" test run rather than a "grails" one?
The error you're getting seems to imply that grails didn't start up when running your test.
Your test won't be executed if it does not has the suffix Tests.
Copied From Grails documentation home page (http://grails.org/doc/latest/guide/testing.html):
The default class name suffix is Tests but as of Grails 1.2.2, the
suffix of Test is also supported.
j-
I was wondering if any of you guys had any experience generating code coverage reports in TFS Build Server 2010 while running NUnit tests.
I know it can be easily done with the packaged alternative (MSTest + enabling coverage on the testrunconfig file), but things are a little more involved when using NUnit. I've found some info here and there pointing to NCover, but it seems outdated. I wonder if there are other alternatives and whether someone has actually implemented this or not.
Here's more info about our environment/needs:
- TFS Build Server 2010
- Tests are in plain class libraries (not Test libraries - i.e., no testrunconfig files associated), and are implemented in NUnit. We have no MSTests.
- We are interested in running coverage reports as part of each build and if possible setting coverage threshold requirements for pass/fail criteria.
We 've done it with NUnit-NCover and are pretty happy with our results. NUnit execution is followed by NUnitTfs execution in order to get our testing results published in the Build Log. Then NCover kicks in, generating our code coverage results.
One major thing that poses as a disadvantage is fact that setting up the arguments for properly invoking NCover wasn't trivial. But since I installed it, I never had to maintain it.
Two things could pose as disadvantages:
NUnitTfs doesn't work well with NCover (at least I couldn't find a way to execute both in the same step, so (since NCover invokes NUnit) I have to run Unit tests twice: (1) to get the test results and (2) to get coverage results over NCover. Naturally, that makes my builds last longer.
Setting up the arguments for properly invoking NCover wasn't trivial. But since I installed it, I never had to maintain it .
In any case, the resulting reporting (especially the Trend aspect) is very useful in monitoring how our code evolves within time. Especially if you 're working on a Platform (as opposed to short-timed Projects), Trend reports are of great value.
EDIT
I 'll try to present in a quick & dirty manner how I 've implemented this, I hope it can be useful. We currently have NCover 3.4.12 on our build server.
Our simple naming convention regarding our NUnit assemblies is that if we have a production assembly "123.dll", then another assembly named "123_nunit.dll" exists that implements its tests. So, each build has several *_nunit.dll assemblies that are of interest.
The part in the build process template under "If not disable tests" is the one that has been reworked in order to achieve our goals, in particular the section that was named "Run MSTest for Test Assemblies". The whole implementation is here, after some cleanups to make the flow easier to be understood (pic was too large to be directly inserted here).
At first, some additional Arguments are implemented in the Build Process Template & are then available to be set in each build definition:
We then form the NUnit args in "Formulate nunitCommandLine":
String.Format("{0} /xml={1}\\{2}.xml", nunitDLL, TestResultsDirectory, Path.GetFileNameWithoutExtension(nunitDLL))
This is then used in the "Invoke NUnit"
In case this succeeds & we have set coverage for this build we move to "Generate NCover NCCOV" (the coverage file for this particular assembly). For this we invoke NCover.Console.exe with the following as Args:
String.Format("""{0}"" ""{1}"" //w ""{2}"" //x ""{3}\{4}"" //literal //ias {5} //onlywithsource //p ""{6}""",
NUnitPath,
Path.GetFileName(nunitDLL),
Path.GetDirectoryName(nunitDLL),
Path.GetDirectoryName(Path.GetDirectoryName(nunitDLL)),
Path.GetFileName(nunitDLL).Replace("_nunit.dll", ".nccov"),
Path.GetFileNameWithoutExtension(nunitDLL).Replace("_nunit", ""),
BuildDetail.BuildNumber)
All these run in the foreach loop "For all nunit dlls". When we exit the loop, we enter "Final NCover Activities" & at first the part "Merge NCCovs", where NCover.Console.exe is executed again - this time with different args:
String.Format("""{0}\*.nccov"" //s ""{0}\{1}.nccov"" //at ""{2}\{3}\{3}.trend"" //p {1} ",
Path.GetDirectoryName(Path.GetDirectoryName(testAssemblies(0))),
BuildDetail.BuildNumber,
NCoverDropLocation,
BuildDetail.BuildDefinition.TeamProject
)
When this has run, we have reached the point where all NCCOV files of this build are merged into one NCCOV-file named after the build + the Trend file (that monitors the build throughout its life) has been updated with the elements of this current build.
We now have to only generate the final HTML report, this is done in "Generate final NCover rep" where we invoke NCover.reporting with the following args:
String.Format(" ""{0}\{1}.nccov"" //or FullCoverageReport //op ""{2}\{1}_NCoverReport.html"" //p ""{1}"" //at ""{3}\{4}\{4}_{5}.trend"" ",
Path.GetDirectoryName(Path.GetDirectoryName(testAssemblies(0))),
BuildDetail.BuildNumber,
PathForNCoverResults,
NCoverDropLocation,
BuildDetail.BuildDefinition.TeamProject,
BuildType
)
In Jenkins there is a possibility to create free project which can contain a script execution. The build fails (becomes red) when the return level of the script is not 0.
Is there a possibility to make it "yellow"?
(Yellow usually indicates successful build with failed tests)
The system runs on Linux.
Give the Log Parser Plugin a try. That should do the trick for you.
One slightly hacky way do do it, is to alter the job to publish test results and supply fake results.
I've got a job that is publishing the test results from a file called "results.xml". The last step in my build script checks the return value of the build, copies eihter "results-good.xml" or "results-unstable.xml" to "results.xml" and then returns a zero.
Thus if the script fails on one of the early steps, the build is red. But if the build succeeds its green or yellow based on the return code it would have retunred without this hack.