I am running Jenkins 1.571. I am building my project with a pom.xml. I have two executions of maven-surefire-plugin to execute two testng suites in a forked mode.
But Jenkins build#/testReport page shows test results for only first suite. Build logs show test cases from both suites are successfully executed.
I want to include results from both suites into report, so I added Publish TestNG Results plug-in, but that doesn't show any results at all. Any idea on what I might be missing?
In my Jenkins configuration, I specified 'TestNG XML report pattern' = '**/target/surefire-reports/testng-results.xml'
Build log shows:
TestNG Reports Processing: START
Looking for TestNG results report in workspace using pattern: **/target/surefire-reports/testng-results.xml
Did not find any matching files.
Finished: SUCCESS
My relevant pom.xml:
<!-- run unit test cases -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<environmentVariables>
<outputDirectory>${project.build.directory}</outputDirectory>
</environmentVariables>
</configuration>
<executions>
<execution>
<id>default-test</id>
<configuration>
<forkMode>once</forkMode>
<suiteXmlFiles>
<suiteXmlFile>${basedir}/src/test/testngSuite1.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</execution>
<execution>
<id>special-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<forkMode>always</forkMode>
<suiteXmlFiles>
<suiteXmlFile>${basedir}/src/test/testngSuite2.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</execution>
</executions>
</plugin>
Eventually I found out that I need to use forward slash for TestNG plugin to locate the file (My Jenkins runs on Windows).
TestNG Reports Processing: START
Looking for TestNG results report in workspace using pattern: C:/workspace/project/target/surefire-reports/Suite1/UnitTests.xml;C:/workspace/project/target/surefire-reports/Suite2/UnitTests.xml
Saving reports...
Processing 'C:\programs\Jenkins\jobs\project\builds\2014-10-02_21-57-48\testng\testng-results-1.xml'
Processing 'C:\programs\Jenkins\jobs\project\builds\2014-10-02_21-57-48\testng\testng-results.xml'
TestNG Reports Processing: FINISH
Finished: SUCCESS
Also this reference helped:
http://www.seleniumtests.com/p/testing-forum.html#nabble-td5001883
1.Select "Editable Email Notification" on Post-Build actions.
2.In the Default Content section : remove $DEFAULT_CONTENT.Place the following code to inject your html report into the default content :-
${FILE,path=”relative path to html file”}
Eg: ${FILE,path=”MyWorkspace/MyProj/test-output/emailable-report.html”}
More detailed steps given below in my tech blog:-
https://itisatechiesworld.wordpress.com/2015/01/06/including-testng-reports-within-the-jenkins-email-notifications/
Also let me know if you have any other reporting related questions using Jenkins.
Will try to help out. :)
Regards,
VJ
Related
Below are my Jenkins setup:
Created Freestyle project
In Build section added :- Invoke top-level Maven targets In
In Post-Build Actions added:- Cucumber reports
However the same is working fine when I am executing from maven. here is the pom.xml setting related to cucumber reports.
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>5.4.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>Sample</projectName>
<skip>false</skip>
<outputDirectory>${project.build.directory}/cucumber-reports</outputDirectory>
<inputDirectory>${project.build.directory}/cucumber-reports</inputDirectory>
<buildNumber>2</buildNumber>
<jsonFiles>
<param>**/*.json</param>
</jsonFiles>
<classificationDirectory>${project.build.directory}/cucumber-reports</classificationDirectory>
<classificationFiles>
<param>*.properties</param>
</classificationFiles>
<checkBuildResult>false</checkBuildResult>
<setSkippedAsNotFailing>true</setSkippedAsNotFailing>
<treatPendingAsFailed>false</treatPendingAsFailed>
<treatUndefinedAsFailed>false</treatUndefinedAsFailed>
</configuration>
</execution>
</executions>
</plugin>
Please let ne know what I am missing..
Unfortunately, cucumber-reports-plugin does not have a setting for its own notFailingStatuses parameter(aka setSkippedAsNotFailing in pom) from cucumber-reporting library.
I have forked the plugin and made some changes to enable that setting. You can check it out here and build a custom plugin yourself.
I am using maven-soapui-extension-plugin:test-multi to run tests splitted in multiple soapUI projects.
I see that a lot of TEST-*.xml are generated in target/surefire-reports but I don't see them on the jenkins build page like I do with other projects containing regular junit tests. It seems that jenkins does not detect those reports.
My plugin is configured as such:
<plugin>
<groupId>com.github.redfish4ktc.soapui</groupId>
<artifactId>maven-soapui-extension-plugin</artifactId>
<version>4.6.4.2</version>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>test-multi</goal>
</goals>
<configuration>
<projectFiles>
<scan>
<baseDirectory>${soapui.folder}</baseDirectory>
<includes>
<include>**/*-soapui-project.xml</include>
</includes>
</scan>
</projectFiles>
<host>${soapui.endpoint}</host>
<outputFolder>${basedir}/target/surefire-reports</outputFolder>
<junitReport>true</junitReport>
<exportAll>true</exportAll>
<testFailIgnore>true</testFailIgnore>
<useOutputFolderPerProject>false</useOutputFolderPerProject>
</configuration>
</execution>
</executions>
</plugin>
Do I need something else in my configuration or in jenkins?
It seems to be a problem with Jenkins which does not publish surefire reports generated outside the surefire-plugin for maven (non-freestyle) projects.
I have a maven build which contains multiple projects which in-turn contains lot of sub maven modules. At the test phase each project will execute their test and programatically I am generating junit test reports and put it in corresponding modules target folder in /target/surefire-reports/. After the build get success in the status page of the build in Jenkins I am able to see the test result as graph. But the problem is Jenkins is not taking all the generated junit xml it takes only partial amount of it. I have totally 850 test cases in the whole build but it only shows 449 in the graph and test results. What will be the cause of it.
There is no problem in generating junit reports all the test cases are generating reports but the Jenkins is not able to identify all. The count of the test cases in Jenkins varies for each build without adding or removing any of the test files.
FYI :
I am manually generating the junit reports. To notify it to Jenkins I have enabled the maven sure fire plugin and added the reports in surefire-reports folder in target. I have created maven build project not the free style project.
Is there is anything i am missing ?
I ran into this as well, and here are my thougths on it:
http://javamemento.blogspot.no/2016/02/sonar-jacoco-maven-multi-module.html
The issue is you need to concatenate the results into one report file.
So your pom should have something like this:
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.5.201505241946</version>
<executions>
<execution>
<id>agent-for-ut</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.reportPath}</destFile>
<append>true</append>
</configuration>
</execution>
<execution>
<id>agent-for-it</id>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.itReportPath}</destFile>
<append>true</append>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
The keyword here is
<append>true</append>
Now all your data will be stored in 1 report file.
Make sure you have called your maven-surefire-plugin in only your parent pom. If you include it in your child poms as well, the results will be overwritten by the parent pom. More quirks here: http://javamemento.blogspot.no/2016/06/sonar-maven-surefire-plugin-quirks.html
Lastly make sure you are using junit and void methods correctly. Test for the side effects of the method or surefire will behave strangely.
http://junit.org/junit4/faq.html#atests_4
I am running Jmeter tests using Jmeter Maven Plugin. After I run the test I want to generate simple reports that indicate if the tests passed/failed. I referred to
https://stackoverflow.com/questions/4669467/jmeter-how-to-create-summary-report-from-jtl-file?lq=1
for generating the reports. I have added a shell script to generate html reports from the jmeter jtl result files.Then I use the exec-maven plugin to execute the script that in turn generates the html report files.So far Everything works fine. The problem that I am facing is that if one of the Jmeter tests fails then the report isn't generated at all.
So I am assuming that maven exits once it detects error in the test-suites and doesn't execute the exec-maven plugin and hence the shell script isnt called at all.
Can someone give me directions on this? Is there any property in the pom file or any settings that I can try to get around this. Any help would be truly appreciated!
You can set the <ignoreResultFailures> configuration setting to true to make the plugin ignore the failures and continue executing everything else.
+---+
<project>
[...]
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>${project.groupId}</groupId>
<artifactId>${project.artifactId}</artifactId>
<version>${project.version}</version>
<executions>
<execution>
<id>jmeter-tests</id>
<phase>verify</phase>
<goals>
<goal>jmeter</goal>
</goals>
</execution>
</executions>
<configuration>
<ignoreResultFailures>true</ignoreResultFailures>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
[...]
</project>
+---+
I am refactoring a single "too" big multiple module maven Jenkins job to about 10 smaller maven Jenkins jobs (one parent maven module with childs).
I like to run a single maven job every 2 hours without the tests and source code analyzers like PMD and Checkstyle, and once a day during the night I want to run it with the tests and source code analyzers.
I am not sure how to do this best.
Jenkins is very flexible and I read the Jenkins O'Reilly book, but I am stil not sure how to do it :(
I was thinking about using the Maven Jenkins plugin with job inheritances, but I still end up with many jobs I guess. Is this the way to go ?
Please some advice?
- Ed
One trick I use is to set a property for build phase for any plugin that I want to disable and set it manually in jenkins. for example see the pmd plugin below:
See ${pmd.phase}
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-pmd-plugin</artifactId>
<version>2.5</version>
<configuration>
<targetJdk>1.6</targetJdk>
<linkXref>false</linkXref>
<failOnViolation>true</failOnViolation>
<failurePriority>1</failurePriority>
<rulesets>
<ruleset>${pom.basedir}/pmd-rulesets.xml</ruleset>
</rulesets>
</configuration>
<executions>
<execution>
<phase>${pmd.phase}</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
Now define
<properties>
<pmd.phase>none</pmd.phase>
</properties>
In jenkins set the Goals and options field to clean install -Dpmd.phase=validate
The command line property overrides defined one so pmd will run only if the -Dpmd.phase=validate is present.