For GUI testautomation we have set up a Maven project using JBehave (v4.0.4) as a BDD framework. The stories are executed with a JUnitReportingRunner and run continously in a Jenkins (v1.630) master slave environment.
I recently noticed that in some cases, the Jenkins build is marked as successful despite some failed steps. The XUnit test report correctly indicates that there is a failed test while the Jenkins build does not. We haven't configured any thresholds concerning the build status. Therefore one failed test should cause the build to fail (and it does most of the time).
This problem is very annoying since we heavily rely on the Jenkins mail notifications. Any pointers on how to solve it are very much appreciated.
Related
Is there any way in jenkins where as soon as we detect the failed build, job revert back the perforce code to the last successful build changelist and fire a build again.
Flow -
1. so as soon as we have failed build - Notifcation will be sent out to dev team with possible checkins which causing the build failure
Revert back the recent code to the last working code and submit it
Initiate a build.
It is possible, but I don't see any reasons or use case to do it as it is not a correct workflow and can be confusing.
But if you decided to do it, the next steps are required:
Example how to do it using Perforce source control.
Steps inside job settings:
Before build triggers you need to save latest changelist number $P4_CHANGELIST - 1
Perforce plugin for Jenkins: Perforce plugin for Jenkins
Build
Get last error code
Get last error code from the: batch
If code != 0 then checkout and build changelsit $P4_CHANGELIST - 1
Jenkins is not a production server. It runs tasks and do not have options that I know for that purpose.
What is your source code ? webapps ? others ?
What steps are you performing ?
Are you performing some automatic tests ?
My assumption is that you got some tests that may invalidate the build.
These tests should be runned :
* on a mock server to prevent deploying on your server
* or somewhere else
Like that, if build failed, nothing is deployed.
If build success, you can deploy your project normally.
If this not reply to your answer, please provide requested information to undestrand a bit more your job process.
If your using an artifact repository like Nexus or Artifactory to manage your project artifacts then you could always redeploy the previous working version of your application when a failure is detected.
Your not cancelling any checked in code that potentially broke the build but you are preserving your test environment. You can configure Jenkins to notify the user who checked in the latest erroneous change set and they can work on resolving the issue.
Jenkins also provides a rich API which allows you to delete a job, start a job, get information about previously run jobs. You could leverage some of these services along with your artifact repo to achieve the experience you described.
I'm working on a team that is building a RESTful HTTP service. We're having trouble with setting up a Jenkins CI job which will build the service, run it in the background, execute some tests, and then terminate the servers.
Specifics
The server is built in Node.js using the hapi framework and has some unit tests written in mocha.
The tests are written in Java using Maven. (Why not node.js-based tests? Because our testing dept. has invested time in creating a Java-based REST-testing framework.)
The build should fail if the node-based unit tests fail or if the java tests fail.
Our Jenkins box is run by a support team elsewhere in the company; our builds execute on a Linux slave.
Current Attempt
We've got something that kind-of works right now, but it's unreliable. We use 3 build steps:
The first build step is an Execute Shell step with the following commands:
npm install
npm test
node server.js ./test-config.json &
Second we do a Invoke Maven 3 step that points to the test pom.xml.
And third we run Invoke Standalone Sonar Analysis to do static code analysis.
This mostly works, but we depend on Jenkins' ProcessTreeKiller to stop the services once the job completes. We always get the warnings stating: Process leaked file descriptors. See
http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+buildfor
more information
Unfortunately, we've had cases where the service is terminated too soon (before the tests complete) or where the service doesn't get terminated at all (causing subsequent builds to fail because the port is already in use).
So we need something more reliable.
Failed Attempt
We tried setting up a single shell script which handled starting the service, running maven, killing the service, then outputting an exit code. But this didn't work out because the mvn command wasn't available on the command-line. Our Jenkins has multiple maven versions available (and jdks too) and I don't know where they live on the slaves or how to get at them without using the Invoke Maven 3 build step.
Ideas
We've toyed around with some ideas to solve this problem, but are hoping to get some guidance from others that may have solved similar problems with Jenkins.
Have the service self-terminate after some period of time. Problem is figuring out how long to let them run.
Add a build step to kill the services after we're done. Problem is that if the maven execution fails, subsequent steps won't run. (And if we tell maven to ignore test failures, then the build doesn't show as broken if they fail.)
Try killing any existing service process as the first and last steps of the build. Problem is that other teams also use these Jenkins slaves so we need to make sure that the service is terminated when we're done with our build.
Start and stop the node.js services via Maven doing something like this blog suggests. Problem is that we don't know if Jenkins will identify the spawned background task as a "leaked file descriptor" and kill it before we're done testing.
It would be nice if Jenkins had a "Post-build action" that let you run a clean-up script. Or if it had a "Execute background process" build step which would kill the background items at the end of the build. But I can't find anything like that.
Has anyone managed to get Jenkins to do anything remotely like this?
Some brainstorming:
You can turn off Jenkins ProcessTreeKiller, either globally or per invocation. I am not sure why that is not an option for you.
In response to #2, several options:
Post-build actions get executed regardless if build steps had failed or not. This would be a great way to trigger a "service cleanup" task that will run regardless of the build state.
You can setup any build step as post-build action, using Any Build Step plugin, or you can use Post Build Tasks plugin, the latter even gives options to define triggering criteria.
You can change the build state, based on RegEx criteria using Text-finder plugin
You can setup Conditional Build Steps. The "condition" could even be a result of some script execution
I am using Jenkins for integration testing.
Just to give the context. At the moment I have a separate build server which produces the build daily and Jenkins is not used as the build server. The build server executes the unit testing in my case.
When build process is complete it invokes the Jenkins job. In that job Jenkins start to deploy the build into the Virtual machine. I have a script for doing this.
Followed to that my plan is to run several scripts for doing the end-to-end testing.
Now I have several question in this regard:
How to parallelize the execution of the end-to-end tests?
As I am adding scripts after script I am getting worried how manageable it will be?
I am always using the web interface for adding and changing the scripts. How to do this from the command line?
Any ideas for a good tutorial? Any pointers from all of you? Thanks!
Looks like Build Flow Plugin is what I need.
https://github.com/jenkinsci/build-flow-plugin
You might want to try and see if you can use the Build Pipeline plugin before build flow. Much better visualization of what is going on, less scripting.
I link Build and deploy jobs in one sequence and then have unit and integration test jobs linked separately off the build job. You can then use Fail The Build plugin to have downstream jobs fail upstream ones.
I'm new to Jenkins/Hudson but I didn't find a plugin which provides the functionality I am looking for.
I want to make the build process of my web application dependent on multiple function tests I already implemented in JMeter and soapUI. So after the project is built, the tests should start. If they fail (to a certain amount), the whole build should fail (and not only the job containing the tests).
As I have read, post-build actions and JMeter/soapUI tests are not a problem. The only missing things seems to be the connection to the success of the whole build. Is there a plugin or a workaround for my requirements?
Regards
You need the Performance Plugin:
Add a JMeter Report and define the path to your jtl files.
To mark the build to status as failure you need to specify the Failed threshold.
Example:
If you specify a threshold of 20%: If 20% or more of your jmeter samples fail, the jenkins job is marked as failure.
I would like to use Jenkins for continues testing. Job scenario should look like: 1. connect to svn, call scripts to check-out repo, build and run test. Test will e.g. check if numbers in some output file are same as in reference file. All scripts are in python. The question is - how to force Jenkins to indicate job as "Fail" when numbers in files are different?
Thanks in advance
Typically you would use Jenkins a little differently. It is possible to do exactly what you want but Jenkins and python can eliminate some work for you.
First, Jenkins has the capability to check out stuff from various version control systems, including Subversion. This is the usual way to use Jenkins. When you set up a job, you tell the address of your version control repository and after that you configure the steps to execute after the files have been checked out.
Second, you should use the python test framework http://docs.python.org/library/unittest.html to write your tests. Then you can use e.g. nosetests utility to run the tests. Nosetests can even create test reports in xml format and Jenkins can read the reports and show you which tests failed and why they failed (if the report includes that information.)
But to answer your question: If you exit the build step with non-zero exit code, Jenkins will interpret that to mean the build failed and it will mark that build with a red ball.
Jenkins can mark builds as stable (green), unstable (yellow) and failed (red). If the build step exits with non-zero exit code, the build has failed. If the build step exits with zero, Jenkins can be configured to look for test report files. If the test reports have failed tests, the build will be unstable. If all the tests have passed, the build is stable.
Usually people use the failed (red) to indicate a fatal problem in the test (e.g. failure to install the software that should be tested) and then use test reports to indicate unstable or stable build.