I am using Jenkins for integration testing.
Just to give the context. At the moment I have a separate build server which produces the build daily and Jenkins is not used as the build server. The build server executes the unit testing in my case.
When build process is complete it invokes the Jenkins job. In that job Jenkins start to deploy the build into the Virtual machine. I have a script for doing this.
Followed to that my plan is to run several scripts for doing the end-to-end testing.
Now I have several question in this regard:
How to parallelize the execution of the end-to-end tests?
As I am adding scripts after script I am getting worried how manageable it will be?
I am always using the web interface for adding and changing the scripts. How to do this from the command line?
Any ideas for a good tutorial? Any pointers from all of you? Thanks!
Looks like Build Flow Plugin is what I need.
https://github.com/jenkinsci/build-flow-plugin
You might want to try and see if you can use the Build Pipeline plugin before build flow. Much better visualization of what is going on, less scripting.
I link Build and deploy jobs in one sequence and then have unit and integration test jobs linked separately off the build job. You can then use Fail The Build plugin to have downstream jobs fail upstream ones.
Related
Have seen the following graph representing Jenkins pipeline:
git push --> Git repo --> Jenkins CI server --> Maven build server --> Test server --> Deliver build artifacts --> Deploy
Given that is correct, I am struggling to understand how the above different(?) servers work under the hood and I need some clarification so that Jenkins procedures are not just a black box.
Jenkins CI server, Maven build server and Test server are in reality one physical server?
If the answer to the previous question is yes, these 3 servers are different logical servers?
In my understanding and my case (Java Spring project), Maven build server executes mvn install and since pom.xml contains npm install plus npm run test commands, it is Maven build server that executes the UI tests and not the Test server. Am I right?
Does the Test server execute only the back-end Java acceptance tests?
It could be one physical or virtual server server.
no
To use maven you just need to install maven plugin in jenkins. Configure which version of maven you want to use in "Manage jenkins" -> "Global tools configuration" -> add needed maven version.
Depends on which kind of tests you want to run - you can repeat steps from step 3 for the tool you need or you can install needed tools on the jenkins server manually and add it to the PATH.
At the end just use needed tools in your pipeline or other king of job in jenkins.
Without further context, we must put some points clear:
Jenkins may work on multiples nodes (could be real/virtual/pods)
Jenkins is an orchestrator... This means that it uses different tools in a order given by the pipeline. This order is pretty much: git > build > test > delivery > deploy
Testing server are used for install the software an run a variety of tests, pretty much for several teams.
Build servers are used for running commands to build software
With those clear, these are the questions:
1.- Depends on your infrastructure. Jenkins works with executors that may run on master or in nodes. If your Jenkins is just one server, then yes... It is one physical server. If you use nodes, then it is more probable than you may found one node handling the building, and another one the tests. The definite answer lies within your Jenkins and your pipeline
2.- No... Even with a Jenkins master-agent infrastructure, the building and testing occurs within Jenkins.
3.- Depends on your concept of test server. If you define it as a server where you use as a target for testing purposes, then the answer is no... If you define a test server as a machine that executes test, then the answer is yes.
4.- Depends on what you type or test do you have automatized. You can run unit tests, regression tests, smoke test, etc. For example, you may have some unit test for your back-end, and have some test in karma for your UI. Again, in your case you must check on your pipeline, and the code to check what kind of tests are you running.
I've been asked to create a CI pipeline for a project at my work, I'm creating a load test with JMeter and Taurus so I plan to integrate it with Jenkins to build all the pipeline. I'm just starting on this field and a question that came to my mind is:
What happens to all the data created by the Load Test? does it goes to the deploy phase or it gets deleted once the test is done, should I clean after the tests end?
The data is being kept in the Jenkins workspace and by default it will be kept in the file system forever.
If you decide to publish the artifacts they will be available at Jenkins build dashboard via the web interface.
You might also be interested in Jenkins Performance Plugin which allows plotting performance trend charts and conditionally marking builds as unstable or failed depending on pass/fail thresholds
Example configuration can be found in the How to Run a Taurus Test through Jenkins Pipelines article
I am not completely familiar with your setup but as far as I can see from a quick research, JMeter does the same as every other testing framework and generates HTML reports. Jenkins wont delete them, unless you explicitly delete them (rm file.html) or call cleanWs (clean workspace). If the job is deleted so are the files.
So the test result file should still be present in the deploy phase. You can use a plugin to collect the result. Or just archive it. Or do whatever fits your workflow.
There is generally no need to clean it up (you usually configure Jenkins to delete old builds which takes care of that)
Our requirement for API testing is:
To deploy test-automation module (Karate feature files, custom java classes) into AWS ECS-Fargate cluster.
Trigger the tests via Jenkins pipeline after every build of the actual microservice.
In addition to above, test-automation module should be triggered to run test suite on-demand and/or at scheduled intervals (say nightly) and send reports.
I have gone through Karate Distributed Testing and stand-alone executable jar options, but doesn't seem suitable for my case. Is Distributed Testing supported only for "Web-UI" automation testing?
Any thoughts would be helpful.
For this use-case, just use a Maven + JUnit project and then there is no difference between Karate and any other Jenkins Java pipeline.
It should be Jenkin's responsibility to do a scheduled build. It is up to you how to get all this into Fargate, maybe building a Docker container is part of the answer, but I would recommend trying to keep it simple.
Here is some Docker related discussion that may help: https://github.com/intuit/karate/issues/396
Open a new question with specifics next time.
I've got a Jenkins job that is intended to do the following:
Build a project and deploy it to a test server
Run tests
If the tests fail, roll back the server to the previous version
If the tests succeed, update the version in our source control system
Because we have a single test server, we need to ensure that Jenkins is only running a single version of this job at a time. Unfortunately, we can't seem to find a way to run a job on failure and keep the upstream job from executing while the downstream job is running.
Is there an easy way to do this? Is there a better way?
The Jenkins Post Build Task allows you to run tasks in a job after failure. Rolling the server back sounds more like a task than a job, so that might suit.
Otherwise, there are a couple of plugins that allow for more complex pipelining features. The Pipeline Plugin seems to be the most popular at the moment.
In job configuration, under Advanced Project Options (just before the SCM part), click the Advanced... button. You can now chose to Block build when upstream/downstream is executing
As for running conditional steps on failure:
- Use Post Build Tasks as Paul suggested, or
- Configure logic using Conditional Build steps
I'm working on a team that is building a RESTful HTTP service. We're having trouble with setting up a Jenkins CI job which will build the service, run it in the background, execute some tests, and then terminate the servers.
Specifics
The server is built in Node.js using the hapi framework and has some unit tests written in mocha.
The tests are written in Java using Maven. (Why not node.js-based tests? Because our testing dept. has invested time in creating a Java-based REST-testing framework.)
The build should fail if the node-based unit tests fail or if the java tests fail.
Our Jenkins box is run by a support team elsewhere in the company; our builds execute on a Linux slave.
Current Attempt
We've got something that kind-of works right now, but it's unreliable. We use 3 build steps:
The first build step is an Execute Shell step with the following commands:
npm install
npm test
node server.js ./test-config.json &
Second we do a Invoke Maven 3 step that points to the test pom.xml.
And third we run Invoke Standalone Sonar Analysis to do static code analysis.
This mostly works, but we depend on Jenkins' ProcessTreeKiller to stop the services once the job completes. We always get the warnings stating: Process leaked file descriptors. See
http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+buildfor
more information
Unfortunately, we've had cases where the service is terminated too soon (before the tests complete) or where the service doesn't get terminated at all (causing subsequent builds to fail because the port is already in use).
So we need something more reliable.
Failed Attempt
We tried setting up a single shell script which handled starting the service, running maven, killing the service, then outputting an exit code. But this didn't work out because the mvn command wasn't available on the command-line. Our Jenkins has multiple maven versions available (and jdks too) and I don't know where they live on the slaves or how to get at them without using the Invoke Maven 3 build step.
Ideas
We've toyed around with some ideas to solve this problem, but are hoping to get some guidance from others that may have solved similar problems with Jenkins.
Have the service self-terminate after some period of time. Problem is figuring out how long to let them run.
Add a build step to kill the services after we're done. Problem is that if the maven execution fails, subsequent steps won't run. (And if we tell maven to ignore test failures, then the build doesn't show as broken if they fail.)
Try killing any existing service process as the first and last steps of the build. Problem is that other teams also use these Jenkins slaves so we need to make sure that the service is terminated when we're done with our build.
Start and stop the node.js services via Maven doing something like this blog suggests. Problem is that we don't know if Jenkins will identify the spawned background task as a "leaked file descriptor" and kill it before we're done testing.
It would be nice if Jenkins had a "Post-build action" that let you run a clean-up script. Or if it had a "Execute background process" build step which would kill the background items at the end of the build. But I can't find anything like that.
Has anyone managed to get Jenkins to do anything remotely like this?
Some brainstorming:
You can turn off Jenkins ProcessTreeKiller, either globally or per invocation. I am not sure why that is not an option for you.
In response to #2, several options:
Post-build actions get executed regardless if build steps had failed or not. This would be a great way to trigger a "service cleanup" task that will run regardless of the build state.
You can setup any build step as post-build action, using Any Build Step plugin, or you can use Post Build Tasks plugin, the latter even gives options to define triggering criteria.
You can change the build state, based on RegEx criteria using Text-finder plugin
You can setup Conditional Build Steps. The "condition" could even be a result of some script execution