Does anyobdy know a good solid CI service that provides the common features of build parallelization BUT also support for Junit reports?
The current ones that we have looked at (semaphoreapp, circleCI, travisCI,...) are good but relatively useless as we have to manually investigate what tests failed, since when, and how often, thus negating a lot of the benefits of a hosted service.
Things that we're looking to know (and are all provided by JUnit / Jenkins):
If the build failed, because of what test cases?
Total Number of Failures / Total Number of Tests (trends to better analyze things)
Individual Track record of any test (so we know exactly when it was broken, whether it's intermittent,...)
You mentioned the most famous CI services but there are alternatives where you can get a higher customization level, like installing plugins, fine configuration, etc.
CloudBees and ClinkerHQ are both based on Jenkins offered as a service. You can also get very useful metrics (coverage, failures, graphs, execution times, etc.) thanks to Jenkins Plugins and SonarQube. I think Jenkins and SonarQube are a perfect couple for you.
Notifications are very important too. You want to be notified when something is wrong. This feature is available on both.
Regards,
Antonio.
DISCLAIMER: I'm deeply involved in ClinkerHQ
Related
This is not a programming question, but I don't know any more active forum and besides programmers are the best people to be able to answer my question.
I am trying to understand the rationale behind continuous integration. On one hand, I understand that it is a good practice to daily commit your code before heading to home whether or not the coding and testing is complete or not and then there is continuous integration concept where the minute something is committed, it triggers a build and all the test cases are run. Aren't the two things contradictory?. If we commit daily whatever coding is done, it will cause daily failed builds..Why don't we manually trigger builds once the coding and testing is complete?.
Usually when you save your code daily is to be sure that your work will not be lost.
On the counterpart the CI or Continuous Integration is to test if what you produced is ok, in the majority of projects the CI isn't applied to individual branches ie: feature, bugfix, it's applied on major branches ie: master, develop, releases, etc. And these branches aren't updated daily as they need a pull request to be update and someone to approval that pull request.
The use case for having CI implemented on individual branches (feature, bugfix) is to check before merging a pull request into a major branch when it will check the tests and if the code builds.
So resuming, yes you need to commit your code daily, but you don't need to apply CI to it daily.
I suggest to you check the Gitflow workflow: https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow
The answer is obvious.
1. Committing Code: In general the code is committed only after testing with the environment locally.
Consider Developer_A working on Component_A hence one has to commit with minimum verification as the scope is to develop Component_A.
No imagine complex system with 50 developer developing Component_B...Component_Z++
If someone is committing the code without minimum test it is most probably going to give you failed result.
Or else developer might have it committed on development branch that all together depends on SCM strategy adapted in project.
2. Continues Integration test scope:
On the other hand integrator principally collects and synergies different codes (Software Components) together into 1 container and perform different tests.
Most importantly, integrator need to ensure that all the Components Developed from different developers is fitting good and at the end Software is working as expected. To ensure that, Integrator have acceptance criteria and to proactively prevent something which can go wrong, it is important to have these criteria automated with the help of Continues integration.
But among all factors, it is important to give feedback on the quality of software to the developers. It is best in favor of project (economically), to know about the bug earlier hence Continues Integration and DevOps.
In Complex System it is worth to have automated watcher to catch the sneaked mistakes from developers.
3 Tools and Automation:
To create human independent system, automation tools like Jenkins are helpful.
Based on the testing strategy different testing levels can be performed with the help of Automation tools.
This is sort of an open-ended question/request (hope that's allowed).
On my team we are using Karate API testing for our project, which we love. The tests are easy to write and fairly understandable to people without coding backgrounds. The biggest problem we're facing is that these API tests have some inherent degree of flakiness (since the code we're testing makes calls to other systems). When running the tests locally on my machine, it's easy to see where the test failed. However, we're also using a Jenkins pipeline, and when the tests fail in Jenkins it's difficult to see why/how they failed. By default we get a message like this:
com.company.api.OurKarateTests > [crossdock] Find Crossdock Location.[1:7] LPN is invalid FAILED
com.intuit.karate.exception.KarateException
Basically all this tells us is the file name and starting line of the scenario that failed. We do have our pipeline set up so that we can pass in a debug flag and get more information. There are two problems with this however; one is that you have to remember to put in this flag in every commit you want to see info on; the other is that we go from having not enough information to too much (reading through a 24MB file of the whole build).
What I'm looking for is suggestions on how to improve this process, preferably without making changes to the Jenkins pipeline (another team manages this, and it will likely take a long time). Though if changing the pipeline is the only way to do this, I'd like to know that. I'm willing to "think outside the box" and entertain unorthodox solutions (like, posting to a slack integration).
We're currently on Karate version 0.9.3, but I will probably plan to upgrade to 0.9.5 as part of this effort. I've read a bit about the changes. Would the "ExecutionHook" thing be a good way to do this? I will be experimenting with this on my own a bit.
Have other teams/devs faced this issue? What were your solutions? Again we really love Karate, just struggling with the integration of it to Jenkins.
Aren't you using the Cucumber Reporting library as described here: https://github.com/intuit/karate/tree/master/karate-demo#example-report
If you do - you will get an HTML report with all traffic (and anything you print) in-line with the test-steps, and of-course error traces, and most teams find this sufficient for build troubleshooting, there is no need to dig through logs.
Do try upgrade as well, because we keep trying to improve the usefulness of the logs, and you may see improvements if you had failed in a JS block or karate-config.js.
Else, yes the ExecutionHook would be a good thing to explore - but I would be really surprised if the HTML report does not give you what you need.
I am trying to track the page speed of certain urls of my project on each merging of the pull requests in Github and output the results of report in HTML format or JSON file. On the CI side, I am going to use Jenkins. I have no prior knowledge on performance testing. I want to know about the best approach to automate the speed test, integrate it with Jenkins and output the result.
On researching over the internet, I noted few possibilities which could be done to achieve this goal.
Installing "Page Speed Insights (psi) node package", creating the script that uses the psi for fetching the speed of certain pages, generating the test reports for use with Jenkins. (Referred to this link by Oxagile)
Performance testing using Jmeter and integrating with Jenkins.
Performance analysis using LightHouse. (Referred to this link by Timo Stollenwerk)
Choosing the right approach is very important. Therefore, I would be very grateful if anyone can suggest me different approaches and thus the right one to use(with examples if possible)in my case to achieve this goal.
Thank you in advance.
After quite a bit of research, I found out that sitespeed.io is the best solution for achieving this goal. It is a complete web performance tool that helps us to measure the performance of the website. It is best for running in the continuous integration to find web performance regressions on commits and monitoring them in production and alerting on regressions.
At my workplace, I've been tasked to look into some metrics that the Jenkins tool provides and somehow pull them programatically and display them in some presentable format. The metrics that I need to pull are:
How many unit tests are passing? Failing? Skipping? The total % of passing?
How many integration tests are passing? Failing? Skipping? The total % of passing?
How many acceptance tests are passing? Failing? Skipping? The total % of passing?
How long does it take to execute the test? Make the build?
What is the number of tests executing in pipelines?
... the list goes on
Now I have a very small 1000 ft understanding of Jenkins, and an even smaller understanding of the steps that I need to take to make this program come to life. I am an intern with not much programming experience either, but after some research, I learned that I can navigate through the Jenkins API by adding '.../api' to the link that I want to find API elements for, and I know that I'm going to need to develop a plugin. Aside from that I don't have much direction at all. I don't know what environment I need to develop these plugins (Maven? Never heard of it)... I don't know what languages are supported (I only know C++, Java, and JS)... I don't know how to even install a plugin or get to the plugin on the Jenkins site. I feel like I'm drinking from a firehose with this task and need some guidance.
Does anyone have good guides, advice, tips, tricks, videos... anything that might help me get started on Jenkins plugin development? Any insight into how I might solve this problem too would be much appreciated.
Thanks so much.
First of all there are tools out there which generates HTML reports. You can start there.
For example: MSTest report (.trx) can be converted to HTML by TRXER
and can be published using the HTML Publisher Plugin
However if you're into building your own plugin use NetBeans (I have tried it; and it works)
But creating Jenkins graphs you have to google and see.
I am looking for a comparison between IBM Build Forge (Rational) and Hudson CI.
At work we have full licenses for BuildForge but recently we started using Hudson for doing continuous integration and automating other tasks.
I used BuildForge very little and I would like to see if there are any special advantages of BuildForge over Hudson.
Also it would be very helpful to see a list of specific advantages of Hudson over BuildForge.
I not sure if it important or not, but I found interesting that Build Forge is not listed under continuous integration tools at wikipedia.
Thanks for bringing attention to the fact it was not on the wikipedia list of continuous integration applications. I have now added it. Build Forge has been a leader in providing continuous integration capabilities by use of it's SCM adapters for many, many years. Build Forge has a strength in supporting many platforms through its use of agents. These agents can run on Windows, Linux, AIX, Solaris, System Z, and many more -- they even give you the source code for the agents for free so you can compile it on just about any platform. The interface allows you to easily automate tasks that run sequentially or in parallel on one or multiple boxes. Selectors allow you to select a specific build server by host name or by criteria such as "any windows machine with 2gb of ram" from a pool of available agents. The entire process is fully auditable, utilizes role based permissions, and is stored in a central enterprise database such as DB2, Oracle, SQL Server, and others.
One of the most compelling reasons to use Build Forge is it's Rational Automation Framework for WebSphere. It allows a full integration into WebSphere environments to automate deployments and configurations of WebSphere through out of the box libraries. The full installation, patching, deployment of apps, and configuration of WAS and Portal can be performed using these libraries. To find out more, it is best to contact your IBM Rational representative.
You can use RAFW (IBM Rational Automation Framework for WebSphere) with BuildForge. It does not make sense to use RAFW with other ci servers, since RAFW requires BuildForge.
You have support for BuildForge and it integrates with other IBM software like ClearCase. Theoretically you have only to deal with one vendor if something in the chain does not work, but IBM has different support teams for their products and you might become their ping pong ball. :(
Hudson is open source (if you like that), that means you can get the source and modify it to serve you better. But the release cycle is very short (about 1 week, agile development). There is a more stable version with support available now (for cash of course) from the company of the main author of Hudson.
Hudson is currently main stream and is actively developed. I don't know how the usability of BuildForge is, but Hudson is good (not always perfect). The plugin concept of Hudson is a great plus, not sure if BuildForge has it as well.
Currently, we are using Hudson, but BuildForge was not looked at in detail.
You need to define what you would need continuous integration for (e.g. building, testing). Having used Hudson, I can vouch for its usefulness and effectiveness. There are many plugins that extend Hudson that can suit various needs. And you can't beat the price point (free).
You need to inquire as to why a BuildForge license was obtained at your place of employment. Perhaps someone on your team knows why this was done. If it isn't necessary for your needs, don't renew your BuildForge license and simply continue using Hudson.
Being a BuildForge/RAFW user, I have to object to one point stated above. It is perfectly possible to use RAFW without BuildForge. It is driven by a command line script, and you could use for example Hudson and RAFW together just fine.
A sample command would look like:
rafw.sh -e env -c cell -t was_common_configure_start_dmgr
The primary differentiators IMO:
Hudson/Jenkins is more readily extensible with the many existing plugins. It has a large active community and plenty information and documentation.
BuildForge can be configured with agents running on multiple machines and tasks can be assigned to run on a target agent. Reliable vendor support.