I have a JavaScript-heavy app being built via GRUNT. To test this app, I have tests written with Jasmine. The tests are being run via Karma and Protractor. I would like to show code coverage of these tests in the command-line. My question is, how do I do this? I can't figure out how to integrate code coverage details within my build process. Ideally, I would love to see code coverage of my unit tests and code covereage of my integration tests.
Thank you.
I don't believe code-coverage is possible at this time with Protractor. Karma's code-coverage is made available via istanbul. It appears one would have to write a custom reporter. I agree this would be a nice feature. I may take some time in the near future to see if I can crank this out. If I make any progress, I will post here again.
Related
In our (grails 2.x) projects, we often have plugins, that come with their own unit tests, but are accompanied by testapps, which include the plugin inline, and run functional and integration tests.
To run the tests, we first run the unit tests on the plugin project, and then run the integration and functional tests on the test application.
I would like to start using code coverage metrics, but I need my data to be representative of all the tests that are running, not just unit tests that come embedded in the plugin source tree.
I have not yet chosen a particular code coverage solution, so this is a free variable that could help solve the issue.
Any help or insight appreciated
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
I'm configuring continuous integration with TFS 2012. I have one problem need solve.
I need to exclude some paths from triggering builds.
For e.g. I have:
$/Project1
$/Project2
And I want that after each check-in of $/Project1 - build has been triggered. And it must build both $/Project1 and $/Project2.
But after checking in $/Project2 I don't want to trigger a build for that Build Definition.
In Source Settings of Build Definition are only functions "Active" and "Cloaked", but it isn't what I need.
Thanks a lot in advance.
P.S. The worth solution is to add the comment ***NO_CI*** on check-in. It will be great if there is some other way.
Based on the comments, this boils down to an X-Y problem. You can't do what you want to do, but the reason you want to do it is because you're trying to solve the wrong problem.
You're running the UI tests at the wrong point in your dev-test cycle. UI tests should not run during a build, they should run after a release. A change to your test project should absolutely result in a build.
Someone is developing UI tests against code that's not yet in source control, which makes no sense. If someone is writing tests against code, the code should be source controlled.
I'm guessing that someone is manually pushing uncommitted code out to a dev server, which is being used by someone else to write tests. Don't do this. Use a real release management solution so that as developers write code, each check-in is automatically deployed to a dev/QA environment. Then the folks writing the UI tests will have something to test against. What's the point in writing tests against code that's in such a state of flux that the developer responsible for it isn't even sure it's worth being source controlled? That just results in spending a lot of time rewriting tests as the code evolves.
Assuming you set everything up properly, every commit of the application code will result in the current set of tests being run against the latest commit. Every commit of the test code will result in the new set of tests being run against the existing application code. The two things (application code and test code) should be coupled, and should always build together.
And one last thing, mostly opinion: UI tests are awful and serve very little utility. They are brittle, slow, and hard to maintain. I have never seen a comprehensive UI test suite actually provide value. UI tests are best served as a small set of post-release smoke tests. Business logic should be primarily unit tested, with a smaller suite of integration tests to back it up.
I have an full fledged android application and its source code but there is no JUnit or any other automated test cases are available for the app. How can I do code coverage test for it?
Is Jacoco any plugin like Gradle can be helpful in this scenario?
Thanks in adavance.
Our Java Test Coverage Tool can collect code coverage statistics regardless of how the application is exercised. It is not dependent on any specific unit test framework (although it can work any, including none).
If you run it with tests, you get "code coverage from tests". If you simply exercise the application manually, you get "what code does this manual operation exercise?" If you do a thorough manual exercising of the code, you get the equivalent of "what code does all of my manual testing" exercise.
`No valid coverage data available
Tracking and trending code coverage works best when like is compared with like. In this regard it is best to only track builds when all unit tests are passing.
This plugin will not report code coverage until there is at least one stable build.`
what should i do to have a rails stats report
#krs - From what you've said in your comments, you are correct.
rails stats only provides code coverage metrics for tests at that point in time. rails stats does not intend to keep data between builds and report test results vs. code converage.
You can connect your CI to an external service to see test coverage vs. test results. (EG, https://coveralls.io).
Please add coverage.xml file outside the project.It works fine for me.