is there an api for retrieving the code coverage of the changed lines of code in a pull request?
looking at the docs for the azure devops api, pull requests does not include the code coverage for the changed lines of code, but then I wouldn't expect it to, because that's not part of git, is it?
the code coverage api doesn't seem to include anything about the coverage of changed lines of code.
any pointers much appreciated
Yes you can see code coverage only for changed files directly in Azure DevOps pull requests, see documentation here https://learn.microsoft.com/en-us/azure/devops/pipelines/test/codecoverage-for-pullrequests?view=azure-devops
Related
We are investigating whether we can incorporate Postman Test Runner and Newman into an API testing flow with our Jenkins CI server.
My question is this: once I import a swagger/Open API file into a Postman collection, how can I keep changes from multiple team members in sync?
For example, if a team member adds a new api endpoint to the swagger file, do we have to re-import the swagger file into a postman collection overwriting it? We'd like to keep using the swagger file as the single source of truth so would like to keep that in sync with the postman collection. Ideally we would update the swagger file, commit it to git and get its changes synced to postman.
What about tests created in postman? Is there a way to keep that checked into git? Would we have to export the collection after each test change and check that into git, and re-import changes to postman collections after each git pull?
It looks like since some of the online postman features are built for sharing - the idea is that you would make a change to the collection directly in the postman client and that gets shared out to other postman clients? If so, is there a hook that can be added to sync those changes to git automatically?
To answer your questions in order:
Once I import a swagger/open api file into a postman collection, how
can I keep changes from multiple team members in sync?
To keep your collections in sync everyone will need to sign in and use a team workspace. As of Postman 6.2 a single team workspace is now free.
https://www.getpostman.com/docs/v6/postman/workspaces/intro_to_workspaces
For example, if a team member adds a new api endpoint to the swagger file, do we have to re-import the swagger file into a postman collection overwriting it?
Depends on how you are generating the file. If it is being generated using a run-time tool (e.g., NSwag, Swashbuckle) then you'll most likely end up needing to overwrite the file. If you have a swagger.json your team is directly maintaining, you can probably modify the scripts Postman provides to keep your definitions in sync with Postman: http://blog.getpostman.com/2018/03/02/sync-your-specs/
What about tests created in postman? Is there a way to keep that checked into git?
Yes. You can export Postman collections which include your tests and check those into Git.
If so, is there a hook that can be added to sync those changes to git automatically?
Answered on SO here.
We are using Jenkins in our build cycle to run our specflow tests. We are trying to find a better way to surface the test results. The generated html report file is great, but right now we have to manually log into the build server to check it out. I want to surface that file via a clickable link.
Here is what I am talking about specifically. Here is the test results via our Jenkins console
I want to change the highlighted text to an http url so that everyone can just click that and view the file.
I already have the http link working, so that's not the issue. I just need to figure out how to make specrun output this link here.
Any ideas on if this is possible? And if so how to do it?
Thanks for the help!
Sorry, this is currently not possible in SpecFlow+Runner, but I will put it on the backlog as a feature request.
Perhaps you can do something in Jenkins to parse the output and reformat the link.
Full disclosure: I am on of the developers of the SpecFlow+Runner.
This is a broad question, so any answers are deeply appreciated. I need to continually log the size of several build files (in this case some CSS and JS files), preserve this log and ideally show it as a dashboard in Jenkins.
I know that I can setup a cron job and execute a bash script to grab the files and log their size, but I'm not sure where this file would live and how to display it. Ideally the result would be a dashboard plot or bar graph over time.
Thanks.
P.S. I'm open to other logging suggestions, but Jenkins seems like the appropriate system to do this in.
Update: this isn't perfect but it works. Google Spreadsheets has a simple API for posting data, so this can work as an endpoint for any script you want to write that logs your data.
It's not a Jenkins solution, but gets the job done.
In my search leading up to this, I did come across JMeter, and the Performance Plugin for Jenkins, which were contenders for a possible solution.
I'm trying to get coverage results from Jenkins using the REST API and the Cobertura plugin.
It looks like I can get the top level information about coverage results since the Cobertura plugin puts the bottom line results in the healthReport of the job.
However, I would like more detailed information about code coverage.
So I tried:
<jenkins url>/job/<job>/<buildNo>/cobertura/api/json
I get just empty results. I assume from this the Cobertura plugin just didn't implement the API for returning this information.
Is there another way to get this information programatically other than modifying the Cobertura plugin?
If you append "?depth=2" to your query, you'll get additional percentage info. Higher values for depth show the individual file coverage.
Turns out there is a jenkins plugin defect open on this. But the answer above is correct-- you can get most of the information by adding "depth" (coverage ratios).
However, unfortunately this still does not give all the information you get via html, namely the number of lines scanned in each module, it just gives you pct. This is much better than nothing, but still incomplete. I've annotated the defect to reflect this...
My company is developing a web application that builds in ant. I've been tasked with getting CruiseControl.net to differentiate between a build failure and a unit test failure, which it can't do natively. ( It currently lumps both together but doesn't help developers understand what's broken )
I have CC.net call a script that returns specific exit codes depending on the nature of an ant task failure. I'd like these exit codes to be reflected in the CC.net failure report / dashboard but am having some trouble finding resources on how this might be done.
Any suggestions?
Not directly. All the reports and display works from information in the logs which are XML files. The display and reports work by applying XSLT to these XML files.
Take a look at your build logs and unit test logs, to see if each of those process write the failure information to their respective log files.
If they do, you should be able to write a custom XSLT or modify the existing XSLT to display that information.
Edit:
A different approach based on your comment. You could probably redirect the ANT error code to a file. Then you could have a seperate ccnet task that takes the error code from that file and re-format and display it (depending on how/where you want it displayed)