I am new in Karate but know almost all the basics. I want to generate a report for my test cases which will be kept in the folder for history purposes. currently, I am using the HTML report which overrides or updates every time I run my test cases.
If you can recommend me a step-by-step tutorial or a link, or simply give me a hint on how to approach this.
I have tried to use the allure framework and I am struggling to set it up.
Thought it would be the best approach to solve my problem.
Attached is my HTML report I am talking about. Thanks in advance
The JUnit HTML report which you have referred to as an image is designed for easy development, so when working on a test, you re-run and re-fresh the browser.
What we recommend is to use the parallel runner reports. Most of the time, 3rd party frameworks will solve this problem for you - especially if you use them integrated with CI. See this for example.
But finally, Karate will output the industry-standard Cucumber JSON report format, so you can always write a reporting solution yourself. Note that others have had success integrating Allure - so please refer this answer: https://stackoverflow.com/a/54527955/143475
Related
I'm trying to use SpecFlow for a .net project. I'm new to SpecFlow. The Development team are using NUnit, so it would seem that SpecFlow would be a good option in conjunction with Cucumber. However, the Development team have come back say that SpecFlow cannot be used saying they do not have an API/Service that is available to use at the level required. Currently all of their automated tests are through the UI using Test Complete, I am keen to move to API level testing.
Can anyone explain to me why SpecFlow cannot be used, I'm sorry it's a newbie question but no one can answer it, I've asked everyone I can think of, surely the first steps would be to see if we can use SpecFlow with NUnit but perhaps not.
Can anyone give me a guide on my next steps, how can I be sure this isn't an option without righting it off without concern that it's just being blocked?
Thank you
SpecFlow has a unit test generator that generates unit test code for a variety of unit test frameworks. SpecFlow generates NUnit tests in its default configuration. The getting started page on specflow.org explains a quick way to get up and running with SpecFlow and NUnit, http://www.specflow.org/getting-started/.
If the UI is HTTP based, SpecFlow can be used with WebDriver or another browser automation framework to test the UI. This blog post provides an overview of how to get started with SpecFlow, NUnit and WebDriver, http://blogs.lessthandot.com/index.php/enterprisedev/application-lifecycle-management/using-specflow-to/
I am unclear on the API you want to test. If you could provide more information on the specific API and UI you are trying to test, I could possibly provide some code examples or references for you.
Is the API exposed through HTTP?
Is the UI a web, mobile, or desktop
application?
Have you tried to use SpecFlow at all?
SpecFlow doesn't run tests. It simply maps readable language to tests. If their test can be written as a nunit test, then SpecFlow is available to you to use. With no change, here is how it would look.
Scenario: Running 'testname'
Then I execute the test 'TestName'
You would map that to
[Then(#"I execute the test '(.*)'")]
public void ExecuteSpecificTest(string testname)
{
// Using reflection, execute the method listed
}
Obviously you would want to do better than that. You want a given, when, then so you clearly show the setup, action, and then compare expected verses actual result but it isn't necessary. Best practices however is another discussion.
To sum it up, code is code and SpecFlow simply maps to code. You can use WatiN, WebDriver, or anything else to hook into the UI or an API. SpecFlow doesn't care. It simple executes the methods without knowing what's inside.
I need to know if I can do Continuous Integration with XCode server. In example: set run the tests every night or when someone commit changes, and more..
I am trying to decide one iOS Ui Automation tool to integrate with my Xcode server
Thanks
There are a few problems here:
UIAutomation has no built in support in Xcode Server. I've filed bugs, I've chased down people at WWDC. Most I've ever gotten on this problem is basically "shrug." I'm not sure UIAutomation is a priority for Apple right now. So you're not going to get any official support.
As was noted, you might be able to use a trigger. The trigger won't be able to add anything to the Xcode Server report, besides possibly the error logging. But you're not going to get anything added to the nice report table.
Running on actual devices has traditionally been a problem (if you care about that.) The loading the app part has been a problem for us, but Xcode Server might be able to preload the app for you. In addition, it seems like this might have changed in the iOS 8 SDK.
There is just a lot of uncertainty around this sort of workflow. I'm hoping Apple eventually makes an announcement or adds a new tool, but the best answer I've gotten is if you want to go down this path, use UI Unit Tests. That's a shame because it requires knowledge of Obj-C or Swift, and means interacting with the app at an API level instead of an abstract level, but if you're looking for the direction Apple wants to see people go, that's it.
Edit 7/4/2015: As of WWDC 2015, there is a new UI Testing component as part of Xcode 7 that, in my experience, seems totally supported, and is promising Xcode Server support. I would very strongly recommend using that, and not using the Instruments UIAutomation tool.
With Xcode6 right around the corner they are adding some features to XCode Server specifically it looks like "Triggers" will be helpful for running iOS UIAutomation tools. Since you can run UI automation scripts from cmd line it should be possible to utilize triggers to run your scripts post builds. This along side the logic for when a bot should run will let you decide if it should be nightly or on every commit.
https://developer.apple.com/library/prerelease/ios/documentation/DeveloperTools/Conceptual/WhatsNewXcode/Articles/xcode_6_0.html#//apple_ref/doc/uid/TP40014509-SW1
I wrote a framework around UIAutomation called Illuminator to handle tasks like nightly test runs, pull request tests, and other automated conveniences.
It provides a flexible and extensible command line that can execute any particular subset of tests that you'd like, and produces reports in JUnit format (used by Jenkins).
I'm setting up automated regression testing for an FPGA project, almost exactly as described here:
Continuous integration of complex reconfigurable systems
Now I want to get test results (from VHDL REPORT statements in ModelSim simulation) to appear in Jenkins testing reports. My understanding is that Jenkins only natively supports jUnit format, and I looked for plugins supporting non-XML formats but didn't see any.
Generating valid XML from VHDL REPORT statements would be very difficult, since the simulation may immediately terminate depending on the severity. Which means that the closing tags would have to be duplicated in every single possible exit path for every single test -- not the most maintainable approach.
So, do you know of any straightforward way to convert plain text into jUnit (or another format, if supported by Jenkins)? If something doesn't already exist, is there an advantage to writing a Jenkins plugin vs just throwing together a perl script? Any other suggestions?
You should take a look at the XUnit Plugin. The Plugin reads test results from a number of tools, and seems adaptable to custom formats. From the documentation the plugin is able to read not only xml, but also csv and txt. For custom format you need to specify some style sheet for the transformation, I am not quite sure if this will go all the way for you. But even if it does not, I suppose the plugin should be easy to extend for your own format.
Old post but today there is a unit testing framework for VHDL that we've developed. It solves the problem by generating a report on the JUnit format. It also handles the case when the simulation stops due to a severe error. The tool is free and open source and can be found at https://github.com/LarsAsplund/vunit
Has anyone written (or know about) a .net unit-test runner, preference to NUnit, that runs on iOS ? or do I have to roll my own ?
My goal is to execute the unit tests on the simulator or devices. So far I've seen quite a few posts / blogs on mocking monotouch.dll (but running the tests on a PC) and one answer stating the lack of such tool.
As promised here's my own solution for my problem :-) I hope it can help other people too!
EDIT
Sounds like you'd have to write a MonoDevelop add-in that serves up an API for remote NUnit runners to send data to. Not really a trivial task.
The original Touch.Unit was updated to include support for network logging (albeit not inside MonoDevelop).
EDIT #2 : A similar runner now exists for Mono for Android.
FINAL EDIT: Touch.Unit is now an integreal part of MonoTouch releases (starting with version 5.2) and does not have to be downloaded seperately.
I've wanted the same thing myself for quite some time. I think building one is the only option...
Can't imagine porting this GUI to iOS:
I think the only reasonable solution might be to show a simple UITableView on the device/simulator with test name and red/green checkmarks, and post the full NUnit results via http to server software somewhere.
Sounds like you'd have to write a MonoDevelop add-in that serves up an API for remote NUnit runners to send data to. Not really a trivial task.
I'm looking into setting up a CI environment for our flex projects. I have very little experience in setting up an environment like this, but have read a lot about it and think we could benefit a lot from this in our projects. I do have experience with ANT and we're currently using it for our building. I've been looking at Hudson for a while and it looks really nice and simple while still having the power to support a proper CI environment.
So basically, my question is if anyone has experience in setting up Flex projects with Hudson? If so, please do share some info on issues, cost/benefit as well what kind of effort is required per project to get up and running with Hudson. I've googled for a while and can proudly say that I know more about both the Fast lexical analyzer and the Hudson River, but little more about the topic of this post =)
Just about anything that can be executed from a command line can be executed via hudson. If your flex app can be built via ant from the command line; it will work just fine in hudson.
This might be helpful:
http://www.subotnik.com/blog/?p=100