We need a cloud service to run automated tests written in Python on top of Behave BDD / Appium environment.
1 - What are some good options in the market for cloud automated tests and reporting?
2 - We used App Center with Xamarin.UITests before but I believe they support only Java Appium clients, is that correct?
We use Browserstack for Python+Appium automation, you can test on real devices or on emulators/simulators.
Currently we are on App Automate - 1 parallel test (1 user) plan. This is testing on real Android and iOS devices (either manual or automated) and there is no limit on the number of testing minutes per month. It seems like a good option for now.
Setup is pretty easy, you need to upload .apk or .ipa file to their server with REST command and response will be used for 'app' capability. For starting test you only need to provide username and access key, you use this as server URL when starting Webdriver.
They have around 40 Android devices and most of them are either Samsung or Google devices, for iOS they have all devices.
Reporting could be a bit better, although you can see text and Appium logs along with videos of recorded sessions, you will see each session on it's own in the dashboard, there is currently no option to see how whole test suite performed, so we use nose-html-reporting for test suite reports.
You can test against your dev/internal environments, just need to download their binary file and run it with your access key and you can start testing against local env.
CS replies pretty quick on queries.
Hope that helps, if you've any questions about it let me know.
P.s. I haven't tried other cloud options like Sauce labs or Perfecto so can't say much about it.
There are many options available in market for Cloud Automation namely Xamarin, Perfecto, Testdroid, Saucelabs. Most of these offer paid solutions for automation and most support Python. There is open stf (https://github.com/openstf/stf) for open source options.
Xamarin supports C#, Ruby with Cucumber as well, alongwith Java with appium
Related
While running automated test using appium I used driver.getPageSource() and stored it as a XML file. I also took the screenshot of the same page using driver.takeScreenshot() and saved it as a PNG file. Now how do I view it in Appium app so that it correspondingly maps to the elements with the xml data.
Is there a open option in Appium like in uiautomatorviewer?
I couldn't use uiautomator because it requires uix file. But I couldn't find open option in appium.
UIAutomatorviewer is simple tool to start with mobile automation , however it has some limitations
Less robust:
Throws random errors while working with native apps
Limited functionality:
Limited features as compared to APPIUM Desktop
These limitations do not allow UIAUTOMATORVIEWER to be a universal choice among the automation engineers , and mostly it’s used along with Appium Desktop.
For Installation and Usage you can refer below URL
https://automationlab0000.wordpress.com/2018/12/31/appium-desktop-for-locating-elements/
Due to the complex setup of Appium, it is quite difficult to be dependant only on Appium for mobile automation.
With Appium Studio of SeeTest you can easily identify objects.
Appium Studio mitigates all of Appium's challenges with ease. There is also an object spy for easier object and XPath identification. Tests are recorded simply. Actions carried out on the device reflection are recorded in an easily exported and executable test written in the language of your choice.
Visit this blog for more details.
Refer their docs to get started https://docs.experitest.com/display/public/TD/Appium+Studio
I have used Appium in Mobile automation with IOS and Android. I wanted to know about advantages that offers Calabash and if there are common point in generated scripts of both tools?
I have followed this link : Appium VS Calabash
I have hands on in mobile automation quite a long time. let me tell my point of view on both tools.
both also have advantage and disadvantage find below:
I have used calabash and appium both for automation. Appium has more advantages than calabash.
Advantages of Appium over Calabash:
1. You can write your code in multiple language like Java,Python, C#, php etc. However, Calabash restrict to use RUBY only.
Appium comes with many predefined methods such as OpenNotification(), StartActivity(), InstallApp etc. Calabash does not provide such predefined methods.
Appium set up is very easy as compared to Calabash.
In calabash, it installs the app on the device every time you run the test. However, In appium you have a feature where in you don't need to install the app every time. It works with previously installed app on the device.
Appium works with both native and hybrid app.
There are many advantages of appium. I hope this helps.
Try them both for yourself.
I found calabash easier to get started. I had a complete user flow test working from setup to working test in Calabash in half the time it took for Appium.
I don't like using xpath. It makes tests brittle and subject to breakage if something in the layout changes. Appium requires mainly the use of xpath. Calabash allows you to drive actions and interactions through the selection of elements by their text . I prefer this because that same text is normally in the designs and/or requirements.
Some comments in other posts are not accurate. Calabash does not require the install of the app every time unless you don't look beyond the surface. There is a simple means to enable/disable it. I have logic that runs either way based on passing an argument. Calabash has more than enough existing methods to interact with your app. It has a built in debug that allows you to tinker with element interaction and those same calls can be copied and reused in your test code. Using it with getgauge.io provides reporting akin to the built in Cucumber but is easier to run multi-threaded and easier to write your feature/spec files.
Be fair to both tools and yourself. Create a simple project in each and try for yourself. Don't make a decision solely based on someone else's opinion.
There is no common part in the automation scripts since Calabash uses query for interacting elements but Appium uses selenium find functions.
The best part of the Calabash is that you can use cross-platform frameworks for automating ios and android application with same feature file and most part of automation code. However the worst part is that Calabash is Ruby only.
Look at this post for more details about the differences of Calabash and Appium.
Since you have already read the differnces , Few things to note that Calabash can also be used with Xamarin which is also a good tool to test hybrid apps and you can run the same script for ios and android apps with some minor changes.Also it has been now integrated into visual studio,so microsoft support is inevitable and some users might feel comfort factor.Lambda expressions are also a plus.A major setback with the Calabash however that it only uses ruby.
Appium on the other hand can support various languages which is plus and you can use webdriver as well.However it struggles with hybrid apps but you can overcome that functionality by using google chrome mobile view to find coordinates to click.one thing for sure is that sause labs have many bugs to fix in Appium yet.
I have a very frequent test scenario that requires a new Google account every run.
Is there anything that can automate Google account creation like a script or utility or perhaps even an Android app of some sort?
Thanks.
You can try with some automation tools to automate browsers and create new accounts. Some tools may be helpful:
Microsoft Excel (Automate browser using VBA)
https://www.autoitscript.com/site/autoit/downloads/
https://www.macrorecorder.com/
https://www.jitbit.com/macro-recorder/
A lot more tools are available on the web. Test few tools to find the one that fits your needs.
I am new to automation and trying to use Appium(Mac app) for writing test scripts on iOS app.
I have managed to load my app successfully on Appium and also used Appium Inspector.
There is a Record button which records my actions in Appium inspector.
However, my question is how do I save these recorded scripts for reuse?
Secondly, I also want to manually write test scripts using Java, but currently don't know where to write code using Appium Mac app.
I have read the starter guide on Appium website and seen some videos but cannot find any app usage related information.
Any help(links, tutorial video) will be appreciated.
P.S. I do not use Selenium.
My suggestion is to not use the record feature in Appium, it's not stable and hard to maintain.
You can write your java-automation-code on Eclipse/intellij(and all java IDE)
You have a lot of information on internet, you can start here:
Appium - Git - Sample - java
Try to make it run, and take it from there.
You always can ask at discuss.appium.io and see examples.
Good luck, hope it's helps a little.
You can use any programming language supported by Appium where you feel better.
Setup environment and create test scripts. you can learn from:
Appium
Jave Sample
You can write your script in the same way as you used to right in Selenium means you can choose any IDE (in my case I'm working on Eclipse Juno) and then just pass all the capabilities if you will use TestNG framework then run as TestNG.
For some of my iOS application projects, I would like my CI server to be able to report the following properties:
startup time
frame rate
both as a graph over time, and with "low water marks" so the build fails if the measured results aren't within certain criteria. I've already found some of the things I need.
The CI server will be Jenkins.
I can use Transporter Chief to get the built app onto an iPad.
To measure the startup time I could find the duration between launching main() and leaving application:didFinishLaunchingWithOptions:.
To measure frame rate I can put a CADisplayLink into the app and sample its duration property.
If those tests output JMeter XML, then Jenkins can display the output via the Performance plugin.
What I haven't worked out is, how should I embed those tests into my app and launch it on the iPad? As described above I can deploy the app to the iPad, but then I don't know how I would launch it to gather the results of the tests. My unit tests are running on the simulator - I don't want to run the performance tests there obviously :-).
I imagine that there's a solution involving jailbreaking the iPad and controlling the app over SSH, I'd prefer not to go down that route if it's possible. If you have done that and can explain how it works, I'd still like to hear about it.
I'm also using fruitstrap to install apps on the device in CI. In terms of booting the app, I know of two ways:
Use fruitstrap with the debugger attached
I know teams that have done this to run KIF integration tests on devices in CI. I have played around with fruitstrap to get it booting apps on the device, but haven't myself taken the extra step of automating the whole thing
shameless plug for my post on fruitstrap: http://www.stewgleadow.com/blog/2011/11/05/installing-ios-apps-on-the-device-from-the-command-line/
Use the instruments command line tool with UIAutomation
I know the instruments tool can boot apps on the device automatically in CI (I wish it also installed them, but we have fruitstrap for that until Apple fixes it). So you could write a really simple little UIAutomation test that gave your app enough time to do its performance analysis.
Jonathan Penn has a nice little demo project for UIAutomation and build script that could be integrated with an 'install' step using fruistrap to try it out on the device
In both cases, I uses a little wrapper around libusb to give me the device ID of attached devices, so the more devices I plug into my CI machine, the more devices it runs tests on, https://github.com/sgleadow/iphone_detect
Can you launch the app on the device using lldb?
If so, it may also be able to capture the log output.