I am looking for a tool to test iPhone app UIs and would prefer to have a test driver executes tests from a Windows or Linux environment, by sending commands to the test server on the phone via HTTP. Is there anything out there like that?
I have been unable to find anything that allows me to test the GUI automatically on a non-OSX environment.
There are alternatives to the Automation part of Instruments.
You might for example use FoneMonkey http://www.drdobbs.com/open-source/231901614
You write that you want to use HTTP to send commands to some sort of magic TestServer running on the devices. I think that this will be very much impossible on a non-rooted phone, and you would at the same time be unable to get the test results automatically, ( asking via. HTTP whether a button is on the screen seems... Wrong ).
What are your reasons for this setup?
Anyway, long story short, my answer is going to be no, there is nothing like what you request out there, and I suggest you look into Apples Automation Instruments which allow you to do quite a lot of nifty stuff
Good luck!
Related
I created an application that uses Watir to automate logging in and perform a couple of functions within a site.
Right now it's written 100% purely in ruby classes that I was just executing in irb, but I want to put it into a Rails application and put it online. I haven't been able to find much information about using something like Capybara or Watir for anything other than testing. Is this because of how slow they are or is it a capabilities issue?
Would I be able to run a background process that opens a browser with Watir and performs a few functions for each user in production?
Another question I have is how to keep the session over a longer period of time. There are two sites that require 2FA that my app logs into. If I wanted to log in and perform a function once an hour with a Watir browser, I could create it as a background process (if that works). But when the process is done the browser closes and when the background process runs again in an hour it requires 2FA again.
My other worry is speed. If I have 50 users that all need to run a Watir browser at the same time I imagine that will be slow. I am not worried as much about speed as long as they run and collect the data and perform the few actions we need, but how it will effect the applications integrity.
WATIR is specifically designed as a testing tool.. WATIR stands for Web Application Testing In Ruby. It's design is centered around interacting with a browser the way a user would, effectively simulating the same actions a user would take when using a site. That it would be sub-optimal for other tasks is to be expected. Due to scraping and testing having very similar activities, there a a number of people that use watir for that task, but it is not designed for that purpose, and it is unlikely that the WATIR developers would ever add features specific to data scraping verses testing.
With the things you are contemplating, you should ask yourself if you are doing the equivalent of using a socket wrench as a hammer, and if there might be a better tool you could use.
If the sites you are interacting with support an API, then that would be the preferred way to interact with them, to get information from the site. If that is not supported, you may want to consider looking at other gems that would let you request the site HTML or parse the HTML directly (e.g. Nokogiri)
You should also inspect the terms of service for the sites you are interacting with (if you don't own them) to ensure that there are not prohibitions against using 'robots' or other automated means to access the site. If so, then using Watir in the way you propose may end up getting you banned from access to the site if the pattern of your access is obviously the result of an automated process.
This is actually done more often than people think. Maybe not specifically with Watir, but running a browser automation task in a job. The jobs should be queued and run asynchronously, preferably in a different process than your main web app.
I wrote about this strategy here: https://blogstephenarifin.wordpress.com/2018/08/23/integrating-a-third-party-without-an-api-using-rails-5-and-heroku/
If you find yourself having to use Watir then the best way is to use it to render the page (for example in headless mode for javascript), save it and then use Nokogiri to process it. Apis are suggested a lot by people who don't and can't find a use for scraping but at times is necessary and perfectly legit (you may even be scraping your own data). Apis are not a universal option.
Second you should probably regulate its use to a background job. If you have end users (and you really shouldn't have many simultaneous users) many services inform customers data will be available in a few hours to a few days
I've been working on a project that I want to add automated tests. I already added some unit tests, but I'm not confident with the process that I've been using, I do not have a great experience with automated tests so I would like to ask for some advice.
The project is integrated with our web API, so it has a login process. According to the logged user the API provides a configuration file which will allow / disallow the access to some modules and permissions within the mobile application. We also have a sync process where the app will access several methods from the API to download files (PDFs, html, videos, etc) and also receive a lot of data through JSON files. The user basically doesn't have to insert data, just use the information received in the sync process.
What I did to add unit tests in this scenario so far was to simulate a logged user, then I added some fixture objects to the user and test them.
I was able to test the web service integration, I used Nocilla to return fake JSONs and assert the result. So far I was only able to test individual request, but I still don't know how should I test the sync process.
I'm having a hard time to create unit tests for my view controllers. Should I unit test just the business logic and do all the rest with tools like KIF / Calabash?
Is there an easy way to setup the fixture data and files?
Thanks!
Everybody's mileage may vary but here's what we settled on and why.
Unit tests: We use a similar strategy. Only difference is we use OHTTPStubs instead of Nocilla because we saw some more flexibility there that we needed and were happy to trade off the easier syntax of Nocilla.
Doing more complicated (non-single query) test cases quickly lost its luster because we were essentially rebuilding whole HTTP request/response flows and that wasn't very "unit". For functional tests, we did end up adopting KIF (at least for dev focused efforts, assuming you don't have a seaparte QA department) for a few reasons:
We didn't buy/need the multi-language abstraction layer afforded by
Appium.
We wanted to be able to run tests on many devices per
build server.
We wanted more whitebox testing and while
Subliminal was attractive, we didn't want to build hooks in our main
app code.
Testing view controller logic (anything that's not unit-oriented) is definitely much more useful using KIF/Calbash or something similar so that's the path I would suggest.
For bonus points, here are some other things we did. Goes to show what you could do I guess:
We have a basic proof of concept that binds KIF commands to a JSON RPC server. So you can run a test target on a device and have that device respond to HTTP requests, which will then fire off test cases or KIF commands. One of the advantage of this is that you can reuse some of the test code you wrote for single device for multiple device test cases.
Our CI server builds integration tests as a downstream build of our main build (which includes unit tests). When the build starts we use XCTool to precompile tests, and then we have some scripts that starts recording a quicktime screen recording, runs the KIF tests, exports the result, and then archive it on our CI server so we can see a live test run along with test logs.
Not really part of this answer but happy to share that if you ping me.
I am currently evaluating how to test a rather big and complex web application, based on Rails 4 on the server side and EmberJS on the client side. In our app, the client exclusively communicates through a restful JSON API with the server.
We did a lot of unit testing based on Konacha so far and are now willing to setup integration/acceptance tests too. We are not sure whether we should start writing end-to-end tests, so tests including a running instance of our server, or whether we should go for integration testing the API and the client side separately.
Our preferred choice at the moment is end-to-end testing, because we fear that in the case of integration testing API and client separately we have twice the effort of creating and maintaining tests and that there might be tiny, little specialities of the comunication between API and client which we could not catch.
Well, we like modern & fast testing frameworks like Konacha, so we don't really want to use Selenium. Not only because it feels a little bit old, but also because its performance is quite poor. Still you will need to control instantiation of mock data on the server and reset of the server, why we came up with the following approach:
We implemented a Testing API which conceptually is used to control the state of the server, e.g. it has the following methods:
GET /api/test/setup # Simple bootstrapping of the database, e.g. populate table with ISO language codes etc...
GET /api/test/reset # Reset the database, using `database_cleaner` gem
A konacha test case would then call setup and reset, before and after each test case respectively.
What do you think about this approach?
Not sure what I would call the test of the API and the client separately, but even if you think about running this kind of test, you still should got for the end-to-end test.
So yes, I think your idea of going for end-to-end testing is very good.
Your idea of setting up simple commands to allow test automation for your system (the setup and the reset commands) is very good as well. Be prepared to add more during automation - while
an end-to-end-test is conceptually a black-box test, in reality it's often a grey-box test, i.e. you will need to access the internal state of your system. I would call this the "operation and maintenance interface" of the system under test.
I have need to periodically run a background task in my rails app. I've done some research and it seems that there is not a standard way to do this.
I don't want to generate a new app context so a runner seems to be out of the question. I'd prefer not to run my own thread handler if I don't have to. (It looks like this may be the most promising option at this point other than the below).
Using the whenever gem I am doing the following in schedule.rb:
every 15.minutes do
command "curl http://127.0.0.1:3000/periodic"
end
This feels a little like a hack, but I can't find anything that is more straightforward than this -- Everything else I can find seems like a lot of effort for something seemingly basic.
I should note that this is an internal project and will be behind a firewall, and headless automated middleware, so there is low expectation that a user will ever be accessing this project directly.
Am I missing something or is background processing or timed events really just not supported out of the box with rails?
After some research, Amadan's suggestion of a railtie may get me what I want here. As an ideal solution it isn't perfect, though depending on the context (i.e. a publicly accessible rails project) this is the way to go. For the project I am working on it's a little heavy handed, as I'm doing a rails app that is a bridge between two systems, running headless, with no user input behind a firewall.
Curious, what are you folks doing in as far as automating your unit tests with ruby on rails? Do you create a script that run a rake job in cron and have it mail you results? a pre-commit hook in git? just manual invokation? I understand tests completely, but wondering what are best practices to catch errors before they happen. Let's take for granted that the tests themselves are flawless and work as they should. What's the next step to make sure they reach you with the potentially detrimental results at the right time?
Not sure about what exactly do you want to hear, but there are couple of levels of automated codebase control:
While working on a feature, you can use something like autotest to get an instant feedback on what's working and what's not.
To make sure that your commits really don't break anything use continuous integration server like cruisecontrolrb or Integrity (you can bind these to post-commit hooks in your SCM system).
Use some kind of exception notification system to catch all the unexpected errors that might pop up in production.
To get some more general view of what happened (what was user doing when the exception occured) you can use something like Rackamole.
Hope that helps.
If you are developing with a team, the best practice is to set up a continuous integration server. To start, you can run this on any developers machine. But in general its nice to have a dedicated box so that its always up, is fast, and doesn't disturb a developer. You can usually start out with someone's old desktop, but at some point you may want it to be one of the faster machines so that you get immediate response from tests.
I've used cruise control, bamboo and teamcity and they all work fine. In general the less you pay, the more time you'll spend setting it up. I got lucky and did a full bamboo set up in less than an hour (once)-- expect to spend at least a couple hours the first time through.
Most of these tools will notify you in some way. The baseline is an email, but many offer IM, IRC, RSS, SMS (among others).