I have a section of code that runs if the user needs to re-auth after logging in. During UI tests, this popover is sometimes displayed, so I have a check for it existing
if (XCUIApplication().staticText["authLabel"].exists) {
completeAuthDialog()
}
When this runs locally, it is fine, completes and the framework finds the element no problem. But when the nightly job on the CI is ran, it fails the first time, but once the same build is set to be rebuilt, the test passes. authLabel is the UILabel's accessibility identifier(btw), so I have been trying to figure out what is causing the flickering.
Yesterday I spent time on the issue, and it seems that the framework just doesn't find the elements sometimes? I have used the accessibility inspector in ensure I am query for the same time it sees.
I even expanded that if check with 4 or 5 additional || to check for any element inside of the popover. The Elements all have accessibility identifiers, I have also used the record feature to ensure that it passes back the same element "names" I am using.
I am kind of stuck, I don't know what else to try/could be causing this issue. The worst part is it ran fine for couple of months, but it seems to fail every night now, and as I said when the tests are ran locally inside xcode they pass fine. Could this be a issue with building from command line?
It is often slower when your tests execute on a different machine, this problem seems particularly prevelant with CI machines as they tend to be under-powered.
If you just do a single check for an element existing, the test only has one point in time to get it right and if the app was slow to present the element then the test will fail.
You can defend against having a flaky test by using a waiter to check a few times over a few seconds to ensure that you've given the app enough time to show the authentication dialog before continuing.
let authElement = XCUIApplication().staticText["authLabel"]
let existsPredicate = NSPredicate(format: "exists == true")
let expectation = XCTNSPredicateExpectation(predicate: existsPredicate, object: authElement)
let result = XCTWaiter().wait(for: [expectation], timeout: 5)
if (result == .completed) {
completeAuthDialog()
}
You can adjust the timeout to suit your needs - a longer timeout will result in the test waiting a longer time to continue if the auth dialog doesn't appear, but will give the dialog more time to appear if the machine is slow. Try it out and see how flaky the tests are with different timeouts to optimise.
Related
I'm using WebdriverIO and Appium in Javascript to test Android/iOS apps.
I have a welcome screen that sometimes shows up after a loading screen. The following code is what I'm using at the moment to skip past the welcome screen.
if(welcomeScreenTitle.waitForDisplayed()){
skipWelcomeScreenButton.click();
}
The problem that I'm having is that if the waitForDisplayed() times out (meaning that the screen hasn't showed up this time), it fails the test. Is there a way that I can do this?
I have tried using
browser.wait(10000);
if(welcomeScreenTitle.isDisplayed()){
skipWelcomeScreenButton.click();
}
But the loading screen time is different depending on the speed of the connection (so might be much longer), and if the welcome screen shows up before 10 seconds, I don't want to wait the full 10 seconds (since most of the time it does show up).
One easier way of doing this is adding a try catch block around your code so that you can suppress the error thrown and continue the execution.
try {
browser.waitForDisaplyed(10000);
if(welcomeScreenTitle.isDisplayed()){
skipWelcomeScreenButton.click();
}
} catch (error) {
console.log('Welcomescreen is not displayed.')
}
first time I try to write unit test case. Here am blocking with the following scenario.
When I tap a button, it navigates to a next screen and webview will load. I need to wait for webview loading.
I tried to wait till the staticTexts is visible by below code,
let rewards = self.app.staticTexts[“Rewards”]
let exists = NSPredicate(format: “exists == 1”)
expectationForPredicate(exists, evaluatedWithObject: rewards, handler: nil)
waitForExpectationsWithTimeout(10, handler: nil)
XCTAssert(rewards.exists)
XCTAssert(app.staticTexts[“Rewards Overview”].exists)
But In my scenario, when I tap the button it will navigate to next screen and webview will starts to load. But the webview content is always dynamic one. So I need to wait till the webviewDidFinishLoad.
You can use waitForExistence(timeout:) to return a boolean if an element enters existence within the given timeout - I'd recommend using this for your use case.
https://developer.apple.com/documentation/xctest/xcuielement/2879412-waitforexistence
I would recommend writing your own function for waiting, since with waitForExistence(timeout:), you can only wait for existence, but it ignores other states of element(.isHittable, .isEnabled etc) or simply wait for any other boolean.
Plus waitForExistence(timeout:) is painfully slow, if you have a lot of tests with a lot of waits (we stopped using it after we wrote our own waiting). It is caused by the system method waiting for additional 1-2s before returning true, if the object exists.
This might help you, however its not Swift, its ObjC: https://pspdfkit.com/blog/2016/running-ui-tests-with-ludicrous-speed/
We're using the current version of the Firebase iOS framework (5.9.0) and we're seeing a strange problem when trying to run A/B test experiments that have an activation event.
Since we want to run experiments on first launch, we have a custom splash screen on app start that we display while the remote config is being fetched. After the fetch completes, we immediately activate the fetched config and then check to see if we received info about experiment participation to reconfigure the next UI appropriately. There are additional checks done before we determine that the current instance, in fact, should be part of the test, thus the activation event. Basically, the code looks like:
<code that shows splash>
…
[[FIRRemoteConfig remoteConfig] fetchWithExpirationDuration:7 completionHandler:^(FIRRemoteConfigFetchStatus status, NSError * _Nullable error) {
[[FIRRemoteConfig remoteConfig] activateFetched];
if (<checks that see if we received info about being selected to participate in the experiment and if local conditions are met for experiment participation>) {
[FIRAnalytics logEventWithName:#"RegistrationEntryExperimentActivation" parameters:nil];
<dismiss splash screen and show next UI screen based on experiment variation received in remote config>
} else {
<dismiss splash screen and show next UI screen>
}
}
With the approach above (which is completely straight-forward IMO) does not work correctly. After spending time with the debugger and Firebase logging enabled I can see in the log that there is a race-condition problem occurring. Basically, the Firebase activateFetched() call does not set up a "conditional user property experiment ID" synchronously inside the activateFetched call but instead sets it up some short time afterward. Because of this, our firing of the activation event immediately after activateFetched does not trigger this conditional user property and subsequent experiment funnel/goal events are not properly marked as part of an experiment (the experiment is not even activated in the first place).
If we change the code to delay the sending of the activation event by some arbitrary delay:
<code that shows splash>
…
[[FIRRemoteConfig remoteConfig] fetchWithExpirationDuration:7 completionHandler:^(FIRRemoteConfigFetchStatus status, NSError * _Nullable error) {
[[FIRRemoteConfig remoteConfig] activateFetched];
if (<checks that see if we received info about being selected to participate in the experiment and if local conditions are met for experiment participation>) {
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(0.5 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
[FIRAnalytics logEventWithName:#"RegistrationEntryExperimentActivation" parameters:nil];
<dismiss splash screen and show next UI screen based on experiment variation received in remote config>
}
} else {
<dismiss splash screen and show next UI screen>
}
}
the conditional user property for the experiment gets correctly setup beforehand and triggered by the event (causing experiment activation and subsequent events being correctly marked as part of the experiment).
Now, this code obviously is quite ugly and prone to possible race-conditions. The delay of 0.5 seconds is conservatively set to hopefully be enough on all iOS devices but ¯_(ツ)_/¯. I've read the available documentation multiple times and tried looking at all available API methods with no success in figuring out what the correct point of starting to send events should be. If the activateFetched method uses an asynchronous process of reconfiguring internal objects, one would expect a callback method that indicates to the caller the point in time when everything is done reconfiguring and ready for further use by the application. Seems the framework engineers didn't anticipate a use-case when someone needs to send the activation event immediatly after remote config profile activation…
Has anyone else experienced this problem? Are we missing something in the API? Is there a smarter way of letting activateFetched finish its thing?
Hope some Firebase engineers can chime-in with their wisdom as well :)
Thanks
In Xcode, is there a way for me run a single test case n times automatically?
Reason for doing this is that some of my beta testers are encountering random crashes in my app. I see the crash logs in TestFlight, along with the stack trace, but I can't reproduce the crash.
The crash happens infrequently but when it does, it always happens when users are trying to create a DB record, which then gets uploaded to a server. The problem with the crash logs is that my code does not make an appearance in their stack traces (all UIKit & CoreFoundation stuff - and different each time).
My solution is to run the test for that part of the app 100s of times, with the exception breakpoint set, to try to trigger the bug in my dev environment. But I don't know how to do this automatically.
It took 7 years, but as of Xcode 13, support for test repetition is now built-in.
From the Xcode 13 release notes:
Enable test repetition in your test plan, xcodebuild, or by running your test from the test diamond by Control-clicking and selecting Run Repeatedly to bring up the test repetition dialog.
You can read my answer here.
Basically you need to override invokeTest method
override func invokeTest() {
for time in 0...15 {
print("this test is being invoked: \(time) times")
super.invokeTest()
}
}
In Xcode as such, no.
You can create an XCTestCase class that hooks into the test-running methods it inherits to return multiple runs, but that tends to be annoying and mostly undocumented.
It's probably easier to instead have a "meta-test" that calls out to your other test method repeatedly:
func testOnce() {}
func testManyTimes() {
for _ in 0..<1000 { testOnce() }
}
You might need to call out to some per-test setup/teardown methods. You could perhaps work around that by instead making the loop body be something like:
let test = XCTestCase(selector: #selector(testOnce))
test.invokeTest()
This would lean on the XCTest machinery that your standard tests use, but it might gripe about not being wired into an XCTestCaseRun (or not).
Suppose I use QTPs recovery scenario manager to set the playback synchronization timeout to 0. The handler would return with "continue with next statement".
I'd do that to make sure that any following playback statements don't waste their time waiting for the next non-existing/non-matching step before failing:
I have a lot of GUI tests that kind of get stuck because let's say if 10 controls are missing, their (consecutive) playback steps produce 10 timeout waits before failing. If the playback timeout is 30 seconds, I loose 10x30 seconds=5 minutes execution time while it really would be sufficient to wait for 30 seconds ONCE (because the app does not change anymore -- we waited a full timeout period already).
Now if I have 100 test cases (=action iterations), this possibly happens 100 times, wasting 500 minutes of my test exec time window.
That's why I come up with the idea of a recovery scenario function setting the timeout to 0 after/upon the first failed playback step. This would accelerate the speed while skipping the rightly-FAILED step, yet would not compromise the precision/reliability of identifying the next matching GUI context (which creates a PASSED step).
Then of course upon the next passed playback step, I would want to restore the original timeout value. How could I do that? This is my question.
One cannot define a recovery scenario function that is called for PASSED steps.
I am currently thinking about setting a method function for Reporter.ReportEvent, and "sniffing" for PASSED log entries there. I'd install that method function in the scenario recovery function which sets timeout to 0. Then, when the "sniffer" function senses a ReportEvent call with PASSED status during one of the following playback steps, I'd reset everything (i.e. restore the original timeout, and uninstall the method function). (I am 99% sure, however, that .Click and .Set methods do not call ReportEvent to write their result status...so this option might probably not work.)
Better ideas? This really bugs me.
It sounds to me like you tests aren't designed correctly, if you fail to find an object why do you continue?
One possible (non recovery scenario) solution would be to use RegisterUserFunc to override the methods you are using in order to do an obj.Exist(0) before running the required method.
Function MyClick(obj)
If obj.Exist(1) Then
obj.Click
Else
Reporter.ReportEvent micFail, "Click failed, no object", "Object does not exist"
End If
End Function
RegisterUserFunc "Link", "Click", "MyClick"
RegisterUserFunc "WebButton", "Click", "MyClick"
''# etc
If you have many controls of which some may be missing and you know that after 10 seconds you mentioned (when the first timeout occurs), nothing more will show up, then you can use the exists method with a timeout parameter.
Something like this:
timeout = 10
For Each control in controls
If control.exists(timeout) Then
do something with the control
Else
timeout = 0
End If
Next
Now only the first timeout will be 10 seconds. Each and every subsequent timeout in your collection of controls will have the timeout set to 0 which will save your time.