I have some XCUITests that need to tap annotations on a map, when I run the tests in my mac everything works fine, but when they run on CI the map doesn't load.
https://i.stack.imgur.com/B6YEG.jpg
Ensure your CI device/simulator is connected to the internet. Your map load in CI may be slower than on a real device, so you may need to increase wait times.
Related
Xcode performance tests written with the XCTest.measure() API pass or fail depending on the "baseline" performance which must be set manually in Xcode and stored in ProjectName.xcodeproj/xcshareddata/xcbaselines/. As of Xcode 13.2.1 these baselines can only be set for your current device + simulator combo. For example, if you run the performance tests on another machine with a different processor it will ignore the baseline you set before. This makes sense because different configurations could have very different performance characteristics.
But this presents a problem when running performance test on a headless CI server. How can I set the performance baselines for my headless CI server?
I don't have access to the Xcode UI and xcodebuild doesn't seem to provide a way to set baselines.
And to be clear, the tests do execute sucessfully on my CI server, but they will never fail because baselines are not set.
How can we setup access to different builds so that they can easily / quickly viewed in the simulator..
We have several builds - production, development and one or more feature branches.
From a development and testing point of view, it would be really useful to be able to quickly see each of these versions of our app.
I'm just wondering if there are any services or some sort of configuration we could employ to make this easier.
I'm thinking one major obsticle might be the app id is always the same, so this would stop of from running the builds / app in the same simulator...
I'm wondering if the app could be bundled locally (perhaps from some kind of snapshot) and a script could copy in app to the simulator ?
I guess this could be some kind of tool which would use use the simulator command line tool and swap app builds.
I found a solution, all you need to do is keep a copy of the app and drag and drop it into the simulator
https://medium.com/swift2go/how-transfer-your-app-from-one-xcode-simulator-to-another-f4225da9eb7b
I do want to run shell script between two XCUITests. But since package is created and installed on the device or simulator, how this can be achieved? Is there a way to execute shell script on the host machine which has device connected(Either for Simulator or Real iPhone) between the tests?
You should probably set up Client-Server communication between device and host machine in order to do things like this.
This exact approach has been already implemented in
https://github.com/Subito-it/SBTUITestTunnelHost
Another option is to move shell code entirely to the test code. For example, if you use a shell script to communicate with a remote server, you should consider doing it on the device.
I think you can try inserting a script inside the test plan.
From Xcode:
Go to tab product
Press Test Plan
Manage Test Plans
Then, from the right side, select Test, and insert your script inside the pre-actions and/or pos-actions.
PS: I'm not sure if you can do that for selected tests. Maybe the scripts will run before/after each test.
I have a project which has an Xcode Tests suite and I want to implement a CI system by using "xcodebuild ... tests" command. The only limitation that I have is the hardware. I'm limited to only 2 Macs and 2 iPads and I also want to build for Debug and Release. The problem is that I have different app flavors and I would like to run a part of them (e.g. all the flavours on debug) on one iPad and the others on the other iPad.
I tried to call multiple times "xcodebuild" commands using the "tests" argument, hoping that it would be parallelized, but the problem is that it kills the tests app instance that is running already on the device.
I spent some time searching for someone who did this, but the only thing that was close to what I need was the usage of multiple iOS Simulators which I'm not comfortable because real device testing is required for the project.
Is it doable to run parallel tests on the same device? Am I missing some arguments that I need to add to the xcodebuild command?
No you can only run one test operation on 1 device at a time. Thats why people use multiple simulators as you can set them all to the same OS if you need to run the same test multiple times.
Not the answer you are looking for, but you could use Xcode server to have multiple bots, one for each flavour. If they all point to the same repo / branch, then they should all run one after the other. It would mean less manual effort building a tool to do it.
I also thought I read somewhere that with Xcode 10 you can load balance across multiple Xcode servers, but can't find a link to it. That might be helpful if you have 2 machines. If not you could duplicate the bots across both Xcode servers. Maybe using the API to control the creation of bots?
Xcode server setup guide
We have a TFS2010 instance that is hosted overseas. We have a proxy locally so pulling source code is really no problem. Even full gets are quite speedy. One issue that we are experiencing is that when an automated build completes it takes forever for the test results to be published back to the server. The entire build and test process takes about 5 minutes. The publishing of the test results takes an additional 10 minutes. Twice the time it took to get the source, build it, and run all 1500 unit tests!
Is there any way I can speed this up?
Try setting up a build agent at your location and set a name filter in the build definition to always select that build agent.