Show External Window When Mirroring (SwiftUI) - ios

I'm implementing a feature in my iPhone/iPad app where when the iPhone is connected to an external display, the iPhone acts as a controller and the external display shows a non-interactive view. I'm using SwiftUI.
My application is actually quite a similar concept to the one in Apple's documentation. The iPhone/iPad will always be used as a controller while the external display will always show the content, in Apple's case, a game. For this reason, when mirroring the game to QuickTime or StreamLabs, the actual game needs to be mirrored, not the controller. Here's the example I'm talking about in Apple's documentation so that you can visualize it.
Current Implementation
Here's how I've implemented it in the SceneDelegate, following Apple's documentation.
func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
guard let windowScene = scene as? UIWindowScene else { return }
if session.role == .windowApplication {
let window = UIWindow(windowScene: windowScene)
window.rootViewController = UIHostingController(rootView: ContentView())
window.makeKeyAndVisible()
}
if session.role == .windowExternalDisplayNonInteractive {
let window = UIWindow(windowScene: windowScene)
window.rootViewController = UIHostingController(rootView: ExternalView())
window.makeKeyAndVisible()
}
}
Where, of course, ContentView should be displayed on the iPhone and ExternalView is displayed on the external display or QuickTime. And here's my info.plist entry if that's relevant to you.
Outcome and the Problem
Ok, so this works fine for most cases, like when screen mirroring from my iPhone to a TV using AirPlay. Both views are displayed correctly. The problem I'm having, though, is that in certain cases, like trying to do a movie recording in QuickTime with the iPhone as the source or adding the iPhone as a video capture device in StreamLabs, the iPhone's screen (with the ContentView) is mirrored, instead of showing the ExternalView. The ExternalView needs to be shown when using these apps.
When looking at UIApplication.shared.openSessions, only one session is listed as described bellow
▿ 1 member
- <UISceneSession: 0x283594ac0; role: UIWindowSceneSessionRoleApplication; persistentIdentifier: ED82F2B9-17EC-435F-8E20-439CECCA92F6> {
scene = <UIWindowScene: 0x117d051e0>;
configuration = <UISceneConfiguration: 0x283595cc0; name: ContentView; role: UIWindowSceneSessionRoleApplication> {
sceneClass = 0x0;
delegateClass = SwiftUI.AppSceneDelegate;
storyboard = 0x0;
};
} #0
- super: NSObject
Is there a way to "create" a new session for screen mirroring? I'm beginning to worry that this is not possible with apps like QuickTime and StreamLabs and not something I can fix on my end.
Anyway, if you have any solutions to this issue I would very much appreciate any feedback at all.

Sadly this is not possible yet as far as I can tell. I have looked into this quite a bit and hope someone proves me wrong.
No screen mirroring or recording app supports recording the second display.
You can either mirror to your Mac or PC (with Reflector and record from there or go the Hardware route by using a capture card like the Elgato Game Capture HD60s with either a Lightning or USB-C adapter and then stream the gameplay form your PC.
If you hope to make your game streamable this won't help you though.

Related

Airplay button not showing for VolumeView

I've implemented the volumeView class onto my app so that I can pick and output my audio via airplay however the button for this is not showing.
I don't get any compiler errors and after reading Airplay Button is not showing in Player Controls with AVPlayer I can't work out what I'm doing differently.
The code:
override func viewDidLoad() {
super.viewDidLoad()
let volumeView = MPVolumeView(frame: myView.bounds);
self.view.addSubview(volumeView);
volumeView.showsRouteButton = true;
self.view.backgroundColor = UIColor.red;
volumeView.center = CGPoint(x:380, y:150)
Any advice is much appreciated.
I'm going to guess that this isn't supposed to work any more. On my device, if you change to an AirPlay device using the control center, the MPVolumeView disappears completely.
So I would advise against use of MPVolumeView, except perhaps for the simplest cases. It is extremely old technology and hasn't worked as advertised for years. Properties like showsRouteButton and areWirelessRoutesAvailable were deprecated years ago.
The user can change routes using the control center, or you can present an AVRoutePickerView; and you can detect route changes with AVRouteDetector. So you really don't need this feature of MPVolumeView anyway.

How to detect if another app is running as slide over in iOS 11?

The multitasking features got updates in iOS 11, one of those was slide over which is demonstrated in the gif below.
With these changes it's no longer possible to use the techniques that check frame size from iOS 9 to detect if another app is a "slide over" over my app.
Is there any new method to detect if another app is running as slide over?
I was able to get this working fairly easily on an iPad Pro (which supports side-by-side apps, not just slide-overs). Here's the code:
class ViewController: UIViewController {
override func viewWillLayoutSubviews() {
isThisAppFullScreen()
}
#discardableResult func isThisAppFullScreen() -> Bool {
let isFullScreen = UIApplication.shared.keyWindow?.frame == UIScreen.main.bounds
print("\(#function) - \(isFullScreen)")
return isFullScreen
}
}
The end result is that it will print "true" if the view is full screen and "false" if it's sharing the screen with another app, and this is run every time anything is shown, hidden, or resized.
The problem then is older devices that only support slide-over. With these, your app is not being resized anymore. Instead, it's just resigning active use and the other app is becoming active.
In this case, all you can do is put logic in the AppDelegate to look for applicationWillResignActive and applicationDidBecomeActive. When you slide-over, you get applicationWillResignActive but not applicationDidEnterBackground.
You could look for this as a possibility, but you cannot distinguish between a slide-over and a look at the Notifications from sliding down from the top of the screen. It's not ideal for that reason, but monitoring application lifecycle is probably the best you can do.

How to use AVCaptureSession with Slide Over and Split View in iOS 9?

My team is developing a set of SDKs for barcode scanning, ID scanning and OCR. We use device's camera, specifically, AVCaptureSession, to obtain video frames on which we perform our processing.
We're exploring new iOS 9 multitasking features Slide Over and Split View.
Apple suggests opting out of these features for camera-centric apps, where using the entire screen for preview and capturing a moment quickly is a primary feature (reference). This is the approach the use in their sample app AVCam.
However, our customers might have apps which don't fall into this category (e.g Mobile banking apps), so we cannot force them to opt out, instead, we need to handle new features in the SDK. We're exploring what would be the best approach to do that, since the docs at the moment aren't really telling us what to do.
We used our simple Camera sample app to analyse the use case. The sample app is available on Github and it's developed as of iOS 9 Beta 5.
From the sample app, it can be clearly seen which system events happen when Slide Over is used, and when Split View is used.
When our app is primary, and Slide Over is used, we get UIApplicationWillResignActiveNotification and AVCaptureSessionDidStopRunningNotification
When Slide Over is used, and our app is secondary, we get UIApplicationWillEnterForegroundNotification and AVCaptureSessionDidStopRunningNotification immediately after that
When Split View is used, on each divider drag, our app gets UIApplicationWillResignActiveNotification.
However, if the Camera is launched when in Split View, it immediately gets AVCaptureSessionDidStopRunningNotification
So, empirically, it looks like AVCaptureSession is immediately stopped when Slide Over or Split View are used.
What's confusing is that UIImagePickerController, which our sample app also supports, exhibits completely different behaviour.
UIImagePickerController isn't stopped when the app goes into Slide Over/ Split View, instead, it functions completely normally. One can normally take a photo in Split View. In fact, two apps, both of which present UIImagePickerController, can work side by side, with UIImagePickerController of the active app being active. (You can try that by running our sample app, and Contacts app -> New Contact -> Add photo)
With all this in mind, our questions are the following:
If AVCaptureSession is immediately paused when Slide Over and Split View are used, is it a good idea to monitor AVCaptureSessionDidStopRunningNotification, and present a message "Camera Paused" to the user, so that he clearly knows that the app isn't performing scanning?
Why is behaviour of UIImagePickerController different than AVCaptureSession?
Can we expect from Apple than in future beta versions behaviour of AVCaptureSession changes to match UIImagePickerController?
In case you haven't found out yet. After some more investigation I can now answer your first question:
If AVCaptureSession is immediately paused when Slide Over and Split
View are used, is it a good idea to monitor
AVCaptureSessionDidStopRunningNotification, and present a message
"Camera Paused" to the user, so that he clearly knows that the app
isn't performing scanning?
The notification you actually want to observe is this one: AVCaptureSessionWasInterruptedNotification
And you want to check for the newly introduced in iOS9 reason: AVCaptureSessionInterruptionReason.VideoDeviceNotAvailableWithMultipleForegroundApps
override func viewWillAppear(animated: Bool)
{
super.viewWillAppear(animated)
self.addObserverForAVCaptureSessionWasInterrupted()
}
func addObserverForAVCaptureSessionWasInterrupted()
{
let mainQueue = NSOperationQueue.mainQueue()
NSNotificationCenter.defaultCenter().addObserverForName(AVCaptureSessionWasInterruptedNotification, object: nil, queue: mainQueue)
{ (notification: NSNotification) -> Void in
guard let userInfo = notification.userInfo else
{
return
}
// Check if the current system is iOS9+ because AVCaptureSessionInterruptionReasonKey is iOS9+ (relates to Split View / Slide Over)
if #available(iOS 9.0, *)
{
if let interruptionReason = userInfo[AVCaptureSessionInterruptionReasonKey] where Int(interruptionReason as! NSNumber) == AVCaptureSessionInterruptionReason.VideoDeviceNotAvailableWithMultipleForegroundApps.rawValue
{
// Warn the user they need to get back to Full Screen Mode
}
}
else
{
// Fallback on earlier versions. From iOS8 and below Split View and Slide Over don't exist, no need to handle anything then.
}
}
}
override func viewWillDisappear(animated: Bool)
{
super.viewWillDisappear(true)
NSNotificationCenter.defaultCenter().removeObserver(self)
}
You can also know when the interruption was ended by observing:
AVCaptureSessionInterruptionEndedNotification
Answer based on these two links:
http://asciiwwdc.com/2015/sessions/211
https://developer.apple.com/library/ios/samplecode/AVCam/Introduction/Intro.html
Since iOS 16.0+, it's possible to use isMultitaskingCameraAccessEnabled flag.
Reference
Since iOS 13.5+ and iPadOS 13.5+, it's possible to use entitlement com.apple.developer.avfoundation.multitasking-camera-access, allowing the app to continue using the camera while running alongside another foreground app.
Reference
More information about accessing the camera while multitasking
here

Display one thing on iPad and another on Apple Tv?

I have an app idea, but I'm not sure if it's possible.
I was wondering if I'm able to display one thing on the iPad ( or iPhone )
screen, and something totally different on the Apple Tv at the same time.
For example, a quiz app, where the question is displayed on the Apple Tv, and the multiple choices are listed on the iPad for the user to pick.
I'm not sure if this is possible or if you can only Mirror the iPad screen onto the Apple Tv.
If there is some "Proof of Concept" example code, I'd love to take a look.
Thank you so much.
Chris
Turns out that is is pretty simple to support two screens: the primary screen of the iOS device and a secondary screen (either an external display or mirroring on an Apple TV).
Based on information from the blog post Creating a Dual-Screen AirPlay Experience for iOS and Apple TV, you don't need to do much.
Basically you need to check the screens property from UIScreen. There are also notifications you should listen for (UIScreenDidConnectNotification and UIScreenDidDisconnectNotification) so you know if the number of screens changes while your app is running.
Once you have a second screen, you need to create a new window for it. Code like the following can be used:
if ([UIScreen screens].count > 1) {
if (!_secondWin) {
UIScreen *screen = [UIScreen screens][1];
_secondWin = [[UIWindow alloc] initWithFrame:screen.bounds];
_secondWin.screen = screen;
}
}
where _secondWin is a UIWindow ivar.
Once the window is setup, create a view controller, make it the window's root view controller, and show the window:
SomeViewController *vc = [[SomeViewController alloc] init...];
_secondWin.rootViewController = vc;
_secondWin.hidden = NO;
This is pretty much it other than proper handling of the notifications. Keep in mind that you can't get any touch events on the 2nd display so make sure whatever you show is basically display-only.
Depending on your app, you might have the 2nd screen/window being used throughout the lifetime of the app (as long as the 2nd screen is available any way). Or you might only create and use the 2nd window/screen under certain circumstances. When you don't setup the 2nd window/screen, your app will simply be mirrored to the 2nd display or Apple TV.
The last piece is to turn on mirroring to the Apple TV. This is done on the iOS device, not in the app.
The blog post I linked has a few more details worth reviewing.

Launch iOS simulator from Xcode and getting a black screen, followed by Xcode hanging and unable to stop tasks

I'm having trouble running my basic iPhone application (while going through the Stanford iTunes CS193p lectures) in the iOS simulator.
I've been searching for a while (both Google and SO), but unable to find a solution so far. There are many similar bugs, but the solutions don't seem to fix this.
In Xcode I click "run". It compiles and builds successfully, launches iOS simulator but it never gets to loading the app. Only the status bar at the top. With a black screen.
I've only written very basic code (following along with the lectures) and can't get past this problem.
To confuse matters more, I wrote a web wrapper (UIWebView) before these lectures and this works fine. But there is barely any difference in the code. All new apps I create from scratch all fail with the same black screen problem.
If I hit the home button on the simulator and launch the app, it will display. But Xcode doesn't seem to know what's going on.
It's as if Xcode has lost the ability to talk to iOS Simulator and assumes it's running (even if I quit iOS simulator). I try and quit Xcode, and it asks me to stop the tasks. Then it just hangs. So I have to force restart to get out of Xcode.
I'm using:
OSX 10.8.2
Xcode 4.5.2
iOS Simulator 6.0
CalculatorAppDelegate.h
#import <UIKit/UIKit.h>
#interface CalculatorAppDelegate : UIResponder <UIApplicationDelegate>
#property (strong, nonatomic) UIWindow *window;
#end
CalculatorAppDelegate.m
#import "CalculatorAppDelegate.h"
#implementation CalculatorAppDelegate
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions (NSDictionary *)launchOptions
{
// Override point for customization after application launch.
return YES;
}
- (void)applicationWillResignActive:(UIApplication *)application
{
// Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state.
// Use this method to pause ongoing tasks, disable timers, and throttle down OpenGL ES frame rates. Games should use this method to pause the game.
}
- (void)applicationDidEnterBackground:(UIApplication *)application
{
// Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later.
// If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits.
}
- (void)applicationWillEnterForeground:(UIApplication *)application
{
// Called as part of the transition from the background to the active state; here you can undo many of the changes made on entering the background.
}
- (void)applicationDidBecomeActive:(UIApplication *)application
{
// Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface.
}
- (void)applicationWillTerminate:(UIApplication *)application
{
// Called when the application is about to terminate. Save data if appropriate. See also applicationDidEnterBackground:.
}
#end
CalculatorViewController.h
#import <UIKit/UIKit.h>
#interface CalculatorViewController : UIViewController
#property (weak, nonatomic) IBOutlet UILabel *display;
#end
CalculatorViewController.m
#import "CalculatorViewController.h"
#implementation CalculatorViewController
#synthesize display = _display;
- (IBAction)digitPressed:(UIButton *)sender
{
NSString *digit = [sender currentTitle];
NSLog(#"digit pressed = %#", digit);
}
#end
Surprisingly, what worked for me was going to iOS Simulator menu, and pressing "Reset Content and Settings". (in iOS 13, its under Hardware)
Before resetting the emulator first go to your projects "project navigator" screen and under the general -> depoyment info screen check that the main interface property is properly setup!
If you're using SwiftUI
If you're updating from a previous version of Xcode 11, there are some changes to the SceneDelegate willConnectTo session: options connectionOptions initialization:
The main window is now initialized using UIWindow(windowScene: windowScene), where it use to be UIWindow(frame: UIScreen.main.bounds)
On previous version:
let window = UIWindow(frame: UIScreen.main.bounds)
window.rootViewController = UIHostingController(rootView: ContentView())
self.window = window
window.makeKeyAndVisible()
In new version:
if let windowScene = scene as? UIWindowScene {
let window = UIWindow(windowScene: windowScene)
window.rootViewController = UIHostingController(rootView: ContentView())
self.window = window
window.makeKeyAndVisible()
}
I am a newbie to the iOS app development. I was practising to develop iOS apps from the very beginning and while running a very basic Hello World app, I also faced same issue that only a black blank screen appears after building and running the app in iOS simulator.
Somehow while struggling to find out a solution to the problem I accidentally clicked Window-->Scale-->50% in iOS simulator and it did solve my problem.
I could then see the Home page with my app and clicking on app icon I was able to run my app successfully.
App versions:
Xcode 5.1
iOS Simulator: 7.1
OSX: 10.9.3
Restarting your computer should be all you need to do.
This could result from not setting the correct deployment info. (i.e. if your storyboard isn't set as the main interface)
I struggled with this for a couple of hours.
Finally what solved it for me was:
sudo xcrun simctl erase all
xcrun simctl shutdown all
Solution 1 : You can Quit simulator and try again.
Solution 2 (Recommended) : Go to iOS Simulator -> Reset Content and Settings...
This will pop-up an alert stating 'Are you sure you want to reset the iOS Simulator content and settings?'
Select Reset if you wish to or else Don't Reset button.
Note that once you reset simulator content all app installed on simulator will be deleted and it will reset to initial settings.
Another Solution is that if you are building UI programatically in Objective C project, then you might need to add some code to inflate the view, so you will need to add those 3 lines to make the window's interface interacts with the code (the outer 2 are actually interacting with the code and the other one will change the screen to white so you will know it works).
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
self.window = [[UIWindow alloc] initWithFrame: [[UIScreen mainScreen] bounds]];
// Override point for customization after application launch.
...
self.window.backgroundColor = [UIColor whiteColor];
[self.window makeKeyAndVisible];
return YES;
}
If you should loose your entry point in your Storyboard or simply wish to change the entry point you can specify this in Interface Builder. To set a new entry point you must first decide which ViewController will act as the new entry point and in the Attribute Inspector select the Initial Scene checkbox.
You can try:
http://www.scott-sherwood.com/ios-5-specifying-the-entry-point-of-your-storyboard/
Please make sure you have done this,if you are getting black screen after copying the storyboard from another project
I fixed this issue by cleaning the project by pressing
cmd + shift + k
and exited my simulator and built again.
For me all it was needed was to shake the device (Device > Shake). Weird but worked for me. If this doesn't work try resetting (Device > Erase All Content and Settings) and then shake the device again.
I solved this question with set window background color like this in iOS 13:
func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
guard let _ = (scene as? UIWindowScene) else { return }
if let windowScene = scene as? UIWindowScene {
window = UIWindow(windowScene: windowScene)
window?.backgroundColor = .white
let storyBoard = UIStoryboard(name: "Main", bundle: nil)
// workaround for svprogresshud
let appDelegate = UIApplication.shared.delegate as! AppDelegate
appDelegate.window = window
}
}
I've managed to find the fix for this. It was found courtesy of this blog post:
http://vandadnp.wordpress.com/2012/03/18/xcode-4-3-1-cannot-attach-to-ios-simulator/
The solution is to press cmd+shift+, (command, shift and then comma ",").. that loads some options for release or debugging.
Change the debugger from LLDB to GDB. This fixes the issue.
To make sure it's a simulator issue, see if you can connect to the simulator with a brand new project without changing any code. Try the tab bar template.
If you think it's a simulator issue, press the iOS Simulator menu. Select "Reset Content and Settings...". Press "Reset."
I can't see your XIB and what #properties you have connected in Interface Builder, but it could also be that you're not loading your window, or that your window is not loading your view controller.
I had the same issue with Xcode... black screen on launching apps, no debugging and clicking would lock up Xcode.
I finally found the problem... following the lead that the simulator could not connect to Xcode I took a look at my etc/hosts file and found that months ago to solve a different issue I had edited the host file to map localhost to my fixed IP instead of the default... my value:
10.0.1.17 localhost
This should work since that is my IP, but changing it back to the default IP fixed Xcode...
127.0.0.1 localhost
Hope this helps.
I was doing what doug suggests ("Reset Content and Settings") which works but takes a lot of time and it is really annoying... until I recently found completely accidental another solution that is much quicker and seems to also work so far! Just hit cmd+L on your simulator or go to the simulator menu "Hardware -> Lock", which locks the screen, when you unlock the screen the app works like nothing ever happened :)
What happened with me was the Type of class was not UIViewController for the script attached to my view controller. It was for a UITabController..... I had mistakenly quickly created the wrong type of class.... So make sure the class if the correct type.. Hope this helps someone. I was in a rush and made this mistake.
Just reset your simulator by clicking Simulator-> Reset Contents and Settings..
you could also go to Hardware -> reboot, then Hardware -> Home, and click on your App
Be sure that Initial View Controller is set
I solved this only after removing simulators with prior iOS versions.
I had black simulator screens only for iOS 11 sims. After trying to reset, reboot, reinstall and creating a brand new useraccount on my machine I found this solution:
defaults write com.apple.CoreSimulator.IndigoFramebufferServices FramebufferRendererHint 3
found in this answer here: Xcode 9 iOS Simulator becoming black screen after installing Xcode 10 beta
Please check for all the constraints for all the views and review complete storyboard. Usually this happens because of that.
Regards,
Arora
In case you recently updated to Xcode 11 beta 3 and try to run an older SwiftUI project, you have to make some changes in the SceneDelegate where the views are loaded, otherwise the screens will remain black on devices running iOS 13 beta 3 which of course includes all simulators.
func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
// Use this method to optionally configure and attach the UIWindow `window` to the provided UIWindowScene `scene`.
// If using a storyboard, the `window` property will automatically be initialized and attached to the scene.
// This delegate does not imply the connecting scene or session are new (see `application:configurationForConnectingSceneSession` instead).
// Use a UIHostingController as window root view controller
if let windowScene = scene as? UIWindowScene {
let window = UIWindow(windowScene: windowScene)
window.rootViewController = UIHostingController(rootView: ContentView())
self.window = window
window.makeKeyAndVisible()
}
}
The behavior not strictly limited to the simulators, but since most people will run beta software exclusively on simulators it will only occur in this context. It baffled me for quite a while, so hope it helps.
When i have carried my project on Xcode 11.1, i got that problem. That black screen problem may occur any presentation inter ViewControllers.
That
answer helped me. Because modal presentation changed with iOS 13.
If you don't get that problem before iOS 13, please try to add line below to your ViewController before its presentation;
viewController.modalPresentationStyle = .fullScreen
after your code may seem like below;
let vc = UIViewController()
vc.modalPresentationStyle = .fullScreen //or .overFullScreen for transparency
self.present(vc, animated: true, completion: nil)
I was developing flutter using android studio and running the simulator, the simulator was dark and I reset the contents as per accepted answer. But that didn't fix my issue. My CPU was running somehow high enough, so I restarted , but not fixed the issue. I opened Xcode and run one of my native iOS apps, the simulator re-opened as usual. I closed the Xcode and opened Android Studio, then started developing flutter app again.
This was my reason for having this issue.
For some reason, the "Is Initial View Controller" (for my main View Controller) was unchecked which was causing the black screen on load up.
Hope this helps someone out there!
I noticed I had accidentally deleted var window:UIWindow? line from AppDelegate. Introducing it again fixed the issue for me!

Resources