When creating a custom video player using the AVPlayer + AVPlayerLayer + AVPictureInPictureController for a iPhone running iOS 14 (beta 7) the video does not automatically enter picture-in-picture-mode when the app enters the background after player.start() is called from a UIButton action.
The issue does not reproduce using the AVPlayerViewController which seems to indicate a problem with the AVPictureInPictureController on iOS 14 in general, but I was wondering if anyone else had run into this problem and know of any workarounds. I've also filed this problem with Apple under rdar://8620271
Sample code.
import UIKit
import AVFoundation
import AVKit
class ViewController: UIViewController {
private let player = AVPlayer(url: URL(string: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4")!)
private var pictureInPictureController: AVPictureInPictureController!
private var playerView: PlayerView!
private var playButton: UIButton!
override func viewDidLoad() {
super.viewDidLoad()
playerView = PlayerView(frame: CGRect(x: 0, y: 44, width: view.bounds.width, height: 200))
playerView.backgroundColor = .black
playerView.playerLayer.player = player
view.addSubview(playerView)
playButton = UIButton(frame: CGRect(x: view.bounds.midX - 50, y: playerView.frame.maxY + 20, width: 100, height: 22))
playButton.setTitleColor(.blue, for: .normal)
playButton.setTitle("Play", for: .normal)
playButton.addTarget(self, action: #selector(play), for: .touchUpInside)
view.addSubview(playButton)
pictureInPictureController = AVPictureInPictureController(playerLayer: playerView.playerLayer)
do {
let audioSession = AVAudioSession.sharedInstance()
try audioSession.setCategory(.playback)
try audioSession.setMode(.moviePlayback)
try audioSession.setActive(true)
} catch let e {
print(e.localizedDescription)
}
}
#objc func play() {
player.play()
}
}
class PlayerView: UIView {
override class var layerClass: AnyClass {
return AVPlayerLayer.self
}
var playerLayer: AVPlayerLayer! {
return layer as? AVPlayerLayer
}
}
The root cause of the problem ended up being twofold:
AVAudioSession.sharedInstance().setActive(true) must be called before the AVPictureInPictureController is initialised.
The frame size for the AVPlayerLayer must have a aspect ratio no greater than 16/9 (filed as a separate bug, rdar://8689203)
For iPads, the video must be the same width as the device (in any given orientation). No separate rdar, as Apple have acknowledged the other bug already.
(The 2nd issues is not present in the example above)
Apple have acknowledged these bugs, and reported back to me that they have been / will be fixed (a rare case of a radar actually resulting in a reply!)
Starting iOS 14.2, Apple has exposed an api to start PIP when app goes into background:
if #available(iOS 14.2, *) {
pictureInPictureController.canStartPictureInPictureAutomaticallyFromInline = true
}
Additionally, it is worth noting that Apple has forbidden to start picture-in-picture without user manually tapping the button. It will result in app rejection.
Best bet is to use Apple's API mentioned above to avoid rejection.
Related
Note: The question remains unsolved for now; the marked answer provides a good workaround - it works while the application is still open. Answers are still welcomed!
Background
I'm currently developing an app that consists of a full-screen PDFView, and I want the program to remember the position in the document before the view is dismissed so the user can pick up where they've left.
Implementation
A simplified version of the app can be understood as a PDF Viewer using PDFKit.PDFView. The storyboard consists of an UIView that's connected to a PDFView class, DocumentView (which conforms to UIViewController). The view is initialised through the following process:
In viewDidLoad:
let PDF: PDFDocument = GetPDFFromServer()
DocumentView.document = PDF!
DocumentView.autoScales = true
... (other settings)
// Set PDF Destination
let Destination: PDFDestination = GetStoredDestination()
// Code with issues
DocumentView.go(to: Destination)
In viewWillDisappear:
StoreDestination(DocumentView.currentDestination)
The Issue & Tests
I realised that the code does not work as expected; the view does not return to its previous location.
Through debugging, I realised that this might be due to the inconsistent behaviour of DocumentView.go(to destination: PDFDestination) and DocumentView.currentDestination.
To ensure the bug is not introduced by errors while storing the location, the following code is used to verify the issue, with a multi-page document:
In viewDidLoad
Timer.scheduledTimer(withTimeInterval: 1, repeats: true, block: { _ in
DispatchQueue.main.async {
self.DocumentView.go(to:self.DocumentView.currentDestination!)
}
})
Expected & Observed behaviour
Expected: The location of the document should not change - the code is going to its current destination every 1 second which should have no effects. as "currentDestination" should be the "current destination of the document, per docs")
Observed: Upon execution, the page would spontaneously scroll down by a fixed offset.
The same outcome was observed on an iPadOS 14.5 simulator and an iPadOS 15 iPad Air (Gen 4).
What might have gone wrong?
It'd be great if somebody can help.
Cheers,
Lincoln
This question was originally published on the Apple Developer Forum over a week ago; No responses were heard for over a week, so I thought I might try my luck here on StackOverflow <3
I tried PDFView.go() for this scenario and I managed to get it work for some cases but found that it fails in some other scenarios such as with zoomed documents, changed orientations.
So going back to what you are trying to achieve,
I'm currently developing an app that consists of a full-screen
PDFView, and I want the program to remember the position in the
document before the view is dismissed so the user can pick up where
they've left.
this can be done from a different approach. With this approach, you need to always keep a reference to the PDFView you created. If the previous pdf needs to be loaded again, then you pass the PDFView instance you have to the viewController as it is. Otherwise you load the new pdf to the PDFView instance and pass it to the viewController.
DocumentViewController gets the PDFView when it gets initialized.
import UIKit
import PDFKit
protocol DocumentViewControllerDelegate: AnyObject {
func needsContinuePDF(continuePDF: Bool)
}
class DocumentViewController: UIViewController {
var pdfView: PDFView!
weak var delegate: DocumentViewControllerDelegate!
init(pdfView: PDFView, delegate: DocumentViewControllerDelegate){
self.pdfView = pdfView
self.delegate = delegate
super.init(nibName: nil, bundle: nil)
}
required init?(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
override func loadView() {
super.loadView()
self.view.backgroundColor = .white
view.addSubview(pdfView)
NSLayoutConstraint.activate([
pdfView.leadingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.leadingAnchor),
pdfView.trailingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.trailingAnchor),
pdfView.topAnchor.constraint(equalTo: view.safeAreaLayoutGuide.topAnchor),
pdfView.bottomAnchor.constraint(equalTo: view.safeAreaLayoutGuide.bottomAnchor)
])
}
override func viewWillDisappear(_ animated: Bool) {
delegate.needsContinuePDF(continuePDF: true)
}
}
You can initialize DocumentViewController like below. MainViewController has the responsibility for initializing PDFView.
import UIKit
import PDFKit
class MainViewController: UIViewController {
var pdfView: PDFView = PDFView()
var continuePreviousPDF = false
let button = UIButton(frame: .zero)
override func loadView() {
super.loadView()
button.setTitle("View PDF", for: .normal)
button.setTitleColor(.white, for: .normal)
button.backgroundColor = .systemBlue
button.addTarget(self, action: #selector(openDocumentView(_:)), for: .touchUpInside)
self.view.backgroundColor = .systemGray5
self.view.addSubview(button)
button.translatesAutoresizingMaskIntoConstraints = false
NSLayoutConstraint.activate([
button.widthAnchor.constraint(equalToConstant: 100),
button.heightAnchor.constraint(equalToConstant: 50),
button.centerXAnchor.constraint(equalTo: self.view.safeAreaLayoutGuide.centerXAnchor),
button.centerYAnchor.constraint(equalTo: self.view.safeAreaLayoutGuide.centerYAnchor),
])
}
#objc func openDocumentView(_ sender: UIButton) {
//open a nee PDF if not continue previous one
if !self.continuePreviousPDF {
pdfView.autoScales = true
pdfView.displayMode = .singlePageContinuous
pdfView.translatesAutoresizingMaskIntoConstraints = false
guard let path = Bundle.main.url(forResource: "sample copy", withExtension: "pdf") else { return }
if let document = PDFDocument(url: path) {
pdfView.document = document
}
}
let documentViewController = DocumentViewController(pdfView: pdfView, delegate: self)
self.present(documentViewController, animated: true, completion: nil)
}
}
extension MainViewController: DocumentViewControllerDelegate {
func needsContinuePDF(continuePDF: Bool) {
self.continuePreviousPDF = continuePDF
}
}
When navigating to a subview, I have set it up so that a video plays automatically. At the bottom of the video, there is a group of links that go to related content. When clicking one of them, a new view is pushed onto the stack and a different video starts playing.
The problem happens when using the automatically generated '< Back' button to go back to the prior view (which had a different video). This original view can be operated using the player controls, but nothing shows up on the screen.
I've tried to update the CGRect frame, use onAppear to reinitialize the video player, and also followed the advice here.
So far nothing seems to work. Here is the code I am using for the actual video player (adapted from Chris Mash's website):
import SwiftUI
import AVKit
import UIKit
import AVFoundation
let playerLayer = AVPlayerLayer()
class PlayVideo: UIView {
init(frame: CGRect, url: URL) {
super.init(frame: frame)
// Create the video player using the URL passed in.
let player = AVPlayer(url: url)
player.volume = 100 // Will play audio if you don't set to zero
player.play() // Set to play once created
// Add the player to our Player Layer
playerLayer.player = player
playerLayer.videoGravity = .resizeAspectFill // Resizes content to fill whole video layer.
playerLayer.backgroundColor = UIColor.black.cgColor
layer.addSublayer(playerLayer)
}
required init?(coder: NSCoder) {
super.init(coder: coder)
}
override func layoutSubviews() {
super.layoutSubviews()
playerLayer.frame = bounds
}
static func pauseVideo() {
playerLayer.player?.pause()
}
}
struct ViewVideo: UIViewRepresentable {
var videoURL:URL
var previewLength:Double?
func makeUIView(context: Context) -> UIView {
return PlayVideo(frame: .zero, url: videoURL)
}
func updateUIView(_ uiView: UIView, context: Context) {
}
}
This is called from the main view using:
ViewVideo(videoURL: videoURL)
The only work around I can think of is to disable the back button and force the user to go back to the main view every time. That's a terrible option and I'm hoping someone will have some helpful advice here. Thanks -
If I understand this correctly, you play a different video, when you navigate to the new view. So you create a new PlayVideo view?
Then the problem is that your playerLayer is a static property. The new view will set a new player into the playerLayer and replace the old one. Similarly, if you pause one player, both are paused. Additionally, adding the player as a sublayer to the new view will remove it from the old view.
You need the playerLayer as a local property to your view. Or at least a AVPlayerLayer for every video you want to play. Then you need a mechanism for pausing/restarting each video, when it becomes visible. For example by implementing the viewWillAppear. This method gets always called, when you navigate back to a view/ the view becomes visible.
Thank you to #dominik-105 for helping with this question. I was able to fix the problem using the suggestions that were made.
Specifically, I removed the global definition of playerLayer and instead placed it as a local variable in my main view call:
var playerLayer = AVPlayerLayer()
I then call ViewVideo with the playerLayer: playerLayer tag, and the ViewVideo then calls PlayVideo with playerLayer:AVPlayerLayer as part of the init.
Interestingly, this leads to problems with the override layout function I was using to define the size of the video box. I define the frame directly in the init now and removed the old override function code. The full code is now:
import SwiftUI
import AVKit
import UIKit
import AVFoundation
class PlayVideo: UIView {
init(frame: CGRect, url: URL, playerLayer: AVPlayerLayer, width: CGFloat, height: CGFloat) {
super.init(frame: frame)
// Create the video player using the URL passed in.
let player = AVPlayer(url: url)
player.volume = 100 // Will play audio if you don't set to zero
player.play() // Set to play once created
// Add the player to our Player Layer
playerLayer.player = player
playerLayer.videoGravity = .resizeAspectFill // Resizes content to fill whole video layer.
playerLayer.backgroundColor = UIColor.black.cgColor
playerLayer.player?.actionAtItemEnd = .pause
layer.addSublayer(playerLayer)
playerLayer.frame = CGRect(x: 0, y: 0, width: width, height: height)
}
required init?(coder: NSCoder) {
super.init(coder: coder)
}
}
struct ViewVideo: UIViewRepresentable {
var videoURL:URL
var playerLayer: AVPlayerLayer
var width: CGFloat
var height: CGFloat
func makeUIView(context: Context) -> UIView {
return PlayVideo(frame: .zero, url: videoURL, playerLayer: playerLayer, width: width, height: height)
}
func updateUIView(_ uiView: UIView, context: Context) {
}
}
I use GeometryReader to define the size of the box, and then pass along the width and height to the struct, which passes it along to the class.
Is it possible to put a loading animation over the VNDocumentViewController? As in, when the user presses the Save button, is there a way for me to somehow indicate that the Vision is processing the image and hasn't frozen? Right now, in my app, there is a long pause between the user pressing Save and the actual image being processed.Here is an example from another post of what I'm trying to create
Here is one example of adding a loading indicator using UIActivityIndicatorView().
startAnimating() to start the animation and stopAnimation() to stop the animation.
iOS - Display a progress indicator at the center of the screen rather than the view
guard let topWindow = UIApplication.shared.windows.last else {return}
let overlayView = UIView(frame: topWindow.bounds)
overlayView.backgroundColor = UIColor.clear
topWindow.addSubview(overlayView)
let hudView = UIActivityIndicatorView()
hudView.bounds = CGRect(x: 0, y: 0, width: 20, height: 20)
overlayView.addSubview(hudView)
hudView.center = overlayView.center
hudView.startAnimating()
Alternatively, you could look into using Cocoapod MBProgressHud
https://cocoapods.org/pods/MBProgressHUD
There's a way you can extend a class in Swift that captures this problem well. The idea is you want a UIActivityIndicator in your VNDocumentCameraViewController. But we'd like that to be a part of every version of this we use. We could simply embed the DocumentVC's view into our current view and superimpose a UIActivityIndicator above it in the view stack, but that's pretty hacky. Here's a quick way we can extend any class and solve this problem
import VisionKit
import UIKit
extension VNDocumentCameraViewController {
private struct LoadingContainer {
static var loadingIndicator = UIActivityIndicatorView()
}
var loadingIndicator: UIActivityIndicatorView {
return LoadingContainer.loadingIndicator
}
func animateLoadingIndicator() {
if loadingIndicator.superview == nil {
view.addSubview(loadingIndicator)
//Setup your constraints through your favorite method
//This constrains it to the very center of the controller
loadingIndicator.frame = CGRect(
x: view.frame.width / 2.0,
y: view.frame.height / 2.0,
width: 20,
height: 20)
//Setup additional state like color/etc here
loadingIndicator.color = .white
}
loadingIndicator.startAnimating()
}
func stopAnimatingLoadingIndicator() {
loadingIndicator.stopAnimating()
}
}
The place we can call these functions are in the delegate methods for VNDocumentCameraViewController that you implement in your presenting ViewController:
func documentCameraViewController(
_ controller: VNDocumentCameraViewController,
didFinishWith scan: VNDocumentCameraScan
) {
controller.animateLoadingIndicator()
}
Context:
I am trying to play multiple MIDI sequences in an iOS app using AVFoundation. The tracks are loaded in MIDI files and I have successfully managed to play them one by one if I load them in a AVAudioSequencer. I also have an AVAudioUnitSampler object which is connected in an AVAudioEngine and it successfully plays the selected instrument from the loaded sound bank in the sampler.
The setup works correctly if I create a new AVAudioSequencer each time I play a sound. However, if I would like to reuse a sequencer after it's finished, it sounds like it's not using the sampler's instrument.
I suspect when I create the AVAudioSequencer objects they are automatically connected to the AVAudioEngine but only the last object will get the connected to the sampler.
I've tried to manually connect the destinationAudioUnit of the tracks in the sequencer to the sampler, but then it doesn't play a sound at all. I also tried to make multiple samplers and connect them all to the engine, but that didn't work either.
My main question is: What is the proper way of using multiple AVAudioSequencer objects with one AVAudioUnitSampler? Or do I need to create a sampler for each sequencer and connect them somehow?
Here's a very basic playground example of two sequencers. When I run it, sequencer B successfully plays the sound through the sampler, but A is not using the instrument.
import UIKit
import PlaygroundSupport
import AVFoundation
class MyViewController : UIViewController {
let buttonA = UIButton(type: .system), buttonB = UIButton(type: .system)
let engine = AVAudioEngine()
lazy var sequencerA = AVAudioSequencer(audioEngine: engine)
lazy var sequencerB = AVAudioSequencer(audioEngine: engine)
let sampler = AVAudioUnitSampler()
// UI setup
override func loadView() {
let view = UIView()
view.backgroundColor = .white
buttonA.setTitle("Sequencer A", for: .normal)
buttonB.setTitle("Sequencer B", for: .normal)
buttonA.addTarget(self, action: #selector(playSequencerA), for: .touchUpInside)
buttonB.addTarget(self, action: #selector(playSequencerB), for: .touchUpInside)
view.addSubview(buttonA)
view.addSubview(buttonB)
buttonA.frame = CGRect(x: 150, y: 200, width: 100, height: 100)
buttonB.frame = CGRect(x: 150, y: 300, width: 100, height: 100)
self.view = view
}
// Sound engine setup
override func viewDidLoad() {
engine.attach(sampler)
engine.connect(sampler, to: engine.mainMixerNode, format: nil)
let soundBankPath = playgroundSharedDataDirectory.appendingPathComponent("gs_instruments.dls")
let midiA = playgroundSharedDataDirectory.appendingPathComponent("sfx_a.mid")
let midiB = playgroundSharedDataDirectory.appendingPathComponent("sfx_b.mid")
try! sampler.loadSoundBankInstrument(at: soundBankPath, program: 11, bankMSB: UInt8(kAUSampler_DefaultMelodicBankMSB), bankLSB: UInt8(kAUSampler_DefaultBankLSB))
try! sequencerA.load(from: midiA, options: [])
try! sequencerB.load(from: midiB, options: [])
try! engine.start()
}
#objc public func playSequencerA() { play(sequencerA) }
#objc public func playSequencerB() { play(sequencerB) }
func play(_ sequencer: AVAudioSequencer) {
if sequencer.isPlaying { sequencer.stop() }
sequencer.currentPositionInBeats = 0
try! sequencer.start()
}
}
PlaygroundPage.current.liveView = MyViewController()
Edit:
After some additional experiments I suspect that the AVAudioEngine cannot have more than one AVAudioSequencer instance (or I'm still doing something wrong). As a workaround, I have created a separate AVAudioEngine object for each MIDI file that I need to play and they all have their own sampler and sequencer inputs, which plays the sounds just fine. I'm pretty sure this solution is not optimal, so I would be glad for any tips about a better setup.
what is the most efficient way to add a GIF/Video to the background of the landing screen ( home screen or first view controller) of my app in Xcode? i.e apps like spotify, uber, insagram etc. Being that my app is universal, how would i make it fit accordingly?
Do you mean the first screen that is displayed after your app is launched? If so: unfortunately you can't have dynamic content; you won't be able to use a gif/video.
That said, what you can do if you have some app-setup on background threads that will take some time anyway, or if you simply want the user to wait longer before interaction so that you can display the gif/video, you can make the static image match the first frame of the gif/video, and have your your entry point be a ViewController that displays the actual gif/video. Because this would delay the time to interaction, though, this would never be recommended.
As for making it fit: as of iOS 8 Apple recommends using LaunchScreen.xib. With it you can use Auto Layout to achieve universality.
To add a video you can use MPMoviePlayerController, AVPlayer, or if you're using SPritekit you can use an SKVideoNode.
EDIT (in response to follow-up comments):
An NSURL is a reference to a local or remote file. This link will give you a decent overview. Just copy the movie in and follow that guide.
In addition to the MPMoviePlayerController solution Saqib Omer suggested, here's an alternative method that uses a UIView with an AVPlayerLayer. It has a button on top of the video as an example, since that's what you're looking for.
import AVKit
import AVFoundation
import UIKit
class ViewController: UIViewController {
override func viewDidLoad() {
super.viewDidLoad()
// Start with a generic UIView and add it to the ViewController view
let myPlayerView = UIView(frame: self.view.bounds)
myPlayerView.backgroundColor = UIColor.blackColor()
view.addSubview(myPlayerView)
// Use a local or remote URL
let url = NSURL(string: "http://eoimages.gsfc.nasa.gov/images/imagerecords/76000/76740/iss030-e-6082_sdtv.mov") // See the note on NSURL above.
// Make a player
let myPlayer = AVPlayer(URL: url)
myPlayer.play()
// Make the AVPlayerLayer and add it to myPlayerView's layer
let avLayer = AVPlayerLayer(player: myPlayer)
avLayer.frame = myPlayerView.bounds
myPlayerView.layer.addSublayer(avLayer)
// Make a button and add it to myPlayerView (you'd need to add an action, of course)
let myButtonOrigin = CGPoint(x: myPlayerView.bounds.size.width / 3, y: myPlayerView.bounds.size.height / 2)
let myButtonSize = CGSize(width: myPlayerView.bounds.size.width / 3, height: myPlayerView.bounds.size.height / 10)
let myButton = UIButton(frame: CGRect(origin: myButtonOrigin, size: myButtonSize))
myButton.setTitle("Press Me!", forState: .Normal)
myButton.setTitleColor(UIColor.whiteColor(), forState: .Normal)
myPlayerView.addSubview(myButton)
}
}
For playing video add following code, declare class variable var moviePlayer:MPMoviePlayerController! . Than in your viewDidLoad()
var url:NSURL = NSURL(string: "YOUR_URL_FOR_VIDEO")
moviePlayer = MPMoviePlayerController(contentURL: url)
moviePlayer.view.frame = CGRect(x: 0, y: 0, width: 200, height: 150)
self.view.addSubview(moviePlayer.view)
moviePlayer.fullscreen = true
moviePlayer.controlStyle = MPMovieControlStyle.Embedded
This will play video. But to fit it you need to add layout contraints. See this link to add constraints pragmatically.
import MediaPlayer
class ViewController: UIViewController {
var moviePlayer: MPMoviePlayerController!
override func viewDidLoad() {
super.viewDidLoad()
// Load the video from the app bundle.
let videoURL: NSURL = NSBundle.mainBundle().URLForResource("video", withExtension: "mp4")!
// Create and configure the movie player.
self.moviePlayer = MPMoviePlayerController(contentURL: videoURL)
self.moviePlayer.controlStyle = MPMovieControlStyle.None
self.moviePlayer.scalingMode = MPMovieScalingMode.AspectFill
self.moviePlayer.view.frame = self.view.frame
self.view .insertSubview(self.moviePlayer.view, atIndex: 0)
self.moviePlayer.play()
// Loop video.
NSNotificationCenter.defaultCenter().addObserver(self, selector: "loopVideo", name: MPMoviePlayerPlaybackDidFinishNotification, object: self.moviePlayer)
}
func loopVideo() {
self.moviePlayer.play()
}
https://medium.com/#kschaller/ios-video-backgrounds-6eead788f190#.2fhxmc2da