I use SCNRenderer with Vuforia. On iOS 13, video content not showing, sound is heard. I tried to set the video material in the following ways:
material.diffuse.contents = AVPlayer
or
material.diffuse.contents = SKVideoNode
But none of these methods work on iOS 13. Unfortunately, I don’t have the opportunity to use SCNView to render a SCNScene, since it is impossible to use with Vuforia. Maybe someone has already encountered this? Or does someone know how to otherwise render the AVPlayer in the SCNScene?
Root Cause:
In my app, we have multiple types of AR modes.
When we switch between these modes with the UISegmentedControls, I have to “reset” the scene by clearing any AR nodes that should not be there. (e.g: previous mode's AR nodes not interfering with the new nodes)
I did this with the following code:
func resetARScene() {
self.node.geometry?.firstMaterial?.normal.contents = nil
self.node.geometry?.firstMaterial?.diffuse.contents = nil
self.node.removeFromParentNode()
self.node = SCNNode()
DispatchQueue.main.async {
self.sceneView.scene.rootNode.addChildNode(self.node)
}
}
The issue arises when in iOS 12, we could re-add the childNode back to the sceneView. Somehow in iOS 13 it doesn’t add properly.
Solution:
func resetARScene() {
self.node.enumerateChildNodes { (existingNode, _) in
existingNode.removeFromParentNode()
}
}
This function achieves the same purpose without removing the node from the sceneView and the video plays. I hope this helps, although I am not sure if your circumstances are the same as mine.
Related
I've just started out learning Swift in Xcode and am creating a simple camera App to get up and running. I have a button that switches between the front and back facing cameras working but want to add the option to also switch between the Tele and the Ultra Wide lens on the iPhone 11 Pro.
I have created the functions to run a new CaptureSession for each lens (if detected) but was just wondering how I can call these functions to UIbutton function
The thing that's got me scratching my head is the if statement used to switch between the front and back camera says if input.device.position == .back {
This only specificities if it's front or back, not the lens itself. What would be an efficient way to make the button then Change from the front to wide, the Tele, the Ultra Wide and back to the front each time the button is pressed?
Apologies for any misuse of terminology, I'm very new to coding in Swift. Thank you!
{
guard let CurrentCameraInput: AVCaptureInput = CaptureSession?.inputs.first else {
return
}
if let input = CurrentCameraInput as? AVCaptureDeviceInput
{
if input.device.position == .back {
SwitchToFrontCamera() }
if input.device.position == .front {
SwitchToBackCamera()
}
}
}
Welcome!
Check you this initializer for AVCaptureDevice. You can specify the DeviceType you want to use, like .builtInUltraWideCamera or .builtInTelephotoCamera.
You can use a AVCaptureDevice.DiscoverySession to get a list of all capture devices available to your app.
I am experimenting with other sound effect options for our game after determining SKAction.playSoundFileNamed is not only buggy and leaky but also creates a crash specifically during IAP interruptions (not OK, Apple). We are attempting to use SKAudioNodes, but the issue is we have overlapping sound effects. We would like to use a system whereby we have a queue of SKAudioNodes that exist with the identical effect: e.g. audioNode1, audioNode2, audioNode3, all with sound "fxmatch," and if audioNode1.isPlaying(), then we will move on in the queue to audioNode2, then audioNode3, etc. We have similar syntax already for AVAudioPlayer, where we check isPlaying(). Is there some method equivalent for SKAudioNodes?
if let playerNode = audioNode.avAudioNode as? AVAudioPlayerNode{
print(playerNode.isPlaying)
}
Or
extension SKAudioNode
{
var isPlaying : Bool { return (avAudioNode as? AVAudioPlayerNode)?.isPlaying ?? false }
}
...
print(audioNode.isPlaying)
I have a problem in my SpriteKit game where audio using playSoundFileNamed(_ soundFile:, waitForCompletion:) will not play after the app is interrupted by a phone call. (I also use SKAudioNodes in my app which aren't affected but I really really really want to be able to use the SKAction playSoundFileNamed as well.)
Here's the gameScene.swift file from a stripped down SpriteKit game template which reproduces the problem. You just need to add an audio file to the project and call it "note"
I've attached the code that should reside in appDelegate to a toggle on/off button to simulate the phone call interruption. That code 1) Stops AudioEngine then deactivates AVAudioSession - (normally in applicationWillResignActive) ... and 2) Activates AVAudioSession then Starts AudioEngine - (normally in applicationDidBecomeActive)
The error:
AVAudioSession.mm:1079:-[AVAudioSession setActive:withOptions:error:]: Deactivating an audio session that has running I/O. All I/O should be stopped or paused prior to deactivating the audio session.
This occurs when attempting to deactivate the audio session but only after a sound has been played at least once.
to reproduce:
1) Run the app
2) toggle the engine off and on a few times. No error will occur.
3) Tap the playSoundFileNamed button 1 or more times to play the sound.
4) Wait for sound to stop
5) Wait some more to be sure
6) Tap Toggle Audio Engine button to stop the audioEngine and deactivate session -
the error occurs.
7) Toggle the engine on and of a few times to see session activated, session deactivated, session activated printed in debug area - i.e. no errors reported.
8) Now with session active and engine running, playSoundFileNamed button will not play the sound anymore.
What am I doing wrong?
import SpriteKit
import AVFoundation
class GameScene: SKScene {
var toggleAudioButton: SKLabelNode?
var playSoundFileButton: SKLabelNode?
var engineIsRunning = true
override func didMove(to view: SKView) {
toggleAudioButton = SKLabelNode(text: "toggle Audio Engine")
toggleAudioButton?.position = CGPoint(x:20, y:100)
toggleAudioButton?.name = "toggleAudioEngine"
toggleAudioButton?.fontSize = 80
addChild(toggleAudioButton!)
playSoundFileButton = SKLabelNode(text: "playSoundFileNamed")
playSoundFileButton?.position = CGPoint(x: (toggleAudioButton?.frame.midX)!, y: (toggleAudioButton?.frame.midY)!-240)
playSoundFileButton?.name = "playSoundFileNamed"
playSoundFileButton?.fontSize = 80
addChild(playSoundFileButton!)
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
if let touch = touches.first {
let location = touch.location(in: self)
let nodes = self.nodes(at: location)
for spriteNode in nodes {
if spriteNode.name == "toggleAudioEngine" {
if engineIsRunning { // 1 stop engine, 2 deactivate session
scene?.audioEngine.stop() // 1
toggleAudioButton!.text = "engine is paused"
engineIsRunning = !engineIsRunning
do{
// this is the line that fails when hit anytime after the playSoundFileButton has played a sound
try AVAudioSession.sharedInstance().setActive(false) // 2
print("session deactivated")
}
catch{
print("DEACTIVATE SESSION FAILED")
}
}
else { // 1 activate session/ 2 start engine
do{
try AVAudioSession.sharedInstance().setActive(true) // 1
print("session activated")
}
catch{
print("couldn't setActive = true")
}
do {
try scene?.audioEngine.start() // 2
toggleAudioButton!.text = "engine is running"
engineIsRunning = !engineIsRunning
}
catch {
//
}
}
}
if spriteNode.name == "playSoundFileNamed" {
self.run(SKAction.playSoundFileNamed("note", waitForCompletion: false))
}
}
}
}
}
Let me save you some time here: playSoundFileNamed sounds wonderful in theory, so wonderful that you might say use it in an app you spent 4 years developing until one day you realize it’s not just totally broken on interruptions but will even crash your app in the most critical of interruptions, your IAP. Don’t do it. I’m still not entirely sure whether SKAudioNode or AVPlayer is the answer, but it may depend on your use case. Just don’t do it.
If you need scientific evidence, create an app and create a for loop that playSoundFileNamed whatever you want in touchesBegan, and see what happens to your memory usage. The method is a leaky piece of garbage.
EDITED FOR OUR FINAL SOLUTION:
We found having a proper number of preloaded instances of AVAudioPlayer in memory with prepareToPlay() was the best method. The SwiftySound audio class uses an on-the-fly generator, but making AVAudioPlayers on the fly created slowdown in animation. We found having a max number of AVAudioPlayers and checking an array for those where isPlaying == false was simplest and best; if one isn't available you don't get sound, similar to what you likely saw with PSFN if you had it playing lots of sounds on top of each other. Overall, we have not found an ideal solution, but this was close for us.
In response to Mike Pandolfini’s advice not to use playSoundFileNamed I’ve converted my code to only use SKAudioNodes.
(and sent the bug report to apple).
I then found that some of these SKAudioNodes don’t play after app interruption either … and I’ve stumbled across a fix.
You need to tell each SKAudioNode to stop() as the app resigns to, or returns from the background - even if they’re not playing.
(I'm now not using any of the code in my first post which stops the audio engine and deactivates the session)
The problem then became how to play the same sound rapidly where it possibly plays over itself. That was what was so good about playSoundFileNamed.
1) The SKAudioNode fix:
Preload your SKAudioNodes i.e.
let sound = SKAudioNode(fileNamed: "super-20")
In didMoveToView add them
sound.autoplayLooped = false
addChild(sound)
Add a willResignActive notification
notificationCenter.addObserver(self, selector:#selector(willResignActive), name:UIApplication.willResignActiveNotification, object: nil)
Then create the selector’s function which stops all audioNodes playing:
#objc func willResignActive() {
for node in self.children {
if NSStringFromClass(type(of: node)) == “SKAudioNode" {
node.run(SKAction.stop())
}
}
}
All SKAudioNodes now play reliably after app interrupt.
2) To replicate playSoundFileNamed’s ability to play the short rapid repeating sounds or longer sounds that may need to play more than once and therefore could overlap, create/preload more than 1 property for each sound and use them like this:
let sound1 = SKAudioNode(fileNamed: "super-20")
let sound2 = SKAudioNode(fileNamed: "super-20")
let sound3 = SKAudioNode(fileNamed: "super-20")
let sound4 = SKAudioNode(fileNamed: "super-20")
var soundArray: [SKAudioNode] = []
var soundCounter: Int = 0
in didMoveToView
soundArray = [sound1, sound2, sound3, sound4]
for sound in soundArray {
sound.autoplayLooped = false
addChild(sound)
}
Create a play function
func playFastSound(from array:[SKAudioNode], with counter:inout Int) {
counter += 1
if counter > array.count-1 {
counter = 0
}
array[counter].run(SKAction.play())
}
To play a sound pass that particular sound's array and its counter to the play function.
playFastSound(from: soundArray, with: &soundCounter)
I have project that is rendering video playback and applying CIFilters to it. I know that I can use video composition to get video with filters, but problems is that filters needs to be swipeable (with preview of next filter so we're using mask for 1st imageview and filtering 2nd one with next filter).
func displayLinkDidRefresh(link: CADisplayLink){
let itemTime = videoOutput.itemTime(forHostTime: CACurrentMediaTime())
if videoOutput.hasNewPixelBuffer(forItemTime: itemTime) {
if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: nil){
unfilteredImage = CIImage(cvImageBuffer: pixelBuffer)
displayFilteredImage(unfilteredImage: unfilteredImage)
}
}
}
This is the code used to create AVPlayer instance and CADisplayLink:
player = AVPlayer(playerItem: item)
player.isMuted = true
displayLink = CADisplayLink(target: self, selector: #selector(displayLinkDidRefresh(link:)))
displayLink!.preferredFramesPerSecond = 24
displayLink!.add(to: RunLoop.main, forMode: RunLoopMode.commonModes)
NotificationCenter.default.addObserver(self, selector: #selector(self.playerItemDidReachEnd(notification:)), name: NSNotification.Name.AVPlayerItemDidPlayToEndTime, object: self.player.currentItem)
When running from debugger I experience no stutter (or it is at minimum level), but when running build from archive it is stuttering a lot. What I do to test is deleting app from phone and then instal it on the phone and test, as said debug builds are fine, but archive are creating issues. Any input on this?
EDIT 1:
Managed to get it to work better, but still stuttering after attaching video composition to player item when item and player status is readyToPlay.
After some time playing with this I found the issue, nothing is wrong with the code. Issue was that we have Appsee analytics inside app and we needed to pause it on these screens in order for them to render properly. Reason why we didn't experience these issues in debug mode was the AppDelegate if condition that was preventing Appsee to work in debug environment.
Edited to explain why its not a duplicate:
I am trying to use MPMusicPlayerController.systemMusicPlayer() to control music playback from my app, but I want to disable the Command Center previous track button. I also want to override the default Command Center play and next track functions. The code should be simple:
This code is in ViewController.swift - viewDidLoad
let commandCenter = MPRemoteCommandCenter.sharedCommandCenter()
commandCenter.previousTrackCommand.enabled = false
commandCenter.previousTrackCommand.addTargetWithHandler({ (commandEvent: MPRemoteCommandEvent!) -> MPRemoteCommandHandlerStatus in
self.empty()
return MPRemoteCommandHandlerStatus.Success
})
//MPRemoteCommandCenter.sharedCommandCenter().previousTrackCommand.addTarget(self, action: "empty")
commandCenter.nextTrackCommand.enabled = true
commandCenter.nextTrackCommand.addTargetWithHandler { (commandEvent: MPRemoteCommandEvent!) -> MPRemoteCommandHandlerStatus in
self.gameOver()
return MPRemoteCommandHandlerStatus.Success
}
commandCenter.playCommand.enabled = true
commandCenter.playCommand.addTargetWithHandler { (commandEvent: MPRemoteCommandEvent!) -> MPRemoteCommandHandlerStatus in
self.playing()
return MPRemoteCommandHandlerStatus.Success
}
Also, in AppDelegate.swift - application
UIApplication.sharedApplication().beginReceivingRemoteControlEvents()
And in the iOS simulator (both iPad and iPhone), it works correctly, as can be seen in the first screenshot (of the simulator).
However, when deploying the app to my iPad, none of the MPRemoteCommandCenter commands work at all, as can be seen in the second screenshot (of an actual device).
This is different from the "dupliate" question (How Do I Get Audio Controls on Lock Screen/Control Center from AVAudioPlayer in Swift) in the following ways:
I am not using AVAudioSession, I am using MPMusicPlayerController.systemMusicPlayer()
I have already called beginReceivingRemoteControlEvents, so that cant be the issue (unless I have somehow called it incorrectly, in which case, I would love an answer explaining how else it should be called).
Thank you.
,