Swift and AVFoundation: Playing audio while animating a label - ios

I have an app that I'm adding sounds to. It has a keypad and when the user taps a button, the number animates to show the user that their press went through.
However, since both are happening on the main thread, adding the following code below, the play() function causes a slight delay in the animation. If the user waits ~2 seconds and taps a keypad number again, they see the delay again. So as long as they're hitting keypad numbers under 2s, they don't see another lag.
I tried wrapping the code below in a DispatchQueue.main.async {} block with no luck.
if let sound = CashSound(rawValue: "buttonPressMelody\(count)"),
let url = Bundle.main.url(forResource: sound.rawValue, withExtension: sound.fileType) {
self.audioPlayer = try? AVAudioPlayer(contentsOf: url)
self.audioPlayer?.prepareToPlay()
self.audioPlayer?.play()
}
How can I play this audio and have the animation run without them interfering and with the audio coinciding with the press?
Thanks

I experienced a rather similar problem in my SwiftUI app, and in my case the solution was in proper resource loading / AVAudioPlayer initialization and preparing.
I use the following function to load audio from disk (inspired by ImageStore class from Apple's Landmarks SwiftUI Tutorial)
final class AudioStore {
typealias Resources = [String:AVAudioPlayer]
fileprivate var audios: Resources = [:]
static var shared = AudioStore()
func audio(with name: String) -> AVAudioPlayer {
let index = guaranteeAudio(for: name)
return audios.values[index]
}
static func loadAudio(with name: String) -> AVAudioPlayer {
guard let url = Bundle.main.url(forResource: name, withExtension: "mp3") else { fatalError("Couldn't find audio \(name).mp3 in main bundle.") }
do { return try AVAudioPlayer(contentsOf: url) }
catch { fatalError("Couldn't load audio \(name).mp3 from main bundle.") }
}
fileprivate func guaranteeAudio(for name: String) -> Resources.Index {
if let index = audios.index(forKey: name) { return index }
audios[name] = AudioStore.loadAudio(with: name)
return audios.index(forKey: name)!
}
}
In the view's init I initialize the player's instance by calling audio(with:) with proper resource name.
In onAppear() I call prepareToPlay() on view's already initialized player with proper optional unwrapping, and finally
I play audio when the gesture fires.
Also, in my case I needed to delay the actual playback by some 0.3 seconds, and for that I despatched it to the global queue. I should stress that the animation with the audio playback was smooth even without dispatching it to the background queue, so I concluded the key was in proper initialization and preparation. To delay the playback, however, you can only utilize the background thread, otherwise you will get the lag.
DispatchQueue.global(qos: .userInitiated).asyncAfter(deadline: .now() + 0.3) {
///your audio.play() analog here
}
Hope that will help some soul out there.

Related

How to interrupt device audio to play a sound and then let the device audio continue on iOS

I have an app where the user can play voice messages received from other users. Playing the voice messages should interrupt device audio (music, podcast, etc playing from other apps), play the voice messages and then let the device audio continue.
Here is a use specific use case I am trying to achieve
the user starts playing music on the device via Apple Music
the user opens the app and taps a voice message
the Apple Music stops
voice message in the app plays
Apple Music continues playing
With setting AVAudioSessions category to .ambient I can play the voice message "over" the playing Apple Music, but that is not what I need exactly.
If I use the .playback category that makes the Apple Music stop, plays the voice message in the app but Apple Music does not continue playing afterwards.
In theory, Apple has provided a "protocol" for interrupting and resuming background audio, and in a downloadable example, I show you what it is and prove that it works:
https://github.com/mattneub/Programming-iOS-Book-Examples/tree/master/bk2ch14p653backgroundPlayerAndInterrupter
In that example, there are two projects, representing two different apps. You run both of them simultaneously. BackgroundPlayer plays sound in the background; Interrupter interrupts it, pausing it, and when it is finished interrupting, BackgroundPlayer resumes.
This, as you will see, is done by having Interrupter change its audio session category from ambient to playback while interrupting, and changing it back when finished, along with first deactivating itself entirely while sending the .notifyOthersOnDeactivation signal:
func playFile(atPath path:String) {
self.player?.delegate = nil
self.player?.stop()
let fileURL = URL(fileURLWithPath: path)
guard let p = try? AVAudioPlayer(contentsOf: fileURL) else {return} // nicer
self.player = p
// error-checking omitted
// switch to playback category while playing, interrupt background audio
try? AVAudioSession.sharedInstance().setCategory(.playback, mode:.default)
try? AVAudioSession.sharedInstance().setActive(true)
self.player.prepareToPlay()
self.player.delegate = self
let ok = self.player.play()
print("interrupter trying to play \(path): \(ok)")
}
func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) { // *
let sess = AVAudioSession.sharedInstance()
// this is the key move
try? sess.setActive(false, options: .notifyOthersOnDeactivation)
// now go back to ambient
try? sess.setCategory(.ambient, mode:.default)
try? sess.setActive(true)
delegate?.soundFinished(self)
}
The trouble, however, is that response to .notifyOthersOnDeactivation is entirely dependent on the other app being well behaved. My other app, BackgroundPlayer, is well behaved. This is what it does:
self.observer = NotificationCenter.default.addObserver(forName:
AVAudioSession.interruptionNotification, object: nil, queue: nil) {
[weak self] n in
guard let self = self else { return } // legal in Swift 4.2
let why = n.userInfo![AVAudioSessionInterruptionTypeKey] as! UInt
let type = AVAudioSession.InterruptionType(rawValue: why)!
switch type {
case .began:
print("interruption began:\n\(n.userInfo!)")
case .ended:
print("interruption ended:\n\(n.userInfo!)")
guard let opt = n.userInfo![AVAudioSessionInterruptionOptionKey] as? UInt else {return}
let opts = AVAudioSession.InterruptionOptions(rawValue: opt)
if opts.contains(.shouldResume) {
print("should resume")
self.player.prepareToPlay()
let ok = self.player.play()
print("bp tried to resume play: did I? \(ok as Any)")
} else {
print("not should resume")
}
#unknown default:
fatalError()
}
}
As you can see, we register for interruption notifications, and if we are interrupted, we look for the .shouldResume option — which is the result of the interrupter setting the notifyOthersOnDeactivation in the first place.
So far, so good. But there's a snag. Some apps are not well behaved in this regard. And the most non-well-behaved is Apple's own Music app! Thus it is actually impossible to get the Music app to do what you want it to do. You are better off using ducking, where the system just adjusts the relative levels of the two apps for you, allowing the background app (Music) to continue playing but more quietly.
You've already discovered that you can interrupt other audio apps by activating a .playback category audio session. When you finish playing your audio and want the interrupted audio to continue, deactivate your audio session and pass the notifyOthersOnDeactivation option.
e.g.
try! audioSession.setActive(false, options: .notifyOthersOnDeactivation)
I think those apps that should continue like Apple Music, Spotify, Radio apps etc implement the functionality to handle interruptions and when another app's audio is deactivated / wants to hand back responsibility of the audio.
So could you try and see if this works
// I play the audio using this AVAudioPlayer
var player: AVAudioPlayer?
// Implement playing a sound
func playSound() {
// Local url for me, but you could configure as you need
guard let url = Bundle.main.url(forResource: "se_1",
withExtension: "wav")
else { return }
do {
// Set the category to playback to interrupt the current audio
try AVAudioSession.sharedInstance().setCategory(.playback,
mode: .default)
try AVAudioSession.sharedInstance().setActive(true)
player = try AVAudioPlayer(contentsOf: url)
// This is to know when the sound has ended
player?.delegate = self
player?.play()
} catch let error {
print(error.localizedDescription)
}
}
extension AVAudioInterruptVC: AVAudioPlayerDelegate {
func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer,
successfully flag: Bool) {
do {
// When the sound has ended, notify other apps
// You can do this where ever you want, I just show the
// example of doing it when the audio has ended
try AVAudioSession.sharedInstance()
.setActive(false, options: [.notifyOthersOnDeactivation])
}
catch {
print(error)
}
}
}

Playing and caching a remote asset with AVPlayer

I am trying to use AVPlayer to play/cache a remote asset using two tools on Github: CachingPlayerItem with Cache. I found the solution elsewhere(scroll down), which nearly gets me there, My issue now is that I have to tap twice on the remote audio asset (a hyperlink in Firebase) to get it to stream. For some mysterious reason, AVPlayer will not play the remote asset unless it is cached in my case. I am aware that I can directly stream the url using AVPlayerItem(url:) but that is not the solution I am seeking; the sample code for CachingPlayerItem say that should not be necessary.
In my tinkering, I think something is happening with the async operations that are performed when I call playerItem.delegate = self. Maybe I am misunderstanding how this asynchronous delegate operation is working... Any clarity and pointers would be appreciated.
import AVKit
import Cache
class AudioPlayer: AVPlayer, ObservableObject, AVAudioPlayerDelegate {
let diskConfig = DiskConfig(name: "DiskCache")
let memoryConfig = MemoryConfig(expiry: .never, countLimit: 10, totalCostLimit: 10)
lazy var storage: Cache.Storage<String, Data>? = {
return try? Cache.Storage(diskConfig: diskConfig, memoryConfig: memoryConfig, transformer: TransformerFactory.forData())
}()
/// Plays audio either from the network if it's not cached or from the cache.
func startPlayback(with url: URL) {
let playerItem: CachingPlayerItem
do {
let result = try storage!.entry(forKey: url.absoluteString)
// The video is cached.
playerItem = CachingPlayerItem(data: result.object, mimeType: "audio/mp4", fileExtension: "m4a")
} catch {
// The video is not cached.
playerItem = CachingPlayerItem(url: url)
}
playerItem.delegate = self // Seems to be the problematic line if the result is not cached.
self.replaceCurrentItem(with: playerItem) // This line is different from what you do. The behaviour doesnt change whether I have AVPlayer as private var.
self.automaticallyWaitsToMinimizeStalling = false
self.play()
}
}
extension AudioPlayer: CachingPlayerItemDelegate {
func playerItem(_ playerItem: CachingPlayerItem, didFinishDownloadingData data: Data) {
// Video is downloaded. Saving it to the cache asynchronously.
storage?.async.setObject(data, forKey: playerItem.url.absoluteString, completion: { _ in })
print("Caching done!")
}
}

Persistence of SCNAudioPlayer on SCNNodes in Xcode Instrument

I have created a subclass of SCNNode. It is made up of few child nodes.
I have declared a method, viz. soundCasual() which adds a SCNAudioPlayer to the instance of this class. Everything is working as expected and audio is being played, when this method is called. This method is called on that node whenever that node is tapped (gesture).
Code:
class MyNode: SCNNode {
let wrapperNode = SCNNode()
let audioSource5 = SCNAudioSource(fileNamed: "audiofile.mp3")
override init() {
super.init()
if let virtualScene = SCNScene(named: "MyNode.scn", inDirectory: "Assets.scnassets/Shapes/MyNode") {
for child in virtualScene.rootNode.childNodes {
wrapperNode.addChildNode(child)
}
}
}
func soundCasual() {
DispatchQueue.global(qos: .userInteractive).async { [weak self] in
if let audioSource = self?.audioSource5 {
let audioPlayer = SCNAudioPlayer(source: audioSource)
self?.wrapperNode.removeAllAudioPlayers()
self?.wrapperNode.addAudioPlayer(audioPlayer)
}
}
}
}
Issue within Instruments (Allocations)
When I analyse my whole codebase (which is several other things), I see that whenever I tap on that node, allocation count of SCNAudioPlayer is increased by one while profiling within Instruments. But all of the increase is persistent. By definition of SCNAudioPlayer, I assumed that this the player is removed after playback, which is why the increment should be in Transient allocations, but it is not working like this. That is why I tried removeAllAudioPlayers() before adding an SCNAudioPlayer to the node, as you can see in the code for soundCasual(). But the issue remains.
Till this snapshot was taken, I had tapped on that node about 17 times, and it also shows 17 against Persistent allocations for SCNAudioPlayer.
Note: SCNAudioSource is 10, as it should be, since there are 10 audio source I am using in the app
And this is happening for all other SCNNodes in my application without fail.
Kindly help as I am not able to understand what exactly am I missing.
EDIT
As per recommended, I changed my init() as
let path = Bundle.main.path(forResource: "Keemo", ofType: "scn", inDirectory: "Assets.scnassets/Shapes/Keemo")
if let path = path , let keemo = SCNReferenceNode(url: URL(fileURLWithPath: path)) {
keemo.load()
}
func soundPlay() {
DispatchQueue.global(qos: .userInteractive).async { [weak self] in
if let audioSource = self?.audioSourcePlay {
audioSource.volume = 0.1
let audioPlayer = SCNAudioPlayer(source: audioSource)
self?.removeAllAudioPlayers()
self?.addAudioPlayer(audioPlayer)
}
}
}
Despite, allocations in Instruments show audioPlayers as persistent. Though on checking node.audioPlayers it shows that at one point, there is only one audioPlayer node attached.
EDIT
This issue appears even in a simple case when I use the boilerplate codebase attached in a Scenekit app created by default in XCode. Hence, this issue has been raised as a bug to Apple. https://bugreport.apple.com/web/?problemID=43482539
WORKAROUND
I am using AVAudioPlayer instead of SCNAudioPlayer, not exactly the same thing, but at least memory this way will not cause a crash.
I am not familiar with SceneKit, but from my experience with UIKit and SpriteKit I suspect that your use of wrapperNode and virtualScene is messing with the garbage collector.
I would try removing wrapperNode and adding everything to self (since self is a SCNNode).
Which node is being used in your scene? self or wrapperNode? And with your sample code wrapperNode is not added to self so it may or may not actually be part of the scene.
Also, you should probably be using SCNReferenceNode instead of the virtual scene thing you're using.
!!! this code has not been tested !!!
class MyNode: SCNReferenceNode {
let audioSource5 = SCNAudioSource(fileNamed: "audiofile.mp3")
func soundCasual() {
DispatchQueue.global(qos: .userInteractive).async { [weak self] in
if let audioSource = self?.audioSource5 {
let audioPlayer = SCNAudioPlayer(source: audioSource)
self?.removeAllAudioPlayers()
self?.addAudioPlayer(audioPlayer)
}
}
}
}
// If you programmatically create this node, you'll have to call .load() on it
let referenceNode = SCNReferenceNode(URL: referenceURL)
referenceNode.load()
HtH!
If you haven't found the answer already, you need to remove the SCNAudioPlayer from the node once it has completed playing:
if let audioSource = self?.audioSourcePlay {
audioSource.volume = 0.1
let audioPlayer = SCNAudioPlayer(source: audioSource)
self?.removeAllAudioPlayers()
self?.addAudioPlayer(audioPlayer)
self?.audioplayer.didFinishPlayback = {
self?.removeAudioPlayer(audioPlayer)
}
}

Music playing over the song when i return to the original VC

I am using xcode 9 and swift 4 for my app. In my app i have music playing in the viewDidLoad. When i exit the view controller to go to another View, it continues to play like it should. How ever, when i return to that view controller the song starts to play again. This song is overlapping the song that first loaded. Do you guys have any ideas on how to stop this from happening?
do
{
let audioPath = Bundle.main.path(forResource: "APP4", ofType: "mp3")
try player = AVAudioPlayer(contentsOf: NSURL(fileURLWithPath: audioPath!) as URL)
}
catch
{
//catch error
}
let session = AVAudioSession.sharedInstance()
do
{
try session.setCategory(AVAudioSessionCategoryPlayback)
}
catch
{
}
player.numberOfLoops = -1
player.play()
It starts playing again, because your viewDidLoad is called again, which asks it to play it again. A simplest fix would be to keep a static bool variable to keep track if you have already made this call.
static var isMusicPlaying: Bool = false
In your viewDidLoad, you can put code before the code that calls the play.
guard !isMusicPlaying else {
return
}
isMusicPlaying = true

How to stream audio for only a known duration using swift

I'm using AVPlayer (I don't need to, but I wanna stream it and start playing as soon as possible) to play an m4a file (it's an iTunes audio preview). Only I only want it to play a part of that file.
I'm able to set a start time but not an end time.
Using a timer is not working because I'm using URL as a http address. I'm playing as it loads, without downloading the file.
I also saw solutions in Objective-C to use KVO to know when music starts playing but I'm thinking this is not the best approach since I'm using swift and also because of glitches that may occur so the song will not stop at the right moment.
You can add a addBoundaryTimeObserverForTimes to your AVPlayer as follow:
update: Xcode 8.3.2 • Swift 3.1
import UIKit
import AVFoundation
class ViewController: UIViewController {
var player: AVPlayer!
var observer: Any!
override func viewDidLoad() {
super.viewDidLoad()
guard let url = URL(string: "https://www.example.com/audio.mp3") else { return }
player = AVPlayer(url: url)
let boundary: TimeInterval = 30
let times = [NSValue(time: CMTimeMake(Int64(boundary), 1))]
observer = player.addBoundaryTimeObserver(forTimes: times, queue: nil) {
[weak self] time in
print("30s reached")
if let observer = self?.observer {
self?.player.removeTimeObserver(observer)
}
self?.player.pause()
}
player.play()
print("started loading")
}
}

Resources