Best way to make an asynchronous call effectively synchronous in iOS 10 - ios

To explain my situation a little better I'm trying to make an app which will play a ping noise when a button is pressed and then proceed to record and transcribe the user's voice immediately after.
For the ping sound I'm using System Sound Services, to record the audio I'm using AudioToolbox, and to transcribe it I'm using Speech kit.
I believe the crux of my problem lies in the timing of the asynchronous System sound services play function:
//Button pressed function
let audiosession = AVAudioSession.sharedInstance()
let filename = "Ping"
let ext = "wav"
if let soundUrl = Bundle.main.url(forResource: filename, withExtension: ext){
var soundId: SystemSoundID = 0
AudioServicesCreateSystemSoundID(soundUrl as CFURL, &soundId)
AudioServicesAddSystemSoundCompletion(soundId, nil, nil, {(soundid,_) -> Void in
AudioServicesDisposeSystemSoundID(soundid)
print("Sound played!")}, nil)
AudioServicesPlaySystemSound(soundId)
}
do{
try audiosession.setCategory(AVAudioSessionCategoryRecord)
try audiosession.setMode(AVAudioSessionModeMeasurement)
try audiosession.setActive(true, with: .notifyOthersOnDeactivation)
print("Changing modes!")
}catch{
print("error with audio session")
}
recognitionRequest = SFSpeechAudioBufferRecognitionRequest()
guard let inputNode = audioEngine.inputNode else{
fatalError("Audio engine has no input node!")
}
guard let recognitionRequest = recognitionRequest else{
fatalError("Unable to create a speech audio buffer recognition request object")
}
recognitionRequest.shouldReportPartialResults = true
recognitionTask = speechRecognizer?.recognitionTask(with: recognitionRequest, delegate: self)
let recordingFormat = inputNode.outputFormat(forBus: 0)
inputNode.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat) { (buffer, when) in
self.recognitionRequest?.append(buffer)
}
audioEngine.prepare()
do{
try audioEngine.start()
delegate?.didStartRecording()
}catch{
print("audioEngine couldn't start because of an error")
}
What happens when I run this code is that it records the voice and transcribes it successfully. However the ping is never played. The two(non-error) print statements I have in there fire in the order:
Changing modes!
Sound played!
So to my understanding, the reason the ping sound isn't being played is because by the time it actually completes I've already changed the audio session category from playback to record. Just to verify this is true, I tried removing everything but the sound services ping and it plays the sound as expected.
So my question is what is the best way to bypass the asynchronous nature of the AudioServicesPlaySystemSound call? I've experimented with trying to pass self into the completion function so I could have it trigger a function in my class which then runs the recording chunk. However I haven't been able to figure out how one actually goes about converting self to an UnsafeMutableRawPointer so it can be passed as clientData. Furthermore, even if I DID know how to do that, I'm not sure if it's even a good idea or the intended use of that parameter.
Alternatively, I could probably solve this problem by relying on something like notification center. But once again that just seems like a very clunky way of solving the problem that I'm going to end up regretting later.
Does anyone know what the correct way to handle this type of situation is?
Update:
As per Gruntcake's request, here is my attempt to access self in the completion block.
First I create a userData constant which is an UnsafeMutableRawPointer to self:
var me = self
let userData = withUnsafePointer(to: &me) { ptr in
return unsafeBitCast(ptr, to: UnsafeMutableRawPointer.self)
Next I use that constant in my callback block, and attempt to access self from it:
AudioServicesAddSystemSoundCompletion(soundId, nil, nil, {(sounded,me) -> Void in
AudioServicesDisposeSystemSoundID(sounded)
let myself = Unmanaged<myclassname>.fromOpaque(me!).takeRetainedValue()
myself.doOtherStuff()
print("Sound played!")}, userData)

Your attempt to call doOtherStuff() in the completion block is a correct approach (the only other one is notifications, those are the only two options)
What is complicating it in this case is the bridging from Obj-C to Swift that is necessary. Code to do that is:
let myData = unsafeBitCast(self, UnsafeMutablePointer<Void>.self)
AudioServicesAddSystemSoundCompletion(YOUR_SOUND_ID, CFRunLoopGetMain(), kCFRunLoopDefaultMode,{ (mSound, mVoid) in
let me = unsafeBitCast(mVoid, YOURCURRENTCLASS.self)
//me it is your current object so if yo have a variable like
// var someVar you can do
print(me.someVar)
}, myData)
Credit: This code was taken from an answer to this question, though it is not the accepted answer:
How do I implement AudioServicesSystemSoundCompletionProc in Swift?

Related

Add some delay between audio processing and sound to user

It's hard to explain, but I will try to do it...
Is that possible to add some delay between audio processing and sound from speakers to user?
I'm use the Speech framework to recognise an audio file (using SFSpeechAudioBufferRecognitionRequest and AVAudioEngine to stream an audio file) and get the final result from recogniser after user already heard a sound from speakers.
But a sound should appears after I get some result from recogniser. Need to add some latency between input to recogniser from my AVAudioMixerNode and sound to user.
if (self.recognizer?.isAvailable)! {
let recordingFormat = self.node.outputFormat(forBus: 0)
self.node.installTap(onBus: 0, bufferSize: 2048, format: recordingFormat) { buffer, _ in
self.request?.append(buffer)
}
self.task = self.recognizer.recognitionTask(with: self.request!, resultHandler: { [weak self] result, error in
guard let self = self, let result = result else { return }
print("Result: \(result.bestTranscription.formattedString)") // Need to print the result before user heard a sound from speakers
})
}

AVPlayer / AVAudioPlayer - Play audio fileURL from iCloud Drive (work on simulator but not iPhone) + (OSStatus error -54.)

I'm making Audio Player.
Importing file from iCloud Drive using .fileImporter.
I get file URL that looks like this: file:///private/var/mobile/Library/Mobile%20Documents/com~apple~CloudDocs/_Storage/Audio-books/%D0%91%D1%80%D0%B5%D0%BD%D0%B4%D1%8F%D1%82%D0%B8%D0%BD%D0%B0/Audiobook.mp3"
Then I pass it to audio player (tried AVPlayer and AVAudioPlayer). Both works on iOS simulator.
On the device after import I get error: The operation couldn’t be completed. (OSStatus error -54.)
I know it's possible, app called Evermusic does quite the same with on device files.
Is there permissions I need to be granted to play audio that stored on device?
How can I access Container for com~apple~CloudDocs?
Thank you very much for help, any suggestions greatly appreciated, I'm seriously stuck.
For future references repo of the project: https://github.com/yaosamo/AudioPlayer
You need to be using startAccessingSecurityScopedResource in order to get read access to those files. See documentation:
https://developer.apple.com/documentation/foundation/nsurl/1417051-startaccessingsecurityscopedreso
https://developer.apple.com/documentation/corefoundation/1543318-cfurlstartaccessingsecurityscope
In addition to #jnpdx answer want to add some details, and my solution example.
Few core things
✅ for my app if you need to access secured audio you need to use startAccessingSecurityScopedResource()
❌ you can't simple store URL and use it later, in fact you don't store fileURL at all. You need to use what's called bookmarkData() on your URL and store it. So later you can restore URL
✅ Watch Apple pres here
Here's how I import file:
.fileImporter(isPresented: $presentImporter, allowedContentTypes: [.mp3]) { result in
switch result {
case .success(let url):
// Start accessing secured url
let StartAccess = url.startAccessingSecurityScopedResource()
defer {
// Must stop accessing once stop using
if StartAccess {
url.stopAccessingSecurityScopedResource()
}
}
// Creating new book
let newBook = Book(context: viewContext)
let _ = print("---- Access Granted?", url.startAccessingSecurityScopedResource())
// Getting bookmarkData of the URL
let bookmarkData = try? url.bookmarkData()
newBook.name = "\(url.lastPathComponent)"
// Save bookmarkURL into CoreData
newBook.urldata = bookmarkData
// Specifiying parent item in CoreData
newBook.origin = playlist.self
try? viewContext.save()
case .failure(let error):
print(error)
}
}
Player retrieving URL:
func Audioplayer(bookmarkData: Data) {
// Restore security scoped bookmark
var bookmarkDataIsStale = false
let playNow = try? URL(resolvingBookmarkData: bookmarkData, bookmarkDataIsStale: &bookmarkDataIsStale)
do {
player = try AVAudioPlayer(contentsOf: playNow!)
// Delegate listen when audio is finished
player?.delegate = del
NotificationCenter.default.addObserver(forName: NSNotification.Name("ended"), object: nil, queue: .main) { (_) in
player?.stop()
ended = true
let _ = print("---- Book has ended ----")
}
} catch let error {
print("Player Error", error.localizedDescription)
}
player?.prepareToPlay()
player?.play()
}
Thank you and again here's repo on git.

AKPlayer crashes when playing from buffer on channelCount condition

I struggle to make the following scenario work as expected (code will be provided below).
Record my microphone input and store an AVAudioPCMBuffer in memory, this is done with AVAudioPCMBuffer extension method copy(from buffer: AVAudioPCMBuffer, readOffset: AVAudioFrameCount = default, frames: AVAudioFrameCount = default). I indeed get the buffer at the end of my recording.
When record is ended pass the buffer to AKPlayer and play. Here is a code snippet to demonstrate what I do (I know it is no the full app code, if needed I can share it):
.
private var player: AKPlayer = AKPlayer()
self.player.buffering = .always
// in the record complete callbak:
self.player.buffer = self.bufferRecorder?.pcmBuffer
self.player.volume = 1
self.player.play()
please note that the plater is connected to a mixer which is eventually connected to the AudioKit output.
when I inspect and debug the application I could see the buffer is with the correct length, and all my output/input setup uses the same processing format (sample rate, channels, bitrate etc) as well as the buffer recorded, but still my app crashes on this line:
2018-10-28 08:40:32.625001+0200 BeatmanApp[71037:6731884] [avae] AVAEInternal.h:70:_AVAE_Check:
required condition is false: [AVAudioPlayerNode.mm:665:ScheduleBuffer: (_outputFormat.channelCount == buffer.format.channelCount)]
when I debug and walk through the AudioKit code I can see that the breaking line is on AKPlayer+Playback.swift on line 162 on the method: playerNode.scheduleBuffer
more information that could be helpful:
the buffer recorded is 16 seconds long.
when I tried to pass the buffer straight to the player node in the tap method it seems as it worked, I did hear a delay from mic to speaker but it indeed played back.
I tried call prepare on the player before play method invoked, no help
thanks!
Ok, this was super uncool debugging session. I had to investigate the AVAudioEngine and how this kind of scenario could be done there, which of course not the final result I was looking. This quest helped me to understand how to solve it with AudioKit (half of my app is implemented using AudioKit's tools so it doesn't make sense to rewrite it with AVFoundation).
AFFoundation solution:
private let engine = AVAudioEngine()
private let bufferSize = 1024
private let p: AVAudioPlayerNode = AVAudioPlayerNode()
let audioSession = AVAudioSession.sharedInstance()
do {
try audioSession.setCategory(.playAndRecord, mode: .default, options: .defaultToSpeaker)
} catch {
print("Setting category to AVAudioSessionCategoryPlayback failed.")
}
let inputNode = self.engine.inputNode
engine.connect(inputNode, to: engine.mainMixerNode, format: inputNode.inputFormat(forBus: 0))
// !!! the following lines are the key to the solution.
// !!! the player has to be attached to the engine before actually connected
engine.attach(p)
engine.connect(p, to: engine.mainMixerNode, format: inputNode.inputFormat(forBus: 0))
do {
try engine.start()
} catch {
print("could not start engine \(error.localizedDescription)")
}
recordBufferAndPlay(duration: 4)
recordBufferAndPlay function:
func recordBufferAndPlay(duration: Double){
let inputNode = self.engine.inputNode
let total: Double = AVAudioSession.sharedInstance().sampleRate * duration
let totalBufferSize: UInt32 = UInt32(total)
let recordedBuffer : AVAudioPCMBuffer! = AVAudioPCMBuffer(pcmFormat: inputNode.inputFormat(forBus: 0), frameCapacity: totalBufferSize)
var alreadyRecorded = 0
inputNode.installTap(onBus: 0, bufferSize: 256, format: inputNode.inputFormat(forBus: 0)) {
(buffer: AVAudioPCMBuffer!, time: AVAudioTime!) -> Void in
recordedBuffer.copy(from: buffer) // this helper function is taken from audio kit!
alreadyRecorded = alreadyRecorded + Int(buffer.frameLength)
print(alreadyRecorded, totalBufferSize)
if(alreadyRecorded >= totalBufferSize){
inputNode.removeTap(onBus: 0)
self.p.scheduleBuffer(recordedBuffer, at: nil, options: .loops, completionHandler: {
print("completed playing")
})
self.p.play()
}
}
}
AudioKit solution:
So in the AudioKit solution these line should be invoked on your AKPlayer object. Note that this should be done before you actually start your engine.
self.player.buffering = .always
AudioKit.engine.attach(self.player.playerNode)
AudioKit.engine.connect(self.player.playerNode, to: self.mixer.inputNode, format: AudioKit.engine.inputNode.outputFormat(forBus: 0))
than the record is done pretty similarly to how you would have done it in AVAudioEngine, you install a tap on your node (microphone or other node) and record the buffer of PCM samples.

Playing scheduled audio in the background

I am having a really difficult time with playing audio in the background of my app. The app is a timer that is counting down and plays bells, and everything worked using the timer originally. Since you cannot run a timer over 3 minutes in the background, I need to play the bells another way.
The user has the ability to choose bells and set the time for these bells to play (e.g. play bell immediately, after 5 minutes, repeat another bell every 10 minutes, etc).
So far I have tried using notifications using DispatchQueue.main and this will work fine if the user does not pause the timer. If they re-enter the app though and pause, I cannot seem to cancel this queue or pause it in anyway.
Next I tried using AVAudioEngine, and created a set of nodes. These will play while the app is in the foreground but seem to stop upon backgrounding. Additionally when I pause the engine and resume later, it won't pause the sequence properly. It will squish the bells into playing one after the other or not at all.
If anyone has any ideas of how to solve my issue that would be great. Technically I could try remove everything from the engine and recreate it from the paused time when the user pauses/resumes, but this seems quite costly. It also doesn't solve the problem of the audio stopping in the background. I have the required background mode 'App plays audio or streams audio/video using Airplay', and it is also checked under the background modes in capabilities.
Below is a sample of how I tried to set up the audio engine. The registerAndPlaySound method is called several other times to create the chain of nodes (or is this done incorrectly?). The code is kinda messy at the moment because I have been trying many ways trying to get this to work.
func setupSounds{
if (attached){
engine.detach(player)
}
engine.attach(player)
attached = true
let mixer = engine.mainMixerNode
engine.connect(player, to: mixer, format: mixer.outputFormat(forBus: 0))
var bell = ""
do {
try engine.start()
} catch {
return
}
if (currentSession.bellObject?.startBell != nil){
bell = (currentSession.bellObject?.startBell)!
guard let url = Bundle.main.url(forResource: bell, withExtension: "mp3") else {
return
}
registerAndPlaySound(url: url, delay: warmUpTime)
}
}
func registerAndPlaySound(url: URL, delay: Double) {
do {
let file = try AVAudioFile(forReading: url)
let format = file.processingFormat
let capacity = file.length
let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(capacity))
do {
try file.read(into: buffer)
}catch {
return
}
let sampleRate = buffer.format.sampleRate
let sampleTime = sampleRate*delay
let futureTime = AVAudioTime(sampleTime: AVAudioFramePosition(sampleTime), atRate: sampleRate)
player.scheduleBuffer(buffer, at: futureTime, options: AVAudioPlayerNodeBufferOptions(rawValue: 0), completionHandler: nil)
player.play()
} catch {
return
}
}

Audible glitches on buffer playback via AVAudioPlayerNode in iOS (Swift) *working in simulator, but not on device

When using an AVAudioPlayerNode to schedule a short buffer to play immediately on a touch event ("Touch Up Inside"), I've noticed audible glitches / artifacts on playback while testing. The audio does not glitch at all in iOS simulator, however there is audible distortion on playback when I run the app on an actual iOS device. The audible distortion occurs randomly (the triggered sound will sometimes sound great, while other times it sounds distorted)
I've tried using different audio files, file formats, and preparing the buffer for playback using the prepareWithFrameCount method, but unfortunately the result is always the same and I'm stuck wondering what could be going wrong..
I've stripped the code down to globals for clarity and simplicity. Any help or insight would be greatly appreciated. This is my first attempt at developing an iOS app and my first question posted on Stack Overflow.
let filePath = NSBundle.mainBundle().pathForResource("BD_withSilence", ofType: "caf")!
let fileURL: NSURL = NSURL(fileURLWithPath: filePath)!
var error: NSError?
let file = AVAudioFile(forReading: fileURL, error: &error)
let fileFormat = file.processingFormat
let frameCount = UInt32(file.length)
let buffer = AVAudioPCMBuffer(PCMFormat: fileFormat, frameCapacity: frameCount)
let audioEngine = AVAudioEngine()
let playerNode = AVAudioPlayerNode()
func startEngine() {
var error: NSError?
file.readIntoBuffer(buffer, error: &error)
audioEngine.attachNode(playerNode)
audioEngine.connect(playerNode, to: audioEngine.mainMixerNode, format: buffer.format)
audioEngine.prepare()
func start() {
var error: NSError?
audioEngine.startAndReturnError(&error)
}
start()
}
startEngine()
let frameCapacity = AVAudioFramePosition(buffer.frameCapacity)
let frameLength = buffer.frameLength
let sampleRate: Double = 44100.0
func play() {
func scheduleBuffer() {
playerNode.scheduleBuffer(buffer, atTime: nil, options: AVAudioPlayerNodeBufferOptions.Interrupts, completionHandler: nil)
playerNode.prepareWithFrameCount(frameLength)
}
if playerNode.playing == false {
scheduleBuffer()
let time = AVAudioTime(sampleTime: frameCapacity, atRate: sampleRate)
playerNode.playAtTime(time)
}
else {
scheduleBuffer()
}
}
// triggered by a "Touch Up Inside" event on a UIButton in my ViewController
#IBAction func triggerPlay(sender: AnyObject) {
play()
}
Update:
Ok I think I've identified the source of the distortion: the volume of the node(s) is too great at output and causes clipping. By adding these two lines in my startEngine function, the distortion no longer occurred:
playerNode.volume = 0.8
audioEngine.mainMixerNode.volume = 0.8
However, I'm still don't know why I need to lower the output- my audio file itself does not clip. I'm guessing that it might be a result of the way that the AVAudioPlayerNodeBufferOptions.Interrupts is implemented. When a buffer interrupts another buffer, could there be an increase in output volume as a result of the interruption, causing output clipping? I'm still looking for a solid understanding as to why this occurs.. If anyone is willing/able to provide any clarification about this that would be fantastic!
Not sure if this is the problem you experienced in 2015, it may be the same issue that #suthar experienced in 2018.
I experienced a very similar problem and was due to the fact that the sampleRate on the device is different to the simulator. On macOS it is 44100 and on iOS Devices (late model ones) it is 48000.
So when you fill your buffer with 44100 samples on a 48000 device, you get 3900 samples of silence. When played back it doesn't sound like silence, it sounds like a glitch.
I used the mainMixer format when connecting my playerNode and also when creating my pcmBuffer. Don't refer to 48000 or 44100 anywhere in the code.
audioEngine.attach( playerNode)
audioEngine.connect( playerNode, to:mixerNode, format:mixerNode.outputFormat(forBus:0))
let pcmBuffer = AVAudioPCMBuffer( pcmFormat:SynthEngine.shared.audioEngine.mainMixerNode.outputFormat( forBus:0),
frameCapacity:AVAudioFrameCount(bufferSize))

Resources