I have a AVAudioPlayerNode that I'm scheduling a lot of buffers on using the following:
node.scheduleBuffer(buffer, at: nil, options: .interrupts, completionHandler: completeFunc)
But I'm having a bit of a problem. If I play another buffer interrupting the currently playing buffer, the completion handler for the buffer getting interrupted is still being called. I guess this makes sense, but I can't seem to find a way to check if the file actually COMPLETED playing or if it was interrupted. Is there a way I can do this?
Any answers will help!
You will need an interruption state variable that is referenced in the completion handler and toggled by the interrupting code. Here's some shorthand code that makes some assumptions on how you're looping through buffers:
var node: AVAudioPlayerNode!
var loopingBuffers: [AVAudioPCMBuffer] = [buffer1, buffer2, buffer3]
var interruptBuffers = false // state variable to handle the interruption
func scheduleNextBuffer() {
/* code to find the next buffer in the loopingBuffers queue */
// assuming you have instantiated an AVAudioPlayerNode elsewhere
node.scheduleBuffer(nextBuffer, completionHandler: bufferCompletion)
}
func bufferCompletion() {
// check the state variable to make certain the completionHandler isn't being triggered by a scheduling interruption
guard !interruptBuffers else { return }
scheduleNextBuffer()
}
func interruptBuffers() {
let newBuffers: [AVAudioPCMBuffer] = [buffer4, buffer5, buffer6]
interruptBuffers = true
node.scheduleBuffer(buffer4, at: nil, options: .interrupts, completionHandler: bufferCompletion)
// cleanup your state and queue up your new buffers
interruptBuffers = false
loopingBuffers = newBuffers
}
Related
I'm trying to improve the time it takes for a task to finish by leveraging multithreading/paralleling.
I'm reading CMSampleBuffers from a video track, manipulating them, then storing them back to an array for later use. Because each manipulation is "costly" in terms of RAM and CPU, I'm using DispatchSemaphore to limit the number of parallel tasks (5 in the example).
Basically, I'm trying to make the "system" process more than a single frame in a time, and not above 5 so the device won't crash due to memory issues. In the current implementation below it's for some reason doing it almost serialize and not in parallel.
Any help will be highly appreciated!
I tried taking reference from here: How to limit gcd queue buffer size for the implementation.
Code:
class MyService {
let semaphore = DispatchSemaphore(value: 5)
let processQueue = DispatchQueue(label: "custom.process", attributes: .concurrent)
func startReading() {
for sampleBuffer in sampleBuffers {
// signal wait
semaphore.wait()
// async queue
processQueue.async {
// run taks
self.process(buffer: sampleBuffer) { pixelBuffer in
// singal to semaphore
self.semaphore.signal()
}
}
}
}
func process(buffer: CMSampleBuffer, completion: #escaping (CVPixelBuffer) -> (Void)) {
// run on a background thread to avoid UI freeze
DispatchQueue.global(qos: .userInteractive).async {
// Do something
// Do something
// Do something
// Do something
completion(processedBuffer)
}
}
}
I’m recording microphone input to match it to a song in the Shazam catalog. This works, but if I start() the AVAudioEngine then something happens like music starts playing via MPMusicPlayerController.applicationMusicPlayer.play(), it seems the audio engine stops or gets interrupted. The microphone recording shuts off, thus the SHSessionDelegate never finds a match or fails with an error, so my UI is stuck showing it's listening when it’s not anymore. Is there a way to be informed when this happens so that I may update the UI to handle cancelation?
private lazy var shazamAudioEngine = AVAudioEngine()
private lazy var shazamSession: SHSession = {
let session = SHSession()
session.delegate = self
return session
}()
...
try? AVAudioSession.sharedInstance().setCategory(.record)
//Create an audio format for our buffers based on the format of the input, with a single channel (mono)
let audioFormat = AVAudioFormat(standardFormatWithSampleRate: shazamAudioEngine.inputNode.outputFormat(forBus: 0).sampleRate, channels: 1)
//Install a "tap" in the audio engine's input so that we can send buffers from the microphone to the session
shazamAudioEngine.inputNode.installTap(onBus: 0, bufferSize: 2048, format: audioFormat) { [weak self] buffer, when in
//Whenever a new buffer comes in, we send it over to the session for recognition
self?.shazamSession.matchStreamingBuffer(buffer, at: when)
}
do {
try shazamAudioEngine.start()
} catch {
...
}
In my testing isRunning tracks this state, so it changes from true to false when you start playing music and the microphone stops being recorded. Unfortunately that property can't be observed with KVO so what I did was set up a repeating Timer to detect if it changes to handle cancelation, making sure to invalidate() the timer when other state changes occur.
audioEngineRunningTimer = Timer.scheduledTimer(withTimeInterval: 0.5, repeats: true) { [weak self] timer in
//the audio engine stops running when you start playing music for example, so handle cancelation here
if self?.shazamAudioEngine.isRunning == false {
//...
}
}
How is it possible to wait for an #escaping closure to complete inline before proceeding?
I am utilizing the write method from AVSpeechSynthesizer, which uses an #escaping closure, so the initial AVAudioBuffer from the callback will return after createSpeechToBuffer has completed.
func write(_ utterance: AVSpeechUtterance, toBufferCallback bufferCallback: #escaping AVSpeechSynthesizer.BufferCallback)
My method writes speech to a buffer, then resamples and manipulates the output, for a workflow, where speech is done in faster than real-time.
The goal is to perform the task inline, to avoid changing the workflow to standby for the 'didFinish' delegate
speechSynthesizer(_ synthesizer: AVSpeechSynthesizer, didFinish utterance: AVSpeechUtterance)
I believe this question can be generalized to dealing with #escaping closures within a function\method
import Cocoa
import AVFoundation
let _speechSynth = AVSpeechSynthesizer()
func resampleBuffer( inSource: AVAudioPCMBuffer, newSampleRate: Float) -> AVAudioPCMBuffer
{
// simulate resample data here
let testCapacity = 1024
let audioFormat = AVAudioFormat(standardFormatWithSampleRate: Double(newSampleRate), channels: 2)
let simulateResample = AVAudioPCMBuffer(pcmFormat: audioFormat!, frameCapacity: UInt32(testCapacity))
return simulateResample!
}
func createSpeechToBuffer( stringToSpeak: String, sampleRate: Float) -> AVAudioPCMBuffer?
{
var outBuffer : AVAudioPCMBuffer? = nil
let utterance = AVSpeechUtterance(string: stringToSpeak)
var speechIsBusy = true
utterance.voice = AVSpeechSynthesisVoice(language: "en-us")
let semaphore = DispatchSemaphore(value: 0)
_speechSynth.write(utterance) { (buffer: AVAudioBuffer) in
guard let pcmBuffer = buffer as? AVAudioPCMBuffer else {
fatalError("unknown buffer type: \(buffer)")
}
if ( pcmBuffer.frameLength == 0 ) {
print("buffer is empty")
} else {
print("buffer has content \(buffer)")
}
outBuffer = resampleBuffer( inSource: pcmBuffer, newSampleRate: sampleRate)
speechIsBusy = false
// semaphore.signal()
}
// wait for completion of func speechSynthesizer(_ synthesizer: AVSpeechSynthesizer, didFinish utterance: AVSpeechUtterance)
// while ( _speechSynth.isSpeaking )
// {
// /* arbitrary task waiting for write to complete */
// }
//
// while ( speechIsBusy )
// {
// /* arbitrary task waiting for write to complete */
// }
// semaphore.wait()
return outBuffer
}
print("SUCCESS is waiting, returning the non-nil output from the resampleBuffer method.")
for indx in 1...10
{
let sentence = "This is sentence number \(indx). [[slnc 3000]] \n"
let outBuffer = createSpeechToBuffer( stringToSpeak: sentence, sampleRate: 48000.0)
print("outBuffer: \(String(describing: outBuffer))")
}
After I wrote the createSpeechToBuffer method and it failed to produce the desired output (inline), I realized that it returns before getting the results of the resampling. The callback is escaping, so the initial AVAudioBuffer from the callback will return after createSpeechToBuffer has completed. The actual resampling does work, however I currently must save the result and continue after being notified by the delegate "didFinish utterance" to proceed.
Attempts at waiting for _speechSynth.isSpeaking, speechIsBusy flag, dispatch queue and semaphore are blocking the write method (using _speechSynth.write) from completing.
How is it possible to wait for the result inline versus recreating a workflow depending on the delegate "didFinish utterance"?
I'm on macOS 11.4 (Big Sur) but I believe this question is applicable to macOS and ios
It looks to me that the commented-out code for DispatchSemaphore would work if the #escaping closure is run concurrently, and I think the problem is that it is run serially, or more accurately, not run at all, because it is scheduled to run serially. I'm not specifically familiar with the AVSpeechSynthesizer API, but from your description, it sounds to me as though it's calling on the main dispatch queue, which is a serial queue. You call wait to block until _speechSynth.write completes, but that's blocking the main thread, which prevents it from ever continuing to the next iteration of the run loop, so the actual work of _speechSynth.write never even starts.
Let's back up. Somewhere behind the scenes your closure is almost certainly called via DispatchQueue.main's async method, either because that's where speechSynth.write does its work then calls your closure synchronously on the current thread at the time, or because it explicitly calls it on the main thread.
A lot of programmers are sometimes confused as to exactly what async does. All async means is "schedule this task and return control to the caller immediately". That's it. It does not mean that the task will be run concurrently, only that it will be run later. Whether it is run concurrently or serially is an attribute of the DispatchQueue whose async method is being called. Concurrent queues spin up threads for their tasks, which either can be run in parallel on different CPU cores (true concurrency), or interleaved with the current thread on the same core (preemptive multitasking). Serial queues on the other hand have a run loop as in NSRunLoop, and run their scheduled tasks synchronously after dequeuing them.
To illustrate what I mean, the main run loop looks vaguely like this, and other run loops are similar:
while !quit
{
if an event is waiting {
dispatch the event <-- Your code is likely blocking in here
}
else if a task is waiting in the queue
{
dequeue the task
execute the task <-- Your closure would be run here
}
else if a timer has expired {
run timer task
}
else if some view needs updating {
call the view's draw(rect:) method
}
else { probably other things I'm forgetting }
}
createSpeechToBuffer is almost certainly being run in response to some event processing, which means that when it blocks, it does not return back to the run loop to continue to the next iteration where it checks for tasks in the queue... which from the behavior you describe, seems to include the work being done by _speechSynth.write... the very thing you're waiting for.
You can try explicitly creating a .concurrent DispatchQueue and using it to wrap the call to _speechSynth.write in an explicit async call, but that probably won't work, and even if it does, it will be fragile to changes Apple might make to AVSpeechSynthesizer's implementation.
The safe way is to not block... but that means re-thinking your work flow a little. Basically whatever code would be called after createSpeechToBuffer returns should be called at the end of your closure. Of course, as currently written createSpeechToBuffer doesn't know what that code is (nor should it). The solution is to inject it as a parameter... meaning createSpeechToBuffer itself would also take an #escaping closure. And of course, that means it can't return the buffer, but instead passes it to the closure.
func createSpeechToBuffer(
stringToSpeak: String,
sampleRate: Float,
onCompletion: #escaping (AVAudioPCMBuffer?) -> Void)
{
let utterance = AVSpeechUtterance(string: stringToSpeak)
utterance.voice = AVSpeechSynthesisVoice(language: "en-us")
let semaphore = DispatchSemaphore(value: 0)
_speechSynth.write(utterance) { (buffer: AVAudioBuffer) in
guard let pcmBuffer = buffer as? AVAudioPCMBuffer else {
fatalError("unknown buffer type: \(buffer)")
}
if ( pcmBuffer.frameLength == 0 ) {
print("buffer is empty")
} else {
print("buffer has content \(buffer)")
}
onCompletion(
resampleBuffer(
inSource: pcmBuffer,
newSampleRate: sampleRate
)
)
}
}
If you really want to maintain the existing API, the other approach is to move the entire workflow itself to a .concurrent DispatchQueue, which you can block to your heart's content without worry that it will block the main thread. AVSpeechSynthesizer could schedule its work wherever it likes without a problem.
If using Swift 5.5 is an option, you might look into its async and await keywords. The compiler enforces a proper async context for them so that you don't block the main thread.
Update to answer how to call my version.
Let's say your code that calls createSpeechToBuffer currently looks like this:
guard let buffer = createSpeechToBuffer(stringToSpeak: "Hello", sampleRate: sampleRate)
else { fatalError("Could not create speechBuffer") }
doSomethingWithSpeechBuffer(buffer)
You'd call the new version like this:
createSpeechToBuffer(stringToSpeak: "Hello", sampleRate: sampleRate)
{
guard let buffer = $0 else {
fatalError("Could not create speechBuffer")
}
doSomethingWithSpeechBuffer(buffer)
}
Recently I got notified about a crash from firebase and this is the message:
[AVCaptureSession stopRunning] stopRunning may not be called between calls to beginConfiguration and commitConfiguration
I went through my code and the weirdest part is I never call and there is no mention nowhere of beginConfiguration() and commitConfiguration().
In my CameraManager class this is the function that triggers the crash, it called on deinit:
func stop() {
guard isStarted else { return Log.w("CameraManager wasn't started") }
queue.async {
self.isStarted = false
self.isCapturingFrame = false
self.isCapturingCode = false
self.session?.stopRunning()
self.session = nil
self.device = nil
}
notificationCenter.removeObserver(self, name: UIApplication.didBecomeActiveNotification, object: nil)
layer.removeFromSuperlayer()
layer.session = nil
}
The queue is just a serial disptach queue.
No matter what I tried, I couldn't reproduce this crash.
Tried pulling the menu, push notification, phone call, simulate memory warning etc...
Just to clarify, there is not a single place in my code calling beginConfiguration and commitConfiguration.
I could imagine that layer.session = nil will cause a re-configuration of the capture session (since the connection to the preview layer is removed). And since you are calling stopRunning() async, I guess you can run into race conditions where stopRunning() gets called right in the middle of the configuration change.
I would suggest you either try to make the cleanup calls synchronous (queue.sync { ... }) or move the layer cleanup into the async block as well.
I'm having troubles with Alamofire using Operation and OperationQueue.
I have an OperationQueue named NetworkingQueue and I push some operation (wrapping AlamofireRequest) into it, everything works fine, but during application living, at one moment all Alamofire request are not sent. My queue is getting bigger and bigger and no request go to the end.
I do not have a scheme to reproduce it anytime.
Does anybody have a clue for helping me?
Here is a sample of code
The BackgroundAlamoSession
let configuration = URLSessionConfiguration.background(withIdentifier: "[...].background")
self.networkingSessionManager = Alamofire.SessionManager(configuration: configuration)
AbstractOperation.swift
import UIKit
import XCGLogger
class AbstractOperation:Operation {
private let _LOGGER:XCGLogger = XCGLogger.default
enum State:String {
case Ready = "ready"
case Executing = "executing"
case Finished = "finished"
var keyPath: String {
get{
return "is" + self.rawValue.capitalized
}
}
}
override var isAsynchronous:Bool {
get{
return true
}
}
var state = State.Ready {
willSet {
willChangeValue(forKey: self.state.rawValue)
willChangeValue(forKey: self.state.keyPath)
willChangeValue(forKey: newValue.rawValue)
willChangeValue(forKey: newValue.keyPath)
}
didSet {
didChangeValue(forKey: oldValue.rawValue)
didChangeValue(forKey: oldValue.keyPath)
didChangeValue(forKey: self.state.rawValue)
didChangeValue(forKey: self.state.keyPath)
}
}
override var isExecuting: Bool {
return state == .Executing
}
override var isFinished:Bool {
return state == .Finished
}
}
A concrete Operation implementation
import UIKit
import XCGLogger
import SwiftyJSON
class FetchObject: AbstractOperation {
public let _LOGGER:XCGLogger = XCGLogger.default
private let _objectId:Int
private let _force:Bool
public var object:ObjectModel?
init(_ objectId:Int, force:Bool) {
self._objectId = objectId
self._force = force
}
convenience init(_ objectId:Int) {
self.init(objectId, force:false)
}
override var desc:String {
get{
return "FetchObject(\(self._objectId))"
}
}
public override func start(){
self.state = .Executing
_LOGGER.verbose("Fetch object operation start")
if !self._force {
let objectInCache:objectModel? = Application.main.collections.availableObjectModels[self._objectId]
if let objectInCache = objectInCache {
_LOGGER.verbose("object with id \(self._objectId) founded on cache")
self.object = objectInCache
self._LOGGER.verbose("Fetch object operation end : success")
self.state = .Finished
return
}
}
if !self.isCancelled {
let url = "[...]\(self._objectId)"
_LOGGER.verbose("Requesting object with id \(self._objectId) on server")
Application.main.networkingSessionManager.request(url, method : .get)
.validate()
.responseJSON(
completionHandler: { response in
switch response.result {
case .success:
guard let raw:Any = response.result.value else {
self._LOGGER.error("Error while fetching json programm : Empty response")
self._LOGGER.verbose("Fetch object operation end : error")
self.state = .Finished
return
}
let data:JSON = JSON(raw)
self._LOGGER.verbose("Received object from server \(data["bId"])")
self.object = ObjectModel(objectId:data["oId"].intValue,data:data)
Application.main.collections.availableobjectModels[self.object!.objectId] = self.object
self._LOGGER.verbose("Fetch object operation end : success")
self.state = .Finished
break
case .failure(let error):
self._LOGGER.error("Error while fetching json program \(error)")
self._LOGGER.verbose("Fetch object operation end : error")
self.state = .Finished
break
}
})
} else {
self._LOGGER.verbose("Fetch object operation end : cancel")
self.state = .Finished
}
}
}
The NetworkQueue
class MyQueue {
public static let networkQueue:SaootiQueue = SaootiQueue(name:"NetworkQueue", concurent:true)
}
How I use it in another operation and wait for for result
let getObjectOperation:FetchObject = FetchObject(30)
SaootiQueue.networkQueue.addOperations([getObjectOperation], waitUntilFinished: true)
How I use it the main operation using KVO
let getObjectOperation:FetchObject = FetchObject(30)
operation.addObserver(self, forKeyPath: #keyPath(Operation.isFinished), options: [.new], context: nil)
operation.addObserver(self, forKeyPath: #keyPath(Operation.isCancelled), options: [.new], context: nil)
queue.addOperation(operation)
//[...]
override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
if let operation = object as? FetchObject {
operation.removeObserver(self, forKeyPath: #keyPath(Operation.isFinished))
operation.removeObserver(self, forKeyPath: #keyPath(Operation.isCancelled))
if keyPath == #keyPath(Operation.isFinished) {
//Do something
}
}
A few clarifications:
My application is a radio player and I need, while playing music and the background, to fetch the currently playing program. This is why I need background Session.
In fact I also use the background session for all the networking I do when the app is foreground. Should I avoid that ?
The wait I'm using is from another queue and is never used in the main queue (I know it is a threading antipattern and I take care of it).
In fact it is used when I do two networking operation and the second one depends of the result of the second. I put a wait after the first operation to avoid KVO observing. Should I avoid that ?
Additional edit:
When I say "My queue is getting bigger and bigger and no request go to the end", it means that at one moment during application livecycle, random for the moment (I can not find a way to reproduce it at every time), Alamofire request don't reach the response method.
Because of that the Operation wrapper don't end and the queue is growing.
By the way I'm working on converting Alamofire request into URLRequest for having clues and I founded some problem on using the main queue. I have to sort what is due to the fact that Alamofire use the main queue for reponse method and I'll see if I find a potential deadlock
I'll keep you informed. Thanks
There are minor issues, but this operation implementation looks largely correct. Sure, you should make your state management thread-safe, and there are other stylistic improvements you could make, but I don't think this is critical to your question.
What looks worrisome is addOperations(_:waitUntilFinished:). From which queue are you waiting? If you do that from the main queue, you will deadlock (i.e. it will look like the Alamofire requests never finish). Alamofire uses the main queue for its completion handlers (unless you override the queue parameter of responseJSON), but if you're waiting on the main thread, this can never take place. (As an aside, if you can refactor so you never explicitly "wait" for operations, that not only avoids the deadlock risk, but is a better pattern in general.)
I also notice that you're using Alamofire requests wrapped in operations in conjunction with a background session. Background sessions are antithetical to operations and completion handler closure patterns. Background sessions continue after your app has been jettisoned and you have to rely solely upon the SessionDelegate closures that you set when you first configure your SessionManager when the app starts. When the app restarts, your operations and completion handler closures are long gone.
Bottom line, do you really need background session (i.e. uploads and downloads that continue after your app terminates)? If so, you may want to lose this completion handler and operation based approach. If you don't need this to continue after the app terminates, don't use background sessions. Configuring Alamofire to properly handle background sessions is a non-trivial exercise, so only do so if you absolutely need to. Remember to not conflate background sessions and the simple asynchronous processing that Alamofire (and URLSession) do automatically for you.
You asked:
My application is a radio player and I need, while playing music and the background, to fetch the currently playing program. This is why I need background Session.
You need background sessions if you want downloads to proceed while the app is not running. If your app is running in the background, though, playing music, you probably don't need background sessions. But, if the user chooses to download a particular media asset, you may well want background session so that the download proceeds when the user leaves the app, whether the app is playing music or not.
In fact I also use the background session for all the networking I do when the app is foreground. Should I avoid that ?
It's fine. It's a little slower, IIRC, but it's fine.
The problem isn't that you're using background session, but that you're doing it wrong. The operation-based wrapping of Alamofire doesn't make sense with a background session. For sessions to proceed in the background, you are constrained as to how you use URLSession, namely:
You cannot use data tasks while the app is not running; only upload and download tasks.
You cannot rely upon completion handler closures (because the entire purpose of background sessions is to keep them running when your app terminates and then fire up your app again when they're done; but if the app was terminated, your closures are all gone).
You have to use delegate based API only for background sessions, not completion handlers.
You have to implement the app delegate method to capture the system provided completion handler that you call when you're done processing background session delegate calls. You have to call that when your URLSession tells you that it's done processing all the background delegate methods.
All of this is a significant burden, IMHO. Given that the system is keeping you app alive for background music, you might contemplate using a standard URLSessionConfiguration. If you're going to use background session, you might need to refactor all of this completion handler-based code.
The wait I'm using is from another queue and is never used in the main queue (I know it is a threading antipattern and I take care of it).
Good. There's still serious code smell from ever using "wait", but if you are 100% confident that it's not deadlocking here, you can get away with it. But it's something you really should check (e.g. put some logging statement after the "wait" and make sure you're getting past that line, if you haven't already confirmed this).
In fact it is used when I do two networking operation and the second one depends of the result of the second. I put a wait after the first operation to avoid KVO observing. Should I avoid that ?
Personally, I'd lose that KVO observing and just establish addDependency between the operations. Also, if you get rid of that KVO observing, you can get rid of your double KVO notification process. But I don't think this KVO stuff is the root of the problem, so maybe you defer that.