audioEngine.start() throws exception when called - ios

Problem
I'm trying to record microphone data using AVAudioEngine. This is my setup.
First I create singleton session, set sessions category and activate it.
let audioSession = AVAudioSession.sharedInstance()
do {
try audioSession.setCategory(.record)
try audioSession.setActive(true)
} catch {
...
}
After that i create input format for bus 0, connect input and output nodes and install tap on input node(also tried to tap output node)
let inputFormat = self.audioEngine.inputNode.inputFormat(forBus: 0)
self.audioEngine.connect(self.audioEngine.inputNode, to: self.audioEngine.outputNode, format: inputFormat)
self.audioEngine.outputNode.installTap(onBus: 0, bufferSize: bufferSize, format: inputFormat) { (buffer, time) in
let theLength = Int(buffer.frameLength)
var samplesAsDoubles:[Double] = []
for i in 0 ..< theLength
{
let theSample = Double((buffer.floatChannelData?.pointee[i])!)
samplesAsDoubles.append( theSample )
}
...
}
All of the above is in one function. I also have another function called startRecording which contains the following.
do {
try audioEngine.start()
} catch {
...
}
I have also verified microphone permissions which are granted.
Start method fails and this is the response
The operation couldn’t be completed. (com.apple.coreaudio.avfaudio error -10875.)
Questions
Based on documentation there are three possible causes
- There’s a problem in the structure of the graph, such as the input can’t route to an output or to a recording tap through converter nodes.
- An AVAudioSession error occurs.
- The driver fails to start the hardware.
```
I don't believe it's the session one, because i set it up via documentation.
If the driver fails, how would i detect that and handle it?
If graph is setup incorrectly, where did i mess it up?

If you want to record the microphone, attach the tap to the inputNode, not the outputNode. Taps observe the output of a node. There is no "output" of the outputNode. It's the sink.
If you need to install a tap "immediately before the outputNode" (which might not be the input node in a more complicated graph), you can insert a mixer between the input(s) and the output and attach the tap to the mixer.
Also, make sure you really want to connect the input node to the output node here. There's no need to do that unless you want to playback audio live. You can just attach a tap to the input node without building any more of the graph if you just want to record the microphone. (This also might be causing part of your problem, since you set your category to .record, i.e. no playback. I don't know that wiring something to the output in that case causes an exception, but it definitely doesn't make sense.)

Related

AudioKit bounce recording sync

I need help with an issue... I am recording a bounce with renderToFile, in AudioKit 4 and an artist has complained that the different tracks are not aligned to the sample. In preRender, we have a list of players with the different tracks and records that are set to play in succession, the problem is that I cannot set the AVAudioTime scheduling because it crashes, I suppose due to the fact of the engine being in manualrenderingmode. Is there a way to sync them to the sample? I suppose this is an issue tied to the underlaying AVAudioEngine...
I cannot use AVMutableComposition because we need the recording to be exactly as the one played by AudioKit, and volume would differ.
I've experienced random crashes after setting offline rendering mode, as well as after going back to online rendering mode after being offline.
This situation seems to be triggered by this kind of conditions:
Setting offline rendering mode when the engine hasn't completely finished processing.
Start processing right after calling disableManualRenderingMode, without giving enough time for the engine to start.
A partial workaround I've found is to wait some time before setting offline mode, as well as waiting a small time interval after going online. So I have functions in my code as follows:
func setOnlineMode(successCompletion: #escaping() -> Void, failCompletion: #escaping() -> Void) {
AKManager.engine.stop()
DispatchQueue.main.asyncAfter(deadline: DispatchTime.now() + 0.5, execute: {
AKManager.engine.disableManualRenderingMode()
do {
try AKManager.engine.start()
} catch {
print("failed to start engine")
failCompletion()
}
DispatchQueue.main.asyncAfter(deadline: DispatchTime.now() + 0.5, execute: {
successCompletion()
})
})
}
I also try to avoid resetting the engine after going to offline mode:
/* From AudioKit renderToFile function */
try AKTry {
// Engine can't be running when switching to offline render mode.
if self.isRunning { self.stop() }
try self.enableManualRenderingMode(.offline, format: audioFile.processingFormat, maximumFrameCount: maximumFrameCount)
// This resets the sampleTime of offline rendering to 0.
self.reset() /* I don't do this */
try self.start()
}
To be honest, I've heavily modified my code to avoid going back and forth between online and offline mode. I only do it now at one point in my app, after a delay as explained above.

Connecting bluetooth headphones while app is recording in the background causes the recording to stop

I am facing the following issue and hoping someone else encountered it and can offer a solution:
I am using AVAudioEngine to access the microphone. Until iOS 12.4, every time the audio route changed I was able to restart the AVAudioEngine graph to reconfigure it and ensure the input/output audio formats fit the new input/output route. Due to changes introduced in iOS 12.4 it is no longer possible to start (or restart for that matter) an AVAudioEngine graph while the app is backgrounded (unless it's after an interruption).
The error Apple now throw when I attempt this is:
2019-10-03 18:34:25.702143+0200 [1703:129720] [aurioc] 1590: AUIOClient_StartIO failed (561145187)
2019-10-03 18:34:25.702528+0200 [1703:129720] [avae] AVAEInternal.h:109 [AVAudioEngineGraph.mm:1544:Start: (err = PerformCommand(*ioNode, kAUStartIO, NULL, 0)): error 561145187
2019-10-03 18:34:25.711668+0200 [1703:129720] [Error] Unable to start audio engine The operation couldn’t be completed. (com.apple.coreaudio.avfaudio error 561145187.)
I'm guessing Apple closed a security vulnerability there. So now I removed the code to restart the graph when an audio route is changed (e.g. bluetooth headphones are connected).
It seems like when an I/O audio format changes (as happens when the user connects a bluetooth device), an .AVAudioEngingeConfigurationChange notification is fired, to allow the integrating app to react to the change in format. This is really what I should've used to handle changes in I/O formats from the beginning, instead of brute forcing restarting the graph. According to the Apple documentation - “When the audio engine’s I/O unit observes a change to the audio input or output hardware’s channel count or sample rate, the audio engine stops, uninitializes itself, and issues this notification.” (see the docs here). When this happens while the app is backgrounded, I am unable to start the audio engine with the correct audio i/o formats, because of point #1.
So bottom line, it looks like by closing a security vulnerability, Apple introduced a bug in reacting to audio I/O format changes while the app is backgrounded. Or am I missing something?
I'm attaching a code snippet to better describe the issue. For a plug-and-play AppDelegate see here - https://gist.github.com/nevosegal/5669ae8fb6f3fba44505543e43b5d54b.
class RCAudioEngine {
​
private let audioEngine = AVAudioEngine()
init() {
setup()
start()
NotificationCenter.default.addObserver(self, selector: #selector(handleConfigurationChange), name: .AVAudioEngineConfigurationChange, object: nil)
}
​
#objc func handleConfigurationChange() {
//attempt to call start()
//or to audioEngine.reset(), setup() and start()
//or any other combination that involves starting the audioEngine
//results in an error 561145187.
//Not calling start() doesn't return this error, but also doesn't restart
//the recording.
}
public func setup() {
​
//Setup nodes
let inputNode = audioEngine.inputNode
let inputFormat = inputNode.inputFormat(forBus: 0)
let mainMixerNode = audioEngine.mainMixerNode
​
//Mute output to avoid feedback
mainMixerNode.outputVolume = 0.0
​
inputNode.installTap(onBus: 0, bufferSize: 4096, format: inputFormat) { (buffer, _) -> Void in
//Do audio conversion and use buffers
}
}
​
public func start() {
RCLog.debug("Starting audio engine")
guard !audioEngine.isRunning else {
RCLog.debug("Audio Engine is already running")
return
}
​
do {
audioEngine.prepare()
try audioEngine.start()
} catch {
RCLog.error("Unable to start audio engine \(error.localizedDescription)")
}
}
}
I see only a fix that had gone into iOS 12.4. I am not sure if that causes the issue.
With the release notes https://developer.apple.com/documentation/ios_ipados_release_notes/ios_12_4_release_notes
"Resolved an issue where running an app in iOS 12.2 or later under the Leaks instrument resulted in random numbers of false-positive leaks for every leak check after the first one in a given run. You might still encounter this issue in Simulator, or in macOS apps when using Instruments from Xcode 10.2 and later. (48549361)"
You can raise issue with Apple , if you are a signed developer. They might help you if the defect is on their part.
You can also test with upcoming iOS release to check if your code works in the future release (with the apple beta program)

Process the text once voice input is stopped from SFSpeechRecognizer

I am developing a Voice to Text application using iOS SFSpeechRecognizer API.
Found a great tutorial here: and it worked fine.
I wanted to process the text and perform some action as soon as the voice input is stopped. So, was curious whether there is a delegate method available for SFSpeechRecognizer which can recognise when the voice input is stopped so that I can capture the input and process further?
So, was curious whether there is a delegate method available for SFSpeechRecognizer which can recognise when the voice input is stopped so that I can capture the input and process further?
Not built into the SFSpeechRecognizer API, no. On the contrary, that is exactly why you must provide interface that allows the user to tell the recognizer that the input is finished (e.g. a Done button of some sort). Your app will be rejected if you omit that interface.
A possible solution maybe to use a third party library like FDSoundActivatedRecorder which start recording when sound is detected and
stops recording when the user is done talking.
Then you can use the recorded audio as in this link to convert it to text in a go.
func transcribeAudio(url: URL) {
// create a new recognizer and point it at our audio
let recognizer = SFSpeechRecognizer()
let request = SFSpeechURLRecognitionRequest(url: url)
// start recognition!
recognizer?.recognitionTask(with: request) { [unowned self] (result, error) in
// abort if we didn't get any transcription back
guard let result = result else {
print("There was an error: \(error!)")
return
}
// if we got the final transcription back, print it
if result.isFinal {
// pull out the best transcription...
print(result.bestTranscription.formattedString)
}
}
}

Matching Input & Output Hardware Settings for AVAudioEngine

I am trying to build a very simple audio effects chain using Core Audio for iOS. So far I have implemented an EQ - Compression - Limiter chain which works perfectly fine in the simulator. However on device, the application crashes when connecting nodes to the AVAudioEngine due to an apparent mismatch in the input and output hardware formats.
'com.apple.coreaudio.avfaudio', reason: 'required condition is false:
IsFormatSampleRateAndChannelCountValid(outputHWFormat)'
Taking a basic example, my Audio Graph is as follows.
Mic -> Limiter -> Main Mixer (and Output)
and the graph is populated using
engine.connect(engine.inputNode!, to: limiter, format: engine.inputNode!.outputFormatForBus(0))
engine.connect(limiter, to: engine.mainMixerNode, format: engine.inputNode!.outputFormatForBus(0))
which crashes with the above exception. If I instead use the limiter's format when connecting to the mixer
engine.connect(engine.inputNode!, to: limiter, format: engine.inputNode!.outputFormatForBus(0))
engine.connect(limiter, to: engine.mainMixerNode, format: limiter.outputFormatForBus(0))
the application crashes with an kAudioUnitErr_FormatNotSupported error
'com.apple.coreaudio.avfaudio', reason: 'error -10868'
Before connecting the audio nodes in the engine, inputNode has 1 channel and a sample rate of 44.100Hz, while the outputNode has 0 channels and a sample rate of 0Hz (deduced using outputFormatForBus(0) property). But this could be because there is no node yet connected to the output mixer? Setting the preferred sample rate on AVAudioSession made no difference.
Is there something that I am missing here? I have Microphone access (verified using AVAudioSession.sharedInstance().recordPermission()), and I have set the AVAudioSession mode to record (AVAudioSession.sharedInstance().setCategory(AVAudioSessionCategoryRecord)).
The limiter is an AVAudioUnitEffect initialized as follows:
let limiter = AVAudioUnitEffect( audioComponentDescription:
AudioComponentDescription(
componentType: kAudioUnitType_Effect,
componentSubType: kAudioUnitSubType_PeakLimiter,
componentManufacturer: kAudioUnitManufacturer_Apple,
componentFlags: 0,
componentFlagsMask: 0) )
engine.attachNode(limiter)
and engine is a global, class variable
var engine = AVAudioEngine()
As I said, this works perfectly fine using the simulator (and Mac's default hardware), but continually crashes on various iPads on iOS8 & iOS9. I have a super basic example working which simply feeds the mic input to a player to the output mixer
do {
file = try AVAudioFile(forWriting: NSURL.URLToDocumentsFolderForName(name: "test", WithType type: "caf")!, settings: engine.inputNode!.outputFormatForBus(0).settings)
} catch {}
engine.connect(player, to: engine.mainMixerNode, format: file.processingFormat)
Here the inputNode has 1 channel and 44.100Hz sampling rate, while the outputNode has 2 channels and 44.100Hz sampling rate, but no mismatching seems to occur. Thus the issue must be the manner in which the AVAudioUnitEffect is connected to the output mixer.
Any help would be greatly appreciated.
This depends on some factors outside of the code you've shared, but it's possible you're using the wrong AVAudioSession category.
I ran into this same issue, under some slightly different circumstances. When I was using AVAudioSessionCategoryRecord as the AVAudioSession category, I ran into this same issue when attempting to connect an audio tap. I not only received that error, but my AVAudioEngine inputNode showed an outputFormat with 0.0 sample rate.
Changing it to AVAudioSessionCategoryPlayAndRecord I received the expected 44.100Hz sample rate and the issue resolved.

Audio Unit Render Callback - change it on the fly?

I have a multichannel mixer and a remote I/O in a graph, setup to play uncompressed caf files. So far so good.
Next, I am experimenting with doing weird stuff on the render callback (say, generate white noise, or play a sine wave, etc. - procedurally generated sounds).
Instead of adding conditionals to the existing render callback (which is assigned on setup to all the buses of the mixer), I would like to be able to switch the render callback attached to each bus, at runtime.
So far I'm trying this code, but it doesn't work: My alternative render callback does not get called.
- (void) playNoise
{
if (_noiseBus != -1) {
// Already playing
return;
}
_noiseBus = [self firstFreeMixerBus];
AUGraphDisconnectNodeInput(processingGraph,
mixerNode,
_noiseBus);
inputCallbackStructArray[_noiseBus].inputProc = &noiseRenderCallback;
inputCallbackStructArray[_noiseBus].inputProcRefCon = NULL;
OSStatus result = AUGraphSetNodeInputCallback(processingGraph,
mixerNode,
_noiseBus,
&inputCallbackStructArray[_noiseBus]);
if (result != noErr) {
return NSLog#"AUGraphSetNodeInputCallback");
}
result = AudioUnitSetParameter(_mixerUnit,
kMultiChannelMixerParam_Enable,
kAudioUnitScope_Input,
_noiseBus,
1,
0);
if (result != noErr) {
return NSLog(#"Failed to enable bus");
}
result = AudioUnitSetParameter (_mixerUnit,
kMultiChannelMixerParam_Volume,
kAudioUnitScope_Input,
_noiseBus,
0.5,
0);
if (result != noErr) {
return NSLog(#"AudioUnitSetParameter (set mixer unit input volume) Failed");
}
unsigned char updated;
AUGraphUpdate(processingGraph, &(updated));
// updated ends un being zero ('\0')
}
In the code above, none of the error conditions are met (no function call fails), but the boolean 'updated' remains false until the end.
Am I missing a step, or is it not possible to switch render callbacks after setup? Do I need to set aside dedicated buses to these alternative callbacks? I would like to be able to set custom callbacks from the client code (the side calling my sound engine)...
EDIT Actually it is working, but only after the second time: I must call -playNoise, then -stopNoise, and from then on it will play normally. I couldn't tell, because I was giving up at the first try...
BTW, The updated flag is still 0.
I added lots of audio unit calls out of desperation, but perhaps some are not necessary. I'll see which ones I can trim, then keep looking for the reason it needs two calls to work...
EDIT 2: After poking around, adding/removing calls and fixing bugs, I got to the point where the noise render callback works from the first time, but after playing the noise at least once, if I attempt to reuse that bus form playing PCM (caf file), it still uses the noise render callback (despite having disconnected it). I'm going with the solution suggested by #hotpaw2 in the comments and using a 'stub' callback and further function pointers...

Resources