Play audio on iOS from a memory data stream - ios

I am porting an audio library to iOS allowing to play audio streams fed from callbacks. The user provides a callback returning raw PCM data, and I need to have this data be played. Moreover, the library must be able to play multiple streams at once.
I figured I would need to use AVFoundation, but it seems like AVAudioPlayer does not support streamed audio buffers, and all the streaming documentation I could find used data coming directly from the network. What is the API I should use here?
Thanks in advance!
By the way, I am not using the Apple libraries through Swift or Objective-C. However I assume everything is exposed still, so an example in Swift would be greatly appreciated anyway!

You need to initialise:
The Audio Session to use input audio unit and output.
-(SInt32) audioSessionInitialization:(SInt32)preferred_sample_rate {
// - - - - - - Audio Session initialization
NSError *audioSessionError = nil;
session = [AVAudioSession sharedInstance];
// disable AVAudioSession
[session setActive:NO error:&audioSessionError];
// set category - (PlayAndRecord to use input and output session AudioUnits)
[session setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker error:&audioSessionError];
double preferredSampleRate = 441000;
[session setPreferredSampleRate:preferredSampleRate error:&audioSessionError];
// enable AVAudioSession
[session setActive:YES error:&audioSessionError];
// Configure notification for device output change (speakers/headphones)
[[NSNotificationCenter defaultCenter] addObserver:self
selector:#selector(routeChange:)
name:AVAudioSessionRouteChangeNotification
object:nil];
// - - - - - - Create audio engine
[self audioEngineInitialization];
return [session sampleRate];
}
The Audio Engine
-(void) audioEngineInitialization{
engine = [[AVAudioEngine alloc] init];
inputNode = [engine inputNode];
outputNode = [engine outputNode];
[engine connect:inputNode to:outputNode format:[inputNode inputFormatForBus:0]];
AudioStreamBasicDescription asbd_player;
asbd_player.mSampleRate = session.sampleRate;
asbd_player.mFormatID = kAudioFormatLinearPCM;
asbd_player.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
asbd_player.mFramesPerPacket = 1;
asbd_player.mChannelsPerFrame = 2;
asbd_player.mBitsPerChannel = 16;
asbd_player.mBytesPerPacket = 4;
asbd_player.mBytesPerFrame = 4;
OSStatus status;
status = AudioUnitSetProperty(inputNode.audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
0,
&asbd_player,
sizeof(asbd_player));
// Add the render callback for the ioUnit: for playing
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = engineInputCallback; ///CALLBACK///
callbackStruct.inputProcRefCon = (__bridge void *)(self);
status = AudioUnitSetProperty(inputNode.audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,//Global
kOutputBus,
&callbackStruct,
sizeof(callbackStruct));
[engine prepare];
}
The Audio Engine callback
static OSStatus engineInputCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
// the reference to the audio controller where you get the stream data
MyAudioController *ac = (__bridge MyAudioController *)(inRefCon);
// in practice we will only ever have 1 buffer, since audio format is mono
for (int i = 0; i < ioData->mNumberBuffers; i++) {
AudioBuffer buffer = ioData->mBuffers[i];
// copy stream buffer data to output buffer
UInt32 size = min(buffer.mDataByteSize, ac.playbackBuffer.mDataByteSize);
memcpy(buffer.mData, ac.streamBuffer.mData, size);
buffer.mDataByteSize = size; // indicate how much data we wrote in the buffer
}
return noErr;
}

Related

Metronome. Timer, music and animations

I develop an app where a user have few cells in which he can put sounds and then playing the built sequence. There is a metronome, it can tick with sound. Users can set metronome speed, that is the same that to set speed of passing to the next cell.
I have realized this mechanism via "timer" with handler, which highlight the current cell and play sounds. Everything works fine. But when I animate some views, my timer stumbles. When animation is finished timer works as expected.
How can I resolve this issue?
I have tried to realize timer via NSTimer, dispatch_after, performSelector:afterDelay:, CADisplayLink and dispatch_source_t. In any case I get problems during the animations. I have even tried to realize my own animation via CADisplayLink, where I calculate animated views frames, this didn't help either.
The only 100% reliable way I found of doing this, is to setup either via CoreAudio or AudioToolbox: https://developer.apple.com/documentation/audiotoolbox an audio stream data provider that gets called by iOS at regular fixed intervals to provide to the audio system the audio samples.
It may looks daunting at first, but once you've got it setup, you have full & precise control about what is generated for audio.
This is the code I used to setup the AudioUnit using AudioToolbox:
static AudioComponentInstance _audioUnit;
static int _outputAudioBus;
...
#pragma mark - Audio Unit
+(void)_activateAudioUnit
{
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryAmbient error:nil];
if([self _createAudioUnitInstance]
&& [self _setupAudioUnitOutput]
&& [self _setupAudioUnitFormat]
&& [self _setupAudioUnitRenderCallback]
&& [self _initializeAudioUnit]
&& [self _startAudioUnit]
)
{
[self _adjustOutputLatency];
// NSLog(#"Audio unit initialized");
}
}
+(BOOL)_createAudioUnitInstance
{
// Describe audio component
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
// Get audio units
OSStatus status = AudioComponentInstanceNew(inputComponent, &_audioUnit);
[self _logStatus:status step:#"instantiate"];
return (status == noErr );
}
+(BOOL)_setupAudioUnitOutput
{
UInt32 flag = 1;
OSStatus status = AudioUnitSetProperty(_audioUnit,
kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Output,
_outputAudioBus,
&flag,
sizeof(flag));
[self _logStatus:status step:#"set output bus"];
return (status == noErr );
}
+(BOOL)_setupAudioUnitFormat
{
AudioStreamBasicDescription audioFormat = {0};
audioFormat.mSampleRate = 44100.00;
audioFormat.mFormatID = kAudioFormatLinearPCM;
audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
audioFormat.mFramesPerPacket = 1;
audioFormat.mChannelsPerFrame = 2;
audioFormat.mBitsPerChannel = 16;
audioFormat.mBytesPerPacket = 4;
audioFormat.mBytesPerFrame = 4;
OSStatus status = AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
_outputAudioBus,
&audioFormat,
sizeof(audioFormat));
[self _logStatus:status step:#"set audio format"];
return (status == noErr );
}
+(BOOL)_setupAudioUnitRenderCallback
{
AURenderCallbackStruct audioCallback;
audioCallback.inputProc = playbackCallback;
audioCallback.inputProcRefCon = (__bridge void *)(self);
OSStatus status = AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Global,
_outputAudioBus,
&audioCallback,
sizeof(audioCallback));
[self _logStatus:status step:#"set render callback"];
return (status == noErr);
}
+(BOOL)_initializeAudioUnit
{
OSStatus status = AudioUnitInitialize(_audioUnit);
[self _logStatus:status step:#"initialize"];
return (status == noErr);
}
+(void)start
{
[self clearFeeds];
[self _startAudioUnit];
}
+(void)stop
{
[self _stopAudioUnit];
}
+(BOOL)_startAudioUnit
{
OSStatus status = AudioOutputUnitStart(_audioUnit);
[self _logStatus:status step:#"start"];
return (status == noErr);
}
+(BOOL)_stopAudioUnit
{
OSStatus status = AudioOutputUnitStop(_audioUnit);
[self _logStatus:status step:#"stop"];
return (status == noErr);
}
+(void)_logStatus:(OSStatus)status step:(NSString *)step
{
if( status != noErr )
{
NSLog(#"AudioUnit failed to %#, error: %d", step, (int)status);
}
}
Finally, once this is started, my registered audio callback will be the one providing the audio:
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
#autoreleasepool {
AudioBuffer *audioBuffer = ioData->mBuffers;
// .. fill in audioBuffer with Metronome sample data, fill the in-between ticks with 0s
}
return noErr;
}
You can use a sound editor like Audacity: https://www.audacityteam.org/download/mac/ to edit and save your file into a RAW PCM mono/stereo data file or you can use one of the AVFoundation libraries to retrieve the audio samples from any of the supported audio files formats. Load your samples into a buffer, keep track of where you left off in between your audio callback frames, and feed in your metronome sample interleaved with 0.
The beauty of this is you can now rely on iOS's AudioToolbox to prioritize your code so both the audio and the view animations don't interfere with each other.
Cheers and Good Luck!
I found a solution, playing with Apple AVAudioEngine example HelloMetronome. I understood the main idea. I have to schedule sounds and handle callbacks in the UI. Using any timers for starting playing sounds and updating UI was absolutely wrong.

CoreAudio: change sample rate of microphone and get data in a callback?

This is my first attempt at using CoreAudio, but my goal is to capture microphone data, resample it to a new sample rate, and then capture the raw 16-bit PCM data.
My strategy for this is to make an AUGraph with the microphone --> a sample rate converter, and then have a callback that gets data from the output of the converter (which I'm hoping is mic output at the new sample rate?).
Right now my callback just fires with a null AudioBufferList*, which obviously isn't correct. How should I set this up and what am I doing wrong?
Code follows:
CheckError(NewAUGraph(&audioGraph), #"Creating graph");
CheckError(AUGraphOpen(audioGraph), #"Opening graph");
AUNode micNode, converterNode;
AudioUnit micUnit, converterUnit;
makeMic(&audioGraph, &micNode, &micUnit);
// get the Input/inputBus's stream description
UInt32 sizeASBD = sizeof(AudioStreamBasicDescription);
AudioStreamBasicDescription hwASBDin;
AudioUnitGetProperty(micUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
kInputBus,
&hwASBDin,
&sizeASBD);
makeConverter(&audioGraph, &converterNode, &converterUnit, hwASBDin);
// connect mic output to converterNode
CheckError(AUGraphConnectNodeInput(audioGraph, micNode, 1, converterNode, 0),
#"Connecting mic to converter");
// set callback on the output? maybe?
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = audioCallback;
callbackStruct.inputProcRefCon = (__bridge void*)self;
CheckError(AudioUnitSetProperty(micUnit,
kAudioOutputUnitProperty_SetInputCallback,
kAudioUnitScope_Global,
kInputBus,
&callbackStruct,
sizeof(callbackStruct)),
#"Setting callback");
CheckError(AUGraphInitialize(audioGraph), #"AUGraphInitialize");
// activate audio session
NSError *err = nil;
AVAudioSession *audioSession = [AVAudioSession sharedInstance];
if (![audioSession setActive:YES error:&err]){
[self error:[NSString stringWithFormat:#"Couldn't activate audio session: %#", err]];
}
CheckError(AUGraphStart(audioGraph), #"AUGraphStart");
and:
void makeMic(AUGraph *graph, AUNode *micNode, AudioUnit *micUnit) {
AudioComponentDescription inputDesc;
inputDesc.componentType = kAudioUnitType_Output;
inputDesc.componentSubType = kAudioUnitSubType_VoiceProcessingIO;
inputDesc.componentFlags = 0;
inputDesc.componentFlagsMask = 0;
inputDesc.componentManufacturer = kAudioUnitManufacturer_Apple;
CheckError(AUGraphAddNode(*graph, &inputDesc, micNode),
#"Adding mic node");
CheckError(AUGraphNodeInfo(*graph, *micNode, 0, micUnit),
#"Getting mic unit");
// enable microphone for recording
UInt32 flagOn = 1; // enable value
CheckError(AudioUnitSetProperty(*micUnit,
kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Input,
kInputBus,
&flagOn,
sizeof(flagOn)),
#"Enabling microphone");
}
and:
void makeConverter(AUGraph *graph, AUNode *converterNode, AudioUnit *converterUnit, AudioStreamBasicDescription inFormat) {
AudioComponentDescription sampleConverterDesc;
sampleConverterDesc.componentType = kAudioUnitType_FormatConverter;
sampleConverterDesc.componentSubType = kAudioUnitSubType_AUConverter;
sampleConverterDesc.componentFlags = 0;
sampleConverterDesc.componentFlagsMask = 0;
sampleConverterDesc.componentManufacturer = kAudioUnitManufacturer_Apple;
CheckError(AUGraphAddNode(*graph, &sampleConverterDesc, converterNode),
#"Adding converter node");
CheckError(AUGraphNodeInfo(*graph, *converterNode, 0, converterUnit),
#"Getting converter unit");
// describe desired output format
AudioStreamBasicDescription convertedFormat;
convertedFormat.mSampleRate = 16000.0;
convertedFormat.mFormatID = kAudioFormatLinearPCM;
convertedFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
convertedFormat.mFramesPerPacket = 1;
convertedFormat.mChannelsPerFrame = 1;
convertedFormat.mBitsPerChannel = 16;
convertedFormat.mBytesPerPacket = 2;
convertedFormat.mBytesPerFrame = 2;
// set format descriptions
CheckError(AudioUnitSetProperty(*converterUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
0, // should be the only bus #
&inFormat,
sizeof(inFormat)),
#"Setting format of converter input");
CheckError(AudioUnitSetProperty(*converterUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
0, // should be the only bus #
&convertedFormat,
sizeof(convertedFormat)),
#"Setting format of converter output");
}
The render callback is used a a source for an audio unit. If you set the kAudioOutputUnitProperty_SetInputCallback property on the remoteIO unit, you must call AudioUnitRender from within the callback you provide, then you would have to manually do the sample rate conversion, which is ugly.
There is an "easier" way. The remoteIO acts as two units, the input (mic) and the output (speaker). Create a graph with a remoteIO, then connect the mic to the speaker, using the desired format. Then you can get the data using a renderNotify callback, which acts as a "tap".
I created a ViewController class to demonstrate
#import "ViewController.h"
#import <AudioToolbox/AudioToolbox.h>
#import <AVFoundation/AVFoundation.h>
#implementation ViewController
- (void)viewDidLoad {
[super viewDidLoad];
//Set your audio session to allow recording
AVAudioSession *audioSession = [AVAudioSession sharedInstance];
[audioSession setCategory:AVAudioSessionCategoryPlayAndRecord error:NULL];
[audioSession setActive:1 error:NULL];
//Create graph and units
AUGraph graph = NULL;
NewAUGraph(&graph);
AUNode ioNode;
AudioUnit ioUnit = NULL;
AudioComponentDescription ioDescription = {0};
ioDescription.componentManufacturer = kAudioUnitManufacturer_Apple;
ioDescription.componentType = kAudioUnitType_Output;
ioDescription.componentSubType = kAudioUnitSubType_VoiceProcessingIO;
AUGraphAddNode(graph, &ioDescription, &ioNode);
AUGraphOpen(graph);
AUGraphNodeInfo(graph, ioNode, NULL, &ioUnit);
UInt32 enable = 1;
AudioUnitSetProperty(ioUnit,kAudioOutputUnitProperty_EnableIO,kAudioUnitScope_Input,1,&enable,sizeof(enable));
//Set the output of the ioUnit's input bus, and the input of it's output bus to the desired format.
//Core audio basically has implicite converters that we're taking advantage of.
AudioStreamBasicDescription asbd = {0};
asbd.mSampleRate = 16000.0;
asbd.mFormatID = kAudioFormatLinearPCM;
asbd.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
asbd.mFramesPerPacket = 1;
asbd.mChannelsPerFrame = 1;
asbd.mBitsPerChannel = 16;
asbd.mBytesPerPacket = 2;
asbd.mBytesPerFrame = 2;
AudioUnitSetProperty(ioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 1, &asbd, sizeof(asbd));
AudioUnitSetProperty(ioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, 0, &asbd, sizeof(asbd));
//Connect output of the remoteIO's input bus to the input of it's output bus
AUGraphConnectNodeInput(graph, ioNode, 1, ioNode, 0);
//Add a render notify with a bridged reference to self (If using ARC)
AudioUnitAddRenderNotify(ioUnit, renderNotify, (__bridge void *)self);
//Start graph
AUGraphInitialize(graph);
AUGraphStart(graph);
CAShow(graph);
}
OSStatus renderNotify(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData){
//Filter anything that isn't a post render call on the input bus
if (*ioActionFlags != kAudioUnitRenderAction_PostRender || inBusNumber != 1) {
return noErr;
}
//Get a reference to self
ViewController *self = (__bridge ViewController *)inRefCon;
//Do stuff with audio
//Optionally mute the audio by setting it to zero;
for (int i = 0; i < ioData->mNumberBuffers; i++) {
memset(ioData->mBuffers[i].mData, 0, ioData->mBuffers[i].mDataByteSize);
}
return noErr;
}
#end

iOS. Record at 96kHz with USB microphone

I am trying to record at full 96kHz with my RØDE iXY USB microphone.
Recording goes without error and when I launch the app with the mic connected, I see that AVAudioSession is running successfully at 96kHz sample rate.
But if I look at the spectrum it is clear that there is nothing but resample noise above 20kHz:
For comparison this is a spectrum of the same recording using the app bundled with the USB mic (RØDE Rec):
Is there anything else I must do to record at native 96kHz?
Or maybe the RØDE Rec app communicates with the mic with some proprietary protocol over USB and I'm out of luck here?
I included the source code that I use:
static AudioStreamBasicDescription AudioDescription24BitStereo96000 = (AudioStreamBasicDescription) {
.mFormatID = kAudioFormatLinearPCM,
.mFormatFlags = kAudioFormatFlagIsPacked | kAudioFormatFlagIsSignedInteger,
.mChannelsPerFrame = 2,
.mBytesPerPacket = 6,
.mFramesPerPacket = 1,
.mBytesPerFrame = 6,
.mBitsPerChannel = 24,
.mSampleRate = 96000.0
};
- (void)setupAudioSession
{
AVAudioSession *session = [AVAudioSession sharedInstance];
[session setCategory:AVAudioSessionCategoryRecord error:&error];
[session setActive:YES error:&error];
[session setPreferredSampleRate:96000.0f error:&error];
//I got my 96000Hz with the USB mic plugged in!
NSLog(#"sampleRate = %lf", session.sampleRate);
}
- (void)startRecording
{
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
AudioComponentInstanceNew(inputComponent, &audioUnit);
AudioUnitScope inputBus = 1;
UInt32 flag = 1;
AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, inputBus, &flag, sizeof(flag));
audioDescription = AudioDescription24BitStereo96000;
AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
inputBus,
&audioDescription,
sizeof(audioDescription));
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = recordingCallback;
callbackStruct.inputProcRefCon = (__bridge void *)(self);
AudioUnitSetProperty(audioUnit,
kAudioOutputUnitProperty_SetInputCallback,
kAudioUnitScope_Global,
inputBus, &callbackStruct,
sizeof(callbackStruct));
AudioOutputUnitStart(audioUnit);
}
static OSStatus recordingCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
AudioBuffer audioBuffer;
audioBuffer.mNumberChannels = 1;
audioBuffer.mDataByteSize = inNumberFrames * audioDescription.mBytesPerFrame;
audioBuffer.mData = malloc( inNumberFrames * audioDescription.mBytesPerFrame );
// Put buffer in a AudioBufferList
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = audioBuffer;
AudioUnitRender(audioUnit, ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, &bufferList);
//I then take the samples and write them to WAV file
}
Check the hardware sample rate audio session property with your microphone plugged in. Also check all audio unit function error return values.
RemoteIO may be using a lower input sample rate and then resampling to a 96k stream.

Sound not playing when App in background (prepareToPlay returns NO)

I've got a strange problem using AVAudioPlayer to play sound files (wav files) on an iPhone in the Background. I am using the following code:
AVAudioPlayer* audioplayer;
NSError* error;
audioplayer = [[AVAudioPlayer alloc] initWithData:soundfile error:&error];
if (error) {
NSLog(#"an error occured while init audioplayer...");
NSLog(#"%#", [error localizedDescription]);
}
audioplayer.currentTime = 0;
if (![audioplayer prepareToPlay])
NSLog(#"could not preparetoPlay");
audioplayer.volume = 1.0;
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayback error:nil];
[[AVAudioSession sharedInstance] setActive: YES error: &error];
if (![audioplayer play])
NSLog(#"could not play sound");
audioplayer.delegate = [myApp sharedInstance];
This works fine while the app is in foreground. However, when moving the app to background [audioplayer prepareToPlay] returns NO.
This happens with AND without "App plays audio" added to the "Required background modes". Is there a way how to get a more precise error report from [audioplayer prepareToPlay]? Or do you have any hints what I am doing wrong or forgot?
You need to initialize your audio session before preparing the AVAudioPlayer instance. Ideally, move the audio session calls to your application delegate's didFinishLaunchingWithOptions method.
I'm not entirely sure this can be achieved using AVFoundation alone, you may need to use the AudioUnit framework and create a stream. Should be relatively simple to send the content of the .WAV file to the audio buffer.
This is how I've been doing it in Piti Piti Pa. The other benefit is you can better control the delay in the audio, in order to synchronize audio and video animations (more obvious when using Bluetooth).
Here's the code I'm using to initialize the audio unit:
+(BOOL)_createAudioUnitInstance
{
// Describe audio component
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
// Get audio units
OSStatus status = AudioComponentInstanceNew(inputComponent, &_audioUnit);
[self _logStatus:status step:#"instantiate"];
return (status == noErr );
}
+(BOOL)_setupAudioUnitOutput
{
UInt32 flag = 1;
OSStatus status = AudioUnitSetProperty(_audioUnit,
kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Output,
_outputAudioBus,
&flag,
sizeof(flag));
[self _logStatus:status step:#"set output bus"];
return (status == noErr );
}
+(BOOL)_setupAudioUnitFormat
{
AudioStreamBasicDescription audioFormat = {0};
audioFormat.mSampleRate = 44100.00;
audioFormat.mFormatID = kAudioFormatLinearPCM;
audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
audioFormat.mFramesPerPacket = 1;
audioFormat.mChannelsPerFrame = 2;
audioFormat.mBitsPerChannel = 16;
audioFormat.mBytesPerPacket = 4;
audioFormat.mBytesPerFrame = 4;
OSStatus status = AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
_outputAudioBus,
&audioFormat,
sizeof(audioFormat));
[self _logStatus:status step:#"set audio format"];
return (status == noErr );
}
+(BOOL)_setupAudioUnitRenderCallback
{
AURenderCallbackStruct audioCallback;
audioCallback.inputProc = playbackCallback;
audioCallback.inputProcRefCon = (__bridge void *)(self);
OSStatus status = AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Global,
_outputAudioBus,
&audioCallback,
sizeof(audioCallback));
[self _logStatus:status step:#"set render callback"];
return (status == noErr);
}
+(BOOL)_initializeAudioUnit
{
OSStatus status = AudioUnitInitialize(_audioUnit);
[self _logStatus:status step:#"initialize"];
return (status == noErr);
}
+(void)start
{
[self clearFeeds];
[self _startAudioUnit];
}
+(void)stop
{
[self _stopAudioUnit];
}
+(BOOL)_startAudioUnit
{
OSStatus status = AudioOutputUnitStart(_audioUnit);
[self _logStatus:status step:#"start"];
return (status == noErr);
}
+(BOOL)_stopAudioUnit
{
OSStatus status = AudioOutputUnitStop(_audioUnit);
[self _logStatus:status step:#"stop"];
return (status == noErr);
}
+(void)_logStatus:(OSStatus)status step:(NSString *)step
{
if( status != noErr )
{
NSLog(#"AudioUnit failed to %#, error: %d", step, (int)status);
}
}
#pragma mark - Mixer
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
#autoreleasepool {
AudioBuffer *audioBuffer = ioData->mBuffers;
_lastPushedFrame = _nextFrame;
[SIOAudioMixer _generateAudioFrames:inNumberFrames into:audioBuffer->mData];
}
return noErr;
}
Now you only need to extract the content of the .Wav files (easier if you export them to RAW format) and send it out to the buffer via the callback.
I hope that helps!
In AppDelegate set the AVAudioSession category like this: (Swift 2)
do {
try AVAudioSession.sharedInstance().setCategory(AVAudioSessionCategoryPlayAndRecord, withOptions: AVAudioSessionCategoryOptions.MixWithOthers)
}catch{
self.fireAnAlert("Set Category Failed", theMessage: "Failed to set AVAudioSession Category")
}
Setting the options to "Mix With Others" Is an important piece!
Then where ever you are going to play sound make sure you call beginReceivingRemoteControlEvents and then set the AVAudioSession to active like this:
do{
UIApplication.sharedApplication().beginReceivingRemoteControlEvents()
try AVAudioSession.sharedInstance().setActive(true)
}catch{
let e = error as NSError
self.appDelegate?.fireAnAlert("Error", theMessage: "\(e.localizedDescription)")
}

iOS app crashes when headphones are plugged in or unplugged

I'm running a SIP audio streaming app on iOS 6.1.3 iPad2 and new iPad.
I start my app on my iPad (nothing plugged in).
Audio works.
I plug in the headphones.
The app crashes: malloc: error for object 0x....: pointer being freed was not allocated or EXC_BAD_ACCESS
Alternatively:
I start my app on my iPad (with the headphones plugged in).
Audio comes out of the headphones.
I unplug the headphones.
The app crashes: malloc: error for object 0x....: pointer being freed was not allocated or EXC_BAD_ACCESS
The app code employs AudioUnit api based on http://code.google.com/p/ios-coreaudio-example/ sample code (see below).
I use a kAudioSessionProperty_AudioRouteChange callback to get change awareness. So there are three callbacks fro the OS sound manager:
1) Process recorded mic samples
2) Provide samples for the speaker
3) Inform audio HW presence
After lots of tests my feeling is that the tricky code is the one that performs the mic capture. After the plug/unplugged action, the most of the times the recording callback is being called a few times before the RouteChange is called causing later 'segmentation fault' and RouteChange callback is never called. Being more specific I think that AudioUnitRender function causes a 'memory bad access' while an Exception is not thrown at all.
My feeling is that a non-atomic recording callback code races with the OS updating of the structures related to sound devices. So as much non-atomic is the recording callback more likely the concurrency of OS HW update and recording callback.
I modified my code to leave the recording callback as thin as possible but my feeling is that the high processing load brought by other threads of my app is feeding concurrency races described before. So the malloc/free error rises in other parts of the code due to the AudioUnitRender bad access.
I tried to reduce recording callback latency by:
UInt32 numFrames = 256;
UInt32 dataSize = sizeof(numFrames);
AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_MaximumFramesPerSlice,
kAudioUnitScope_Global,
0,
&numFrames,
dataSize);
and I tried to boost the problematic code:
dispatch_async(dispatch_get_main_queue(), ^{
Does anybody has a tip or solution for that?
In order to reproduce the error here is my audio session code:
//
// IosAudioController.m
// Aruts
//
// Created by Simon Epskamp on 10/11/10.
// Copyright 2010 __MyCompanyName__. All rights reserved.
//
#import "IosAudioController.h"
#import <AudioToolbox/AudioToolbox.h>
#define kOutputBus 0
#define kInputBus 1
IosAudioController* iosAudio;
void checkStatus(int status) {
if (status) {
printf("Status not 0! %d\n", status);
// exit(1);
}
}
/**
* This callback is called when new audio data from the microphone is available.
*/
static OSStatus recordingCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
// Because of the way our audio format (setup below) is chosen:
// we only need 1 buffer, since it is mono
// Samples are 16 bits = 2 bytes.
// 1 frame includes only 1 sample
AudioBuffer buffer;
buffer.mNumberChannels = 1;
buffer.mDataByteSize = inNumberFrames * 2;
buffer.mData = malloc( inNumberFrames * 2 );
// Put buffer in a AudioBufferList
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = buffer;
NSLog(#"Recording Callback 1 0x%x ? 0x%x",buffer.mData,
bufferList.mBuffers[0].mData);
// Then:
// Obtain recorded samples
OSStatus status;
status = AudioUnitRender([iosAudio audioUnit],
ioActionFlags,
inTimeStamp,
inBusNumber,
inNumberFrames,
&bufferList);
checkStatus(status);
// Now, we have the samples we just read sitting in buffers in bufferList
// Process the new data
[iosAudio processAudio:&bufferList];
NSLog(#"Recording Callback 2 0x%x ? 0x%x",buffer.mData,
bufferList.mBuffers[0].mData);
// release the malloc'ed data in the buffer we created earlier
free(bufferList.mBuffers[0].mData);
return noErr;
}
/**
* This callback is called when the audioUnit needs new data to play through the
* speakers. If you don't have any, just don't write anything in the buffers
*/
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
// Notes: ioData contains buffers (may be more than one!)
// Fill them up as much as you can.
// Remember to set the size value in each
// buffer to match how much data is in the buffer.
for (int i=0; i < ioData->mNumberBuffers; i++) {
// in practice we will only ever have 1 buffer, since audio format is mono
AudioBuffer buffer = ioData->mBuffers[i];
// NSLog(#" Buffer %d has %d channels and wants %d bytes of data.", i,
buffer.mNumberChannels, buffer.mDataByteSize);
// copy temporary buffer data to output buffer
UInt32 size = min(buffer.mDataByteSize,
[iosAudio tempBuffer].mDataByteSize);
// dont copy more data then we have, or then fits
memcpy(buffer.mData, [iosAudio tempBuffer].mData, size);
// indicate how much data we wrote in the buffer
buffer.mDataByteSize = size;
// uncomment to hear random noise
/*
* UInt16 *frameBuffer = buffer.mData;
* for (int j = 0; j < inNumberFrames; j++) {
* frameBuffer[j] = rand();
* }
*/
}
return noErr;
}
#implementation IosAudioController
#synthesize audioUnit, tempBuffer;
void propListener(void *inClientData,
AudioSessionPropertyID inID,
UInt32 inDataSize,
const void *inData) {
if (inID == kAudioSessionProperty_AudioRouteChange) {
UInt32 isAudioInputAvailable;
UInt32 size = sizeof(isAudioInputAvailable);
CFStringRef newRoute;
size = sizeof(CFStringRef);
AudioSessionGetProperty(kAudioSessionProperty_AudioRoute, &size, &newRoute);
if (newRoute) {
CFIndex length = CFStringGetLength(newRoute);
CFIndex maxSize = CFStringGetMaximumSizeForEncoding(length,
kCFStringEncodingUTF8);
char *buffer = (char *)malloc(maxSize);
CFStringGetCString(newRoute, buffer, maxSize,
kCFStringEncodingUTF8);
//CFShow(newRoute);
printf("New route is %s\n",buffer);
if (CFStringCompare(newRoute, CFSTR("HeadsetInOut"), NULL) ==
kCFCompareEqualTo) // headset plugged in
{
printf("Headset\n");
} else {
printf("Another device\n");
UInt32 audioRouteOverride = kAudioSessionOverrideAudioRoute_Speaker;
AudioSessionSetProperty(kAudioSessionProperty_OverrideAudioRoute,
sizeof (audioRouteOverride),&audioRouteOverride);
}
printf("New route is %s\n",buffer);
free(buffer);
}
newRoute = nil;
}
}
/**
* Initialize the audioUnit and allocate our own temporary buffer.
* The temporary buffer will hold the latest data coming in from the microphone,
* and will be copied to the output when this is requested.
*/
- (id) init {
self = [super init];
OSStatus status;
// Initialize and configure the audio session
AudioSessionInitialize(NULL, NULL, NULL, self);
UInt32 audioCategory = kAudioSessionCategory_PlayAndRecord;
AudioSessionSetProperty(kAudioSessionProperty_AudioCategory,
sizeof(audioCategory), &audioCategory);
AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange,
propListener, self);
Float32 preferredBufferSize = .020;
AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration,
sizeof(preferredBufferSize), &preferredBufferSize);
AudioSessionSetActive(true);
// Describe audio component
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType =
kAudioUnitSubType_VoiceProcessingIO/*kAudioUnitSubType_RemoteIO*/;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
// Get component
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
// Get audio units
status = AudioComponentInstanceNew(inputComponent, &audioUnit);
checkStatus(status);
// Enable IO for recording
UInt32 flag = 1;
status = AudioUnitSetProperty(audioUnit,
kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Input,
kInputBus,
&flag,
sizeof(flag));
checkStatus(status);
// Enable IO for playback
flag = 1;
status = AudioUnitSetProperty(audioUnit,
kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Output,
kOutputBus,
&flag,
sizeof(flag));
checkStatus(status);
// Describe format
AudioStreamBasicDescription audioFormat;
audioFormat.mSampleRate = 8000.00;
//audioFormat.mSampleRate = 44100.00;
audioFormat.mFormatID = kAudioFormatLinearPCM;
audioFormat.mFormatFlags =
kAudioFormatFlagsCanonical/* kAudioFormatFlagIsSignedInteger |
kAudioFormatFlagIsPacked*/;
audioFormat.mFramesPerPacket = 1;
audioFormat.mChannelsPerFrame = 1;
audioFormat.mBitsPerChannel = 16;
audioFormat.mBytesPerPacket = 2;
audioFormat.mBytesPerFrame = 2;
// Apply format
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
kInputBus,
&audioFormat,
sizeof(audioFormat));
checkStatus(status);
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
kOutputBus,
&audioFormat,
sizeof(audioFormat));
checkStatus(status);
// Set input callback
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = recordingCallback;
callbackStruct.inputProcRefCon = self;
status = AudioUnitSetProperty(audioUnit,
AudioOutputUnitProperty_SetInputCallback,
kAudioUnitScope_Global,
kInputBus,
&callbackStruct,
sizeof(callbackStruct));
checkStatus(status);
// Set output callback
callbackStruct.inputProc = playbackCallback;
callbackStruct.inputProcRefCon = self;
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Global,
kOutputBus,
&callbackStruct,
sizeof(callbackStruct));
checkStatus(status);
// Disable buffer allocation for the recorder (optional - do this if we want to
// pass in our own)
flag = 0;
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_ShouldAllocateBuffer,
kAudioUnitScope_Output,
kInputBus,
&flag,
sizeof(flag));
flag = 0;
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_ShouldAllocateBuffer,
kAudioUnitScope_Output,
kOutputBus,
&flag,
sizeof(flag));
// Allocate our own buffers (1 channel, 16 bits per sample, thus 16 bits per
// frame, thus 2 bytes per frame).
// Practice learns the buffers used contain 512 frames,
// if this changes it will be fixed in processAudio.
tempBuffer.mNumberChannels = 1;
tempBuffer.mDataByteSize = 512 * 2;
tempBuffer.mData = malloc( 512 * 2 );
// Initialise
status = AudioUnitInitialize(audioUnit);
checkStatus(status);
return self;
}
/**
* Start the audioUnit. This means data will be provided from
* the microphone, and requested for feeding to the speakers, by
* use of the provided callbacks.
*/
- (void) start {
OSStatus status = AudioOutputUnitStart(audioUnit);
checkStatus(status);
}
/**
* Stop the audioUnit
*/
- (void) stop {
OSStatus status = AudioOutputUnitStop(audioUnit);
checkStatus(status);
}
/**
* Change this function to decide what is done with incoming
* audio data from the microphone.
* Right now we copy it to our own temporary buffer.
*/
- (void) processAudio: (AudioBufferList*) bufferList {
AudioBuffer sourceBuffer = bufferList->mBuffers[0];
// fix tempBuffer size if it's the wrong size
if (tempBuffer.mDataByteSize != sourceBuffer.mDataByteSize) {
free(tempBuffer.mData);
tempBuffer.mDataByteSize = sourceBuffer.mDataByteSize;
tempBuffer.mData = malloc(sourceBuffer.mDataByteSize);
}
// copy incoming audio data to temporary buffer
memcpy(tempBuffer.mData, bufferList->mBuffers[0].mData,
bufferList->mBuffers[0].mDataByteSize);
usleep(1000000); // <- TO REPRODUCE THE ERROR, CONCURRENCY MORE LIKELY
}
/**
* Clean up.
*/
- (void) dealloc {
[super dealloc];
AudioUnitUninitialize(audioUnit);
free(tempBuffer.mData);
}
#end
According to my tests, the line that triggers the SEGV error is ultimately
AudioSessionSetProperty(kAudioSessionProperty_OverrideAudioRoute,
sizeof (audioRouteOverride),&audioRouteOverride);
Changing the properties of an AudioUnit chain mid-flight always is tricky, but if you stop the AudioUnit before rerouting, and start it again, it finishes using up all the buffers it has stored and then carries on with the new parameters.
Would that be acceptable, or do you need less of a gap between the change of route and the restart of the recording?
What I did was:
void propListener(void *inClientData,
AudioSessionPropertyID inID,
UInt32 inDataSize,
const void *inData) {
[iosAudio stop];
// ...
[iosAudio start];
}
No more crash on my iPhone 5 (your mileage may vary with different hardware)
The most logical explanation I have for that behavior, somewhat supported by these tests, is that the render pipe is asynchronous. If you take forever to manipulate your buffers, they just stay queued. But if you change the settings of the AudioUnit, you trigger a mass reset in the render queue with unknown side effects. Trouble is, these changes are synchronous, which affect in a retroactive way all the asynchronous calls waiting patiently for their turn.
if you don't care about the missed samples, you can do something like:
static BOOL isStopped = NO;
static OSStatus recordingCallback(void *inRefCon, //...
{
if(isStopped) {
NSLog(#"Stopped, ignoring");
return noErr;
}
// ...
}
static OSStatus playbackCallback(void *inRefCon, //...
{
if(isStopped) {
NSLog(#"Stopped, ignoring");
return noErr;
}
// ...
}
// ...
/**
* Start the audioUnit. This means data will be provided from
* the microphone, and requested for feeding to the speakers, by
* use of the provided callbacks.
*/
- (void) start {
OSStatus status = AudioOutputUnitStart(_audioUnit);
checkStatus(status);
isStopped = NO;
}
/**
* Stop the audioUnit
*/
- (void) stop {
isStopped = YES;
OSStatus status = AudioOutputUnitStop(_audioUnit);
checkStatus(status);
}
// ...

Resources