I have a simple requirement at this point, an iOS app that reads from an audio file and outputs to a speaker using AudioUnits. The reason behind not using high-level APIs is, at some point, I need to process the samples coming out of the audio file and eventually send it across network.
I have a code that works, reads the audio file and plays back to the speaker. The only issue here is, the render callback isn't working. The callback never gets called, neither do I receive any error while registering the same. Help is much appreciated (I am a beginner on Core Audio and this is my first question on stackoverflow, so please pardon any basic mistakes/overlooks). The piece of code I use for initializing the graph is attached.
void createMyAUGraph (MyAUGraphPlayerST *player) {
// Create a new AUGraph
CheckError(NewAUGraph(&player->graph), "New AUGraph failed");
// Generate description for output
AudioComponentDescription outputcd = {0};
outputcd.componentType = kAudioUnitType_Output;
outputcd.componentSubType = kAudioUnitSubType_RemoteIO;
outputcd.componentManufacturer = kAudioUnitManufacturer_Apple;
outputcd.componentFlags = 0;
outputcd.componentFlagsMask = 0;
// Add new node
AUNode outputNode;
CheckError(AUGraphAddNode(player->graph, &outputcd, &outputNode), "Add output node failed");
// Node for file player
AudioComponentDescription fileplayercd = {0};
fileplayercd.componentType = kAudioUnitType_Generator;
fileplayercd.componentSubType = kAudioUnitSubType_AudioFilePlayer;
fileplayercd.componentManufacturer = kAudioUnitManufacturer_Apple;
// Add new node
AUNode fileNode;
CheckError(AUGraphAddNode(player->graph, &fileplayercd, &fileNode), "Add file node failed");
// Open graph
CheckError(AUGraphOpen(player->graph), "Graph open failed");
// Retrive AudioUnit
CheckError(AUGraphNodeInfo(player->graph, outputNode, NULL, &player->outputAU), "file unit retrive failed");
CheckError(AUGraphNodeInfo(player->graph, fileNode, NULL, &player->fileAU), "file unit retrive failed");
// connect nodes
CheckError(AUGraphConnectNodeInput(player->graph, fileNode, 0, outputNode, 0), "failed to connect nodes");
// some other setup
UInt32 flag = 1;
CheckError(AudioUnitSetProperty(player->outputAU,
kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Output,
0,
&flag,
sizeof (flag)), "Set io property failed");
// Register render callback
AURenderCallbackStruct output_cb;
output_cb.inputProc = recording_cb;
output_cb.inputProcRefCon = player;
CheckError(AudioUnitSetProperty(player->outputAU, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Global, 0, &output_cb, sizeof (output_cb)), "callback register failed");
// initialize graph
CheckError(AUGraphInitialize(player->graph), "graph initialization failed");
}
You told the graph to connect your RemoteIO's input to the file player node, not to your render callback. Then you initialized the graph, which overrode your render property.
If you want to pull samples from a file to process, your processing routine or render callback will have to do so, not the connection from the player output to the RemoteIO input. So don't let the graph make that connection.
Updated answer:
On any recent iOS version, you need to first use the Audio Session API to request microphone privacy permission before starting the Audio Units, otherwise you will only get silence from the microphone.
To use Audio Units to record and play thru as well as record, try putting callbacks on both the output and input of RemoteIO, then pass the sample data between the two callbacks using a circular buffer. Inside one or both of the callbacks, you can record or modify the samples as needed. Make sure to heed real-time restrictions inside the audio context (no locks or memory management, etc.)
Related
I'm using AudioUnit to playback audio from a TeamSpeak server but when I call AudioUnitInitialize on the iOS Simulator, I'm getting constantly the macOS prompt to allow microphone access even if I want to playback only.
On a real device everything works fine without any native prompts but it is really annoying when running the app in the simulator because this prompts appear every time I run the app.
- (void)setupRemoteIO
{
AudioUnit audioUnit;
// Describe audio component
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
// Get component
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
// Get audio unit
OSStatus status = AudioComponentInstanceNew(inputComponent, &audioUnit);
if (status != noErr)
{
printf("AudioIO could not create new audio component: status = %i\n", status);
}
UInt32 enableIO;
AudioUnitElement inputBus = 1;
AudioUnitElement outputBus = 0;
//Disabling IO for recording
enableIO = 0;
AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, inputBus, &enableIO, sizeof(enableIO));
//Enabling IO for playback
enableIO = 1;
AudioUnitSetProperty(audioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Output, outputBus, &enableIO, sizeof(enableIO));
// initialize
status = AudioUnitInitialize(audioUnit);
if (status != noErr)
{
printf("AudioIO could not initialize audio unit: status = %i\n", status);
}
}
This is a known bug with Xcode (previous to 10.2) from macOS Mojave (I say known because it has happened to me a lot of time when playing video but also because when I was looking for it I found a lot of people having the same issue); althought I couldn't find any report from Apple.
Probably there can be some workaround depending on the environment, the way you launch the app, the version of Xcode and the version of macOS Mojave you have.
This will happen only in the simulator, and as you also said it won't happen on real device as most of the apps don't need microphone access for playing with Audio/Video features.
In the meantime this bug get resolved, you can try:
Going to "Security & Privacy" settings on your macOS
"Microphone" on the left panel
Then on the right panel disable the option for Xcode
Another thing you can try to get rid of the message is to change Hardware Audio Input to Internal Microphone:
Update in Xcode 10.2:
You’re now only prompted once to authorize microphone access to all simulator devices. (45715977)
I am implementing a C listener to Audio Session Interruption. When it is called for interruption, I would deactivate my audio session. Then when my app resumes, I would activate the audio session again. I have set a number of properties and category for my audio session, do I have to reset everything after re-activation?
Thanks in advance.
Some code for reference:
Initialization, setting category:
OSStatus error = AudioSessionInitialize(NULL, NULL, interuptListenerCallBack, (__bridge void *)(self));
UInt32 category = kAudioSessionCategory_PlayAndRecord;
error = AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, sizeof(category), &category);
if (error) printf("couldn't set audio category!");
//use speaker as default
UInt32 doChangeDefaultOutput = 1;
AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryDefaultToSpeaker, sizeof(doChangeDefaultOutput), &doChangeDefaultOutput);
//allow bluethoothInput
UInt32 allowBluetoothInput = 1;
AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryEnableBluetoothInput, sizeof(allowBluetoothInput), &allowBluetoothInput);
The interuptListenerCallBack is where I deactivate and reactive the Audio Session because of the interruption, using
OSStatus error = AudioSessionSetActive(false);
if (error) printf("couldn't deactivate audio session!");
Or
OSStatus error = AudioSessionSetActive(true);
if (error) printf("AudioSessionSetActive (true) failed");
If you are correctly using the Audio session interruption listener, then no, you should not have to reset the properties. You just need to make sure that you actually call kAudioSessionBeginInterruption and kAudioSessionEndInterruption. I am not sure what your listener looks like, but if you are doing something like this:
if (inInterruptionState == kAudioSessionBeginInterruption) {
AudioSessionSetActive(NO);
}
if (inInterruptionState == kAudioSessionEndInterruption) {
AudioSessionSetActive(YES);
}
And are following the rules of Audio Session, Then theoretically, you should not have to reset your properties.
I don't know what you are using the Audio Session for, but you could also pause and resume playback by using the:
kAudioSessionInterruptionType_ShouldResume
and
kAudioSessionInterruptionType_ShouldNotResume.
You can use these as stated in the Docs:
kAudioSessionInterruptionType_ShouldResume
Indicates that the interruption that has just ended was one for
which it is appropriate to immediately resume playback; for example,
an incoming phone call was rejected by the user.
Available in iOS 4.0 and later.
Declared in AudioSession.h.
kAudioSessionInterruptionType_ShouldNotResume
Indicates that the interruption that has just ended was one for which it is not appropriate to resume playback; for example, your app
had been interrupted by iPod playback.
Available in iOS 4.0 and later.
Declared in AudioSession.h.
You should read the docs because there is a lot of info in there about pausing, resuming, and handling interruptions for the AudioSession.
NOTE:
AudioSession has been deprecated since iOS7. Use AVAudioSession methods instead, or set Pause and Resume option by setting the constant AVAudioSessionInterruptionOptions or AVAudioSessionInterruptionType.
(Available since iOS 6)
I have a strange memory "leak" with AVAssetWriterInput appendSampleBuffer. I'm writing video and audio at the same time, so I have one AVAssetWriter with two inputs, one for video and one for audio:
self.videoWriter = [[[AVAssetWriter alloc] initWithURL:[self.currentVideo currentVideoClipLocalURL]
fileType:AVFileTypeMPEG4
error:&error] autorelease];
...
self.videoWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
outputSettings:videoSettings];
self.videoWriterInput.expectsMediaDataInRealTime = YES;
[self.videoWriter addInput:self.videoWriterInput];
...
self.audioWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeAudio
outputSettings:audioSettings];
self.audioWriterInput.expectsMediaDataInRealTime = YES;
[self.videoWriter addInput:self.audioWriterInput];
I start writing and everything works fine on the surface. The video and audio get written and are aligned, etc. However, I put my code through the Allocations Instrument and noticed the following:
The audio bytes are getting retained in memory, as I'll prove in a second. That's the ramp up in memory. The audio bytes are only released after I call [self.videoWriter endSessionAtSourceTime:...], which you see as the dramatic drop in memory usage. Here is my audio writing code, which is dispatched as a block onto a serial queue:
#autoreleasepool
{
// The objects that will hold the audio data
CMSampleBufferRef sampleBuffer;
CMBlockBufferRef blockBuffer1;
CMBlockBufferRef blockBuffer2;
size_t nbytes = numSamples * asbd_.mBytesPerPacket;
OSStatus status = noErr;
status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault,
data,
nbytes,
kCFAllocatorNull,
NULL,
0,
nbytes,
kCMBlockBufferAssureMemoryNowFlag,
&blockBuffer1);
if (status != noErr)
{
NLog(#"CMBlockBufferCreateWithMemoryBlock error at buffer 1");
return;
}
status = CMBlockBufferCreateContiguous(kCFAllocatorDefault,
blockBuffer1,
kCFAllocatorDefault,
NULL,
0,
nbytes,
kCMBlockBufferAssureMemoryNowFlag | kCMBlockBufferAlwaysCopyDataFlag,
&blockBuffer2);
if (status != noErr)
{
NSLog(#"CMBlockBufferCreateWithMemoryBlock error at buffer 2");
CFRelease(blockBuffer1);
return;
}
// Finally, create the CMSampleBufferRef
status = CMAudioSampleBufferCreateWithPacketDescriptions(kCFAllocatorDefault,
blockBuffer2,
YES, // Yes data is ready
NULL, // No callback needed to make data ready
NULL,
audioFormatDescription_,
1,
timestamp,
NULL,
&sampleBuffer);
if (status != noErr)
{
NSLog(#"CMAudioSampleBufferCreateWithPacketDescriptions error.");
CFRelease(blockBuffer1);
CFRelease(blockBuffer2);
return;
}
if ([self.audioWriterInput isReadyForMoreMediaData])
{
if (![self.audioWriterInput appendSampleBuffer:sampleBuffer])
{
NSLog(#"Couldn't append audio sample buffer: %d", numAudioCallbacks_);
}
} else {
NSLog(#"AudioWriterInput isn't ready for more data.");
}
// One release per create
CFRelease(blockBuffer1);
CFRelease(blockBuffer2);
CFRelease(sampleBuffer);
}
As you can see, I'm releasing each buffer once per create. I've traced the "leak" down to the line where the audio buffers are appended:
[self.audioWriterInput appendSampleBuffer:sampleBuffer]
I proved this to myself by commenting out that line, after which I get the following "leak-free" Allocations graph (although the recorded video now has no audio now, of course):
I tried one other thing, which is to add back the appendSamplebuffer line and instead double-release blockBuffer2:
CFRelease(blockBuffer1);
CFRelease(blockBuffer2);
CFRelease(blockBuffer2); // Double release to test the hypothesis that appendSamplebuffer is retaining this
CFRelease(sampleBuffer);
Doing this did not cause a double-free, indicating that blockBuffer2's retain count at that point is 2. This produced the same "leak-free" allocations graph, with the exception that when I called [self.videoWriter endSessionAtSourceTime:...], I get a crash from a double-release (indicating that self.videoWriter is trying to release all of its pointers to the blockBuffer2s that have been passed in).
If instead, I try the following:
CFRelease(blockBuffer1);
CFRelease(blockBuffer2);
CMSampleBufferInvalidate(sampleBuffer); // Invalidate sample buffer
CFRelease(sampleBuffer);
then [self.audioWriterInput appendSampleBuffer:sampleBuffer] and the call to append video frames begin to fail for every call after that.
So my conclusion is that AVAssetWriter or AVAssetWriterInput is retaining blockBuffer2 until the video has finished recording. Obviously, this can cause real memory problems if the video is recording for long enough. Am I doing something wrong?
Edit: The audio bytes I'm getting are PCM format, whereas the video format I'm writing is MPEG4 and the audio format for that video is MPEG4AAC. Is it possible that the video writer is performing the PCM --> AAC format on the fly, and that's why it's getting buffered?
Since you have been waiting a month for an answer I'll give you a less than ideal but workable answer.
You could use the ExtendedAudioFile functions to write a separate file. Then you could just playback the video and audio together with an AVComposition. I think you might be able to use AVFoundation to composite the caf and video together without reencoding if you need them composited at the end of recording.
That will get you off and running, then you can solve the memory leak at your leisure.
I'm looking to build an incredibly simple application for iOS with a button that starts and stops an audio signal. The signal is just going to be a sine wave, and it's going to check my model (an instance variable for the volume) throughout its playback and change its volume accordingly.
My difficulty has to do with the indefinite nature of the task. I understand how to build tables, fill them with data, respond to button presses, and so on; however, when it comes to just having something continue on indefinitely (in this case, a sound), I'm a little stuck! Any pointers would be terrific!
Thanks for reading.
Here's a bare-bones application which will play a generated frequency on-demand. You haven't specified whether to do iOS or OSX, so I've gone for OSX since it's slightly simpler (no messing with Audio Session Categories). If you need iOS, you'll be able to find out the missing bits by looking into Audio Session Category basics and swapping the Default Output audio unit for the RemoteIO audio unit.
Note that the intention of this is purely to demonstrate some Core Audio / Audio Unit basics. You'll probably want to look into the AUGraph API if you want to start getting more complex than this (also in the interest of providing a clean example, I'm not doing any error checking. Always do error checking when dealing with Core Audio).
You'll need to add the AudioToolbox and AudioUnit frameworks to your project to use this code.
#import <AudioToolbox/AudioToolbox.h>
#interface SWAppDelegate : NSObject <NSApplicationDelegate>
{
AudioUnit outputUnit;
double renderPhase;
}
#end
#implementation SWAppDelegate
- (void)applicationDidFinishLaunching:(NSNotification *)aNotification
{
// First, we need to establish which Audio Unit we want.
// We start with its description, which is:
AudioComponentDescription outputUnitDescription = {
.componentType = kAudioUnitType_Output,
.componentSubType = kAudioUnitSubType_DefaultOutput,
.componentManufacturer = kAudioUnitManufacturer_Apple
};
// Next, we get the first (and only) component corresponding to that description
AudioComponent outputComponent = AudioComponentFindNext(NULL, &outputUnitDescription);
// Now we can create an instance of that component, which will create an
// instance of the Audio Unit we're looking for (the default output)
AudioComponentInstanceNew(outputComponent, &outputUnit);
AudioUnitInitialize(outputUnit);
// Next we'll tell the output unit what format our generated audio will
// be in. Generally speaking, you'll want to stick to sane formats, since
// the output unit won't accept every single possible stream format.
// Here, we're specifying floating point samples with a sample rate of
// 44100 Hz in mono (i.e. 1 channel)
AudioStreamBasicDescription ASBD = {
.mSampleRate = 44100,
.mFormatID = kAudioFormatLinearPCM,
.mFormatFlags = kAudioFormatFlagsNativeFloatPacked,
.mChannelsPerFrame = 1,
.mFramesPerPacket = 1,
.mBitsPerChannel = sizeof(Float32) * 8,
.mBytesPerPacket = sizeof(Float32),
.mBytesPerFrame = sizeof(Float32)
};
AudioUnitSetProperty(outputUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
0,
&ASBD,
sizeof(ASBD));
// Next step is to tell our output unit which function we'd like it
// to call to get audio samples. We'll also pass in a context pointer,
// which can be a pointer to anything you need to maintain state between
// render callbacks. We only need to point to a double which represents
// the current phase of the sine wave we're creating.
AURenderCallbackStruct callbackInfo = {
.inputProc = SineWaveRenderCallback,
.inputProcRefCon = &renderPhase
};
AudioUnitSetProperty(outputUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Global,
0,
&callbackInfo,
sizeof(callbackInfo));
// Here we're telling the output unit to start requesting audio samples
// from our render callback. This is the line of code that starts actually
// sending audio to your speakers.
AudioOutputUnitStart(outputUnit);
}
// This is our render callback. It will be called very frequently for short
// buffers of audio (512 samples per call on my machine).
OSStatus SineWaveRenderCallback(void * inRefCon,
AudioUnitRenderActionFlags * ioActionFlags,
const AudioTimeStamp * inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData)
{
// inRefCon is the context pointer we passed in earlier when setting the render callback
double currentPhase = *((double *)inRefCon);
// ioData is where we're supposed to put the audio samples we've created
Float32 * outputBuffer = (Float32 *)ioData->mBuffers[0].mData;
const double frequency = 440.;
const double phaseStep = (frequency / 44100.) * (M_PI * 2.);
for(int i = 0; i < inNumberFrames; i++) {
outputBuffer[i] = sin(currentPhase);
currentPhase += phaseStep;
}
// If we were doing stereo (or more), this would copy our sine wave samples
// to all of the remaining channels
for(int i = 1; i < ioData->mNumberBuffers; i++) {
memcpy(ioData->mBuffers[i].mData, outputBuffer, ioData->mBuffers[i].mDataByteSize);
}
// writing the current phase back to inRefCon so we can use it on the next call
*((double *)inRefCon) = currentPhase;
return noErr;
}
- (void)applicationWillTerminate:(NSNotification *)notification
{
AudioOutputUnitStop(outputUnit);
AudioUnitUninitialize(outputUnit);
AudioComponentInstanceDispose(outputUnit);
}
#end
You can call AudioOutputUnitStart() and AudioOutputUnitStop() at will to start/stop producing audio. If you want to dynamically change the frequency, you can pass in a pointer to a struct containing both the renderPhase double and another one representing the frequency you want.
Be careful in the render callback. It's called from a realtime thread (not from the same thread as your main run loop). Render callbacks are subject to some fairly strict time requirements, which means that there's many things you Should Not Do in your callback, such as:
Allocate memory
Wait on a mutex
Read from a file on disk
Objective-C messaging (Yes, seriously.)
Note that this is not the only way to do this. I've only demonstrated it this way since you've tagged this core-audio. If you don't need to change the frequency you can just use the AVAudioPlayer with a pre-made sound file containing your sine wave.
There's also Novocaine, which hides a lot of this verbosity from you. You could also look into the Audio Queue API, which works fairly similar to the Core Audio sample I wrote but decouples you from the hardware a little more (i.e. it's less strict about how you behave in your render callback).
For some reason, it seems that stopping at a breakpoint during debugging will kill my audio queue playback.
AudioQueue will be playing audio
output.
Trigger a breakpoint to
pause my iPhone app.
Subsequent
resume, audio no longer gets played.
( However, AudioQueue callback
functions are still getting called.)
( No AudioSession or AudioQueue
errors are found.)
Since the debugger pauses the application (rather than an incoming phone call, for example) , it's not a typical iPhone interruption, so AudioSession interruption callbacks do not get triggered like in this solution.
I am using three AudioQueue buffers at 4096 samples at 22kHz and filling them in a circular manner.
Problem occurs for both multi-threaded and single-threaded mode.
Is there some known problem that you can't pause and resume AudioSessions or AudioQueues during a debugging session?
Is it running out of "queued buffers" and it's destroying/killing the AudioQueue object (but then my AQ callback shouldn't trigger).
Anyone have insight into inner workings of iPhone AudioQueues?
After playing around with it for the last several days, before posting to StackOverflow, I figured out the answer just today. Go figure!
Just recreate the AudioQueue again by calling my "preparation functions"
SetupNewQueue(mDataFormat.mSampleRate, mDataFormat.mChannelsPerFrame);
StartQueue(true);
So detect when your AudioQueue may have "died". In my case, I would be writing data into an input buffer to be "pulled" by AudioQueue callback. If it doesn't occur in a certain time, or after X number of bytes of input buffer have been filled, I then recreate the AudioQueue.
This seems to solve the issue where "halts/fails" audio when you hit a debugging breakpoint.
The simplified versions of these functions are the following:
void AQPlayer::SetupNewQueue(double inSampleRate, UInt32 inChannelsPerFrame)
{
//Prep AudioStreamBasicDescription
mDataFormat.mSampleRate = inSampleRate;
mDataFormat.SetCanonical(inChannelsPerFrame, YES);
XThrowIfError(AudioQueueNewOutput(&mDataFormat, AQPlayer::AQBufferCallback, this,
NULL, kCFRunLoopCommonModes, 0, &mQueue), "AudioQueueNew failed");
// adjust buffer size to represent about a half second of audio based on this format
CalculateBytesForTime(mDataFormat, kBufferDurationSeconds, &mBufferByteSize, &mNumPacketsToRead);
ctl->cmsg(CMSG_INFO, VERB_NOISY, "AQPlayer Buffer Byte Size: %d, Num Packets to Read: %d\n", (int)mBufferByteSize, (int)mNumPacketsToRead);
mBufferWaitTime = mNumPacketsToRead / mDataFormat.mSampleRate * 0.9;
XThrowIfError(AudioQueueAddPropertyListener(mQueue, kAudioQueueProperty_IsRunning, isRunningProc, this), "adding property listener");
//Allocate AQ buffers (assume we are using CBR (constant bitrate)
for (int i = 0; i < kNumberBuffers; ++i) {
XThrowIfError(AudioQueueAllocateBuffer(mQueue, mBufferByteSize, &mBuffers[i]), "AudioQueueAllocateBuffer failed");
}
...
}
OSStatus AQPlayer::StartQueue(BOOL inResume)
{
// if we are not resuming, we also should restart the file read index
if (!inResume)
mCurrentPacket = 0;
// prime the queue with some data before starting
for (int i = 0; i < kNumberBuffers; ++i) {
mBuffers[i]->mAudioDataByteSize = mBuffers[i]->mAudioDataBytesCapacity;
memset( mBuffers[i]->mAudioData, 0, mBuffers[i]->mAudioDataByteSize );
XThrowIfError(AudioQueueEnqueueBuffer( mQueue,
mBuffers[i],
0,
NULL ),"AudioQueueEnqueueBuffer failed");
}
OSStatus status;
status = AudioSessionSetActive( true );
XThrowIfError(status, "\n\n*** AudioSession failed to become active *** \n\n");
status = AudioQueueStart(mQueue, NULL);
XThrowIfError(status, "\n\n*** AudioQueue failed to start *** \n\n");
return status;
}