Is there a way to capture the audio buffers that are being sent out of a remoteIOUnit to the speaker? I am rendering a couple of different loops on different threads to the same IOUnit ( one is a click, the other has music ), and would like to perform analysis on how the music is lining up with the click without having to filter out any noise coming from, say, using the microphone input. My math must be very accurate ( error less then 2ms ), so getting this post-mix buffer would be ideal.
Yes, you just add a render notify callback to the remoteIO with AudioUnitAddRenderNotify. You will then get four callbacks per buffer: input pre-render, input post-render, output pre-render, and output post-render. You just need to act on the appropriate ioActionFlags and inBusNumber.
AudioUnitAddRenderNotify(remoteIO, inputOutputTap, (__bridge void *)self);
OSStatus inputOutputTap (void * inRefCon,
AudioUnitRenderActionFlags * ioActionFlags,
const AudioTimeStamp * inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData) {
if (*ioActionFlags == kAudioUnitRenderAction_PostRender && inBusNumber == 0) {
MyObject *self = (__bridge MyObject *)inRefCon;
MyObjectDoTheThing(self,ioData,inTimeStamp);
}
return noErr;
}
Related
I use Audiokit (in Objective-C) for realtime audio processing. I feed a C++ algorithm through a tap or lazy tap where the buffer is being modified.
I thought that would be obvious but...how can I playback the modified buffer in the output? Are taps only for analysis?
[self->microphoneGain.avAudioNode installTapOnBus:0 bufferSize:1024 format:format block:^(AVAudioPCMBuffer * _Nonnull buffer, AVAudioTime * _Nonnull when) {
if (buffer.frameLength == 0) {
return;
}
// Process data -> return modified buffer
processData(buffer.floatChannelData[0], buffer.floatChannelData[1], buffer.frameLength);
// -> How to play back buffer?
}];
Furthermore, I can't get taps buffer size lower than 4800 samples. What would be my best option to get a better latency? I read about AUAudioUnit subclassing, render callback or realtime mode for AudioEngine, but I'm quite lost when trying to implement one of these with AudioKit. Thanks!
EDIT:
I managed to set a render callback which has apparently solved both of my problems.
AURenderCallbackStruct processingCallback;
processingCallback.inputProc = processingCalbackProc;
processingCallback.inputProcRefCon = (__bridge void *)(self);
OSStatus status = AudioUnitSetProperty(AudioKit.engine.outputNode.audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
0,
&processingCallback,
sizeof(processingCallback));
if(status != noErr) {
return false;
}
OSStatus processingCalbackProc (void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
__unsafe_unretained MyClass *self = (__bridge MyClass *)inRefCon;
printf("%u, ", (unsigned int)inNumberFrames); // -> low latency!
if (!ioData) ioData = self->audioBufferList;
OSStatus status = AudioUnitRender(AudioKit.engine.outputNode.audioUnit,
ioActionFlags,
inTimeStamp,
1,
inNumberFrames,
ioData);
if(status != noErr) { return status; }
// Get buffers
unsigned int inputChannels = 2;
float *buffer[inputChannels];
for (int i = 0; i < inputChannels; i++) {
buffer[i] = (float *)ioData->mBuffers[i].mData;
}
// Process data
processData(buffer[0], buffer[1], inNumberFrames);
return noErr;
}
Now I can easily get buffers as low as 256samples (probably even less but not needed in my case) and when buffer[n]are modified, it outputs the modified buffers.
Everything seems to be fine, I just hope this is the right approach.
I am recording the raw audio data using Remote I/O unit as given below:
static OSStatus performRender(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
OSStatus err = noErr;
if (*callbackStruct.audioChainIsBeingReconstructed == NO)
{
err = AudioUnitRender(callbackStruct.rioUnit, ioActionFlags, inTimeStamp, 1, inNumberFrames, ioData);
{
[callbackStruct.bufferManager CopyAudioDataToDrawBuffer:(Float32 *)ioData->mBuffers[0].mData frames:inNumberFrames];
}
}
return err;
}
What is the unit of measurement of this raw audio data ( mBuffers[0].mData, which is a void pointer if you look up the AudioBuffer structure ).
Is it Pascal or milliVolts (mV) ? Thanks in advance.
It is neither - it could one of a number of audio data formats Common ones are:
float with values -1.0 <= x <= 1.0
16-bit signed short with values -32768 < x < 32768
Where the min and max values are smallest and largest samples values that can be represented without clipping.
In order calculate the output voltage, you would need to determine the gain of the remainder of the audio system. You can assume that SPL in pascals has a more or less linear relationship with output power which is proportional to V^2.
I've successfully recorded audio from the microphone into an audio file using Audio Units with the help of openframeworks and this website http://atastypixel.com/blog/using-remoteio-audio-unit.
I want to be able to stream the file back to audio units and play the audio. According to Play an audio file using RemoteIO and Audio Unit I can use ExtAudioFileOpenURL and ExtAudioFileRead. However, how do I play audio data in my buffer?
This is what I currently have:
static OSStatus setupAudioFileRead() {
//construct the file destination URL
CFURLRef destinationURL = audioSystemFileURL();
OSStatus status = ExtAudioFileOpenURL(destinationURL, &audioFileRef);
CFRelease(destinationURL);
if (checkStatus(status)) { ofLog(OF_LOG_ERROR, "ofxiPhoneSoundStream: Couldn't open file to read"); return status; }
while( TRUE ) {
// Try to fill the buffer to capacity.
UInt32 framesRead = 8000;
status = ExtAudioFileRead( audioFileRef, &framesRead, &inputBufferList );
// error check
if( checkStatus(status) ) { break; }
// 0 frames read means EOF.
if( framesRead == 0 ) { break; }
//play audio???
}
return noErr;
}
From this author: http://atastypixel.com/blog/using-remoteio-audio-unit/, if you scroll down to the PLAYBACK section, try something like this:
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
// Notes: ioData contains buffers (may be more than one!)
// Fill them up as much as you can. Remember to set the size value in each buffer to match how
// much data is in the buffer.
for (int i=0; i < ioData->mNumberBuffers; i++)
{
AudioBuffer buffer = ioData->mBuffers[i];
// copy from your whatever buffer data to output buffer
UInt32 size = min(buffer.mDataByteSize, your buffer.size);
memcpy(buffer.mData, your buffer, size);
buffer.mDataByteSize = size; // indicate how much data we wrote in the buffer
// To test if your Audio Unit setup is working - comment out the three
// lines above and uncomment the for loop below to hear random noise
/*
UInt16 *frameBuffer = buffer.mData;
for (int j = 0; j < inNumberFrames; j++) {
frameBuffer[j] = rand();
}
*/
}
return noErr;
}
If you are only looking for recording from MIC to a file and play it back, the Apple's Speakhere sample is probably much more ready to use.
Basically,
1. Create a RemoteIO unit (See references about how to create RemoteIO);
Create a FilePlayer audio unit which is a dedicated audio unit to read an audio file and provide audio data in the file to output units, for example, the RemoteIO unit created in step 1. To actually use the FilePlayer, a lot of settings (specify which file to play, which part of the file to play, etc.) are needed to be done on the it;
Set kAudioUnitProperty_SetRenderCallback and kAudioUnitProperty_StreamFormat properties of the RemoteIO unit. The first property is essentially a callback function from which the RemoteIO unit pulls audio data and play it. The second property must be set in accordance to StreamFormat that supported by the FilePlayer. It can be derived from a get-property function invoked on the FilePlayer.
Define the callback set in step 3 where the most important thing to do is asking the FilePlayer to render into the buffer provided by the callback for which you will need to invoke AudioUnitRender() on the FilePlayer.
Finally start the RemoteIO unit to play the file.
Above is just a preliminary outline of basic things to do to play files using audio units on iOS. You can refer to Chris Adamson and Kevin Avila's Learning Core Audio for details.
It's a relatively simple approach that utilizes the audio unit mentioned in the Tasty Pixel blog. In the recording callback, instead of filling the buffer with data from the microphone, you could fill it with data from the file using ExtAudioFileRead. I'll try and paste an example below. Mind you this will just work for .caf files.
In the start method call an readAudio or initAudioFile function, something that just gets all the info about the file.
- (void) start {
readAudio();
OSStatus status = AudioOutputUnitStart(audioUnit);
checkStatus(status);
}
Now in the readAudio method you initialize the audio file reference as such.
ExtAudioFileRef fileRef;
void readAudio() {
NSString * name = #"AudioFile";
NSString * source = [[NSBundle mainBundle] pathForResource:name ofType:#"caf"];
const char * cString = [source cStringUsingEncoding:NSASCIIStringEncoding];
CFStringRef str = CFStringCreateWithCString(NULL, cString, kCFStringEncodingMacRoman);
CFURLRef inputFileURL = CFURLCreateWithFileSystemPath(kCFAllocatorDefault, str, kCFURLPOSIXPathStyle, false);
AudioFileID fileID;
OSStatus err = AudioFileOpenURL(inputFileURL, kAudioFileReadPermission, 0, &fileID);
CheckError(err, "AudioFileOpenURL");
err = ExtAudioFileOpenURL(inputFileURL, &fileRef);
CheckError(err, "ExtAudioFileOpenURL");
err = ExtAudioFileSetProperty(fileRef, kExtAudioFileProperty_ClientDataFormat, sizeof(AudioStreamBasicDescription), &audioFormat);
CheckError(err, "ExtAudioFileSetProperty");
}
Now that you have the Audio Data at hand, next step is pretty easy. In the recordingCallback read the data from the file instead of the mic.
static OSStatus recordingCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
// Because of the way our audio format (setup below) is chosen:
// we only need 1 buffer, since it is mono
// Samples are 16 bits = 2 bytes.
// 1 frame includes only 1 sample
AudioBuffer buffer;
buffer.mNumberChannels = 1;
buffer.mDataByteSize = inNumberFrames * 2;
buffer.mData = malloc( inNumberFrames * 2 );
// Put buffer in a AudioBufferList
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = buffer;
// Then:
// Obtain recorded samples
OSStatus err = ExtAudioFileRead(fileRef, &inNumberFrames, &bufferList);
// Now, we have the samples we just read sitting in buffers in bufferList
// Process the new data
[iosAudio processAudio:&bufferList];
// release the malloc'ed data in the buffer we created earlier
free(bufferList.mBuffers[0].mData);
return noErr;
}
This worked for me.
I am trying to write (what should be) a simple app that has a bunch of audio units in sequence in an AUGraph and then writes the output to a file. I added a callback using AUGraphAddRenderNotify. Here is my callback function:
OSStatus MyAURenderCallback(void *inRefCon,
AudioUnitRenderActionFlags *actionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
if (*actionFlags & kAudioUnitRenderAction_PostRender) {
ExtAudioFileRef outputFile = (ExtAudioFileRef)inRefCon;
ExtAudioFileWriteAsync(outputFile, inNumberFrames, ioData);
}
}
This sort of works. The file is playable and I can hear what I recorded but there is horrible amounts of static that makes it barely audible.
Does anybody know what is wrong with this? Or does anyone know of a better way to record the AUGraph output to a file?
Thanks for the help.
I had a epiphany with regards to Audio Units just now which helped me solve my own problem. I had a misconception about how audio unit connections and render callbacks work. I thought they were completely separate things but it turns out that a connection is just short hand for a render callback.
Doing an kAudioUnitProperty_MakeConnection from the output of audio unit A to the input of audio unit B is the same as doing kAudioUnitProperty_SetRenderCallback on the input of unit B and having the callback function call AudioUnitRender on the output of audio unit A.
I tested this by doing a make connection after setting my render callback and the render callback was no longer invoked.
Therefore, I was able to solve my problem by doing the following:
AURenderCallbackStruct callbackStruct = {0};
callbackStruct.inputProc = MyAURenderCallback;
callbackStruct.inputProcRefCon = mixerUnit;
AudioUnitSetProperty(ioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
0,
&callbackStruct,
sizeof(callbackStruct));
And them my callback function did something like this:
OSStatus MyAURenderCallback(void *inRefCon,
AudioUnitRenderActionFlags *actionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
AudioUnit mixerUnit = (AudioUnit)inRefCon;
AudioUnitRender(mixerUnit,
actionFlags,
inTimeStamp,
0,
inNumberFrames,
ioData);
ExtAudioFileWriteAsync(outputFile,
inNumberFrames,
ioData);
return noErr;
}
This probably should have been obvious to me but since it wasn't I'll bet there are others that were confused in the same way so hopefully this is helpful to them too.
I'm still not sure why I had trouble with the AUGraphAddRenderNotify callback. I will dig deeper into this later but for now I found a solution that seems to work.
Here is some sample code from Apple (the project is PlaySequence, but it isn't MIDI specific) that might help:
{
CAStreamBasicDescription clientFormat = CAStreamBasicDescription();
ca_require_noerr (result = AudioUnitGetProperty(outputUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output, 0,
&clientFormat, &size), fail);
size = sizeof(clientFormat);
ca_require_noerr (result = ExtAudioFileSetProperty(outfile, kExtAudioFileProperty_ClientDataFormat, size, &clientFormat), fail);
{
MusicTimeStamp currentTime;
AUOutputBL outputBuffer (clientFormat, numFrames);
AudioTimeStamp tStamp;
memset (&tStamp, 0, sizeof(AudioTimeStamp));
tStamp.mFlags = kAudioTimeStampSampleTimeValid;
int i = 0;
int numTimesFor10Secs = (int)(10. / (numFrames / srate));
do {
outputBuffer.Prepare();
AudioUnitRenderActionFlags actionFlags = 0;
ca_require_noerr (result = AudioUnitRender (outputUnit, &actionFlags, &tStamp, 0, numFrames, outputBuffer.ABL()), fail);
tStamp.mSampleTime += numFrames;
ca_require_noerr (result = ExtAudioFileWrite(outfile, numFrames, outputBuffer.ABL()), fail);
ca_require_noerr (result = MusicPlayerGetTime (player, ¤tTime), fail);
if (shouldPrint && (++i % numTimesFor10Secs == 0))
printf ("current time: %6.2f beats\n", currentTime);
} while (currentTime < sequenceLength);
}
}
Maybe try this. Copy the data from the audio unit callback to a long buffer. Play the buffer to test it, then write the entire buffer to a file after you have verified that the whole buffer is OK.
I'm using the following to play audio:
OSStatus PlayRenderCallback (
void * inRefCon,
AudioUnitRenderActionFlags * ioActionFlags,
const AudioTimeStamp * inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData)
{
EffectState *effectState = (EffectState*) inRefCon;
if(effectState->play)
{
XThrowIfError(ExtAudioFileRead(effectState->sourceFile, &inNumberFrames, ioData), "Read failed!");
if(inNumberFrames == 0)
{
return noErr;
}
Then I process the audio and send it to the buffer.
What I want to know is what do I put in the return to stop playback? I tried:
AudioOutputUnitStop(remoteIOUnit);
And that works, once. I can't start the unit again.
Thanks all,
Jim
It seems you cannot stop a copy of a remote io. Rather than copying I used only the rio in my struct and now it works.