Synchronising with Core Audio Thread - ios

I am using the render callback of the ioUnit to store the audio data into a circular buffer:
OSStatus ioUnitRenderCallback(
void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
OSStatus err = noErr;
AMNAudioController *This = (__bridge AMNAudioController*)inRefCon;
err = AudioUnitRender(This.encoderMixerNode->unit,
ioActionFlags,
inTimeStamp,
inBusNumber,
inNumberFrames,
ioData);
// Copy the audio to the encoder buffer
TPCircularBufferCopyAudioBufferList(&(This->encoderBuffer), ioData, inTimeStamp, kTPCircularBufferCopyAll, NULL);
return err;
}
I then want to read the bytes out of the circular buffer, feed them to libLame and then to libShout.
I have tried starting a thread and using NSCondition to make it wait until data is available but this causes all sorts of issues due to using locks on the Core Audio callback.
What would be the recommended way to do this?
Thanks in advance.
More detail on how I implemented Adam's answer
I ended up taking Adam's advice and implemented it like so.
Producer
I use TPCircularBufferProduceBytes in the Core Audio Render callback to add the bytes to the circular buffer. In my case I have non-interleaved audio data so I ended up using two circular buffers.
Consumer
I spawn a new thread using pthread_create
Within the new thread create a new CFTimer and add it to the current
CFRunLoop (an interval of 0.005 seconds appears to work well)
I tell the current CFRunLoop to run
Within my timer callback I encode the audio and send it to the server (returning quickly if no data is buffered)
I also have a buffer size of 5MB which appears to work well (2MB was giving me overruns). This does seem a bit high :/

Use a repeating timer (NSTimer or CADisplayLink) to poll your lock-free circular buffer or FIFO. Skip doing work if there is not enough data in the buffer, and return (to the run loop). This works because you know the sample rate with high accuracy, and how much data you prefer or need to handle at a time, so can set the polling rate just slightly faster, to be on the safe side, but still be very close to the same efficiency as using conditional locks.
Using semaphores or locks (or anything else with unpredictable latency) in a real-time audio thread callback is not recommended.

You're on the right track, but you don't need NSCondition. You definitely don't want to block. The circular buffer implementation you're using is lock free and should do the trick. In the audio render callback, put the data into the buffer by calling TPCircularBufferProduceBytes. Then in the reader context (a timer callback is good, as hotpaw suggests), call TPCircularBufferTail to get the tail pointer (read address) and number of available bytes to read, and then call TPCircularBufferConsume to do the actual reading. Now you've done the transfer without taking any locks. Just make sure the buffer you allocate is large enough to handle the worst-case condition where your reader thread gets held off by the os for whatever reason, otherwise you can hit a buffer overrun condition and will lose data.

Related

Audio Unit increases render callback inNumberFrames when proximity sensor is covered

I'm using Linphone SDK for a VoIP iOS app. And I found the proximity sensor (the one that will dim your screen when you put the phone close to ear) affects the incoming voice badly.
I found The inBusNumber for input render callback will increase to 1024 when the proximity is covered, normally it's 256. When it happens it also cause about 180ms time gap that Audio Unit doesn't trigger this callback, which destroy Linphone's buffer strategy.
setup render callback:
AURenderCallbackStruct renderCallbackStruct;
renderCallbackStruct.inputProc = au_write_cb;
renderCallbackStruct.inputProcRefCon = card;
auresult=AudioUnitSetProperty (
card->io_unit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
outputBus,
&renderCallbackStruct,
sizeof (renderCallbackStruct)
);
In the render callback:
static OSStatus au_write_cb (
void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
//it changes to 1024 when proximity sensor is triggered
UInt32 inNumberFrames,
AudioBufferList *ioData
) {}
In my understanding the inNumberFrames will only change in circumstance of switching playback devices (such as switching earphone to bluetooth). Is there any way that I can fix this figure when the proximity sensor is triggered?
I also try to set kAudioUnitProperty_MaximumFramesPerSlice to 256 and setPreferredIOBufferDuration of audio session, but both don't work.
I download Apple official demo named Speakerbox, and I found their render callback's inNumberFrames persists to 256 no matter how I trigger the proximity sensor. I compared the Apple's code and mine but I can't find any difference that may cause this. Appreciate any help, thank you.
Your understanding is incorrect. iOS can change inNumberFrames for other reasons, such as for currently running app life cycle state(s), and for power management changes. An app's audio unit buffer management strategy needs to tolerate such changes in buffer size, such as by audio dropout/error concealment or resynchronization.
As for differences in iOS buffer size behavior, those might be modified by the app's choice of audio unit, audio session type and options, and background mode options.

Calling MusicDeviceMIDIEvent from the audio unit's render thread

There's one thing I don't understand about MusicDeviceMIDIEvent. In every single example I ever seen (searched Github and Apple examples) it was always used from the main thread. Now, in order to use the sample offset parameter the documentation states:
inOffsetSampleFrame:
If you are scheduling the MIDI Event from the audio unit's render thread, then you can supply a
sample offset that the audio unit may apply when applying that event in its next audio unit render.
This allows you to schedule to the sample, the time when a MIDI command is applied and is particularly
important when starting new notes. If you are not scheduling in the audio unit's render thread,
then you should set this value to 0
Still, even in the most simple case, in which you only have a sampler audio unit and an io unit, how can you schedule MIDI events from the audio unit's render thread since the sampler doesn't allow a render callback and even if it would (or if you use the io's callback just to tap in), it would feel hackish, since the render callback is not intended for schedule MIDI events?
How does one correctly calls this function from the audio unit's render thread?
A renderNotify callback is a perfect place to do scheduling from the render thread. You can even set the renderNotify on the MusicDevice itself. Here's what it might look like on an AUSampler.
OSStatus status = AudioUnitAddRenderNotify(sampler, renderNotify, sampler);
In this example I passed the sampler in as a reference via the inRefCon argument, and am just sending a note-on(144) to note 64 every 44100 samples, but in an application you would pass in a c struct to inRefCon with a reference to your midi device, and all the values you need to do your scheduling. Note the checking of the render flag for pre-render.
static OSStatus renderNotify(void * inRefCon,
AudioUnitRenderActionFlags * ioActionFlags,
const AudioTimeStamp * inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData) {
AudioUnit sampler = inRefCon;
if (ioActionFlags & kAudioUnitRenderAction_PreRender) {
for (int i = 0; i < inNumberFrames; i++) {
if (fmod(inTimeStamp->mSampleTime + i, 44000) == 0) {
MusicDeviceMIDIEvent(sampler,144, 64, 127, i); // i is the offset from render start, so use it for offset argument.
}
}
}
return noErr;
}

Can't add render callback to output unit

I'm writing an app that should mix several sounds from disk and save resulting file to disk. I'm trying to use Audio Units.
I used Apple's MixerHost as a base for my app. It has Multichannel mixer connected to Remote I/O. When I'm trying to add render callback to remote IO I've got error -10861 "The attempted connection between two nodes cannot be made." when call AUGraphConnectNodeInput(...).
What I'm doing wrong? What's the right way to mix and record file to disk?
callback stub:
static OSStatus saveToDiskRenderCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
return noErr;
}
adding callback to Remote I/O Unit:
AURenderCallbackStruct saveToDiskCallbackStruct;
saveToDiskCallbackStruct.inputProc = &saveToDiskRenderCallback;
result = AUGraphSetNodeInputCallback (
processingGraph,
iONode,
0,
&saveToDiskCallbackStruct
);
error here:
result = AUGraphConnectNodeInput (
processingGraph,
mixerNode, // source node
0, // source node output bus number
iONode, // destination node
0 // desintation node input bus number
);
You are confused on how audio units works.
The node input callback (as set by AUGraphSetNodeInputCallback) and the node input connection (as set by AUGraphConnectNodeInput) are both on the same input side of your remote IO unit. It looks you believe that the input callback will be the output of your graph. This is wrong.
AUGraph offers two paths to feed the input of an AudioUnit:
Either from another upstream node (AUGraphConnectNodeInput)
or from a custom callback (AUGraphSetNodeInputCallback),
So you can't set them both simulatenously, it has no meaning.
Now two possibilities
1) Real time monitoring
This is not what you describe but this is the easier to get from where you are. So I assume you want to listen to the mix on the Remote I/O while it is being processed (in real time).
Then Read this
2) offline rendering
If you don't plan to listen in real time (which is what I understood first from your description), then the remote IO has nothing to do here since its purpose is to talk to a physical output. Then read that. It replaces the remote I/O unit with a Generic Output Unit. Be careful that the graph is not run in the same way.

ios audio unit remoteIO playback while recording

I have been charged to add VOIP into an game (cross-platform, so can't use the Apple gamekit to do it).
For 3 or 4 days now, i'm trying to get my head wrap around audio unit and remoteIO...
I have overlooked tens of examples and such, but every time it is only applying a simple algorithm to the input PCM and play it back on the speaker.
According to Apple's documentation in order to do VOIP we should use kAudioSessionCategory_PlayAndRecord.
UInt32 audioCategory = kAudioSessionCategory_PlayAndRecord;
status = AudioSessionSetProperty(kAudioSessionProperty_AudioCategory,
sizeof(audioCategory),
&audioCategory);
XThrowIfError(status, "couldn't set audio category");
1) But it seems (to me) that playAndRecord will always play what coming from the mic (or more excatly the PerformThru callback // aurioTouch), am I wrong ?
I have the simplest callback, doing nothing but AURender
static OSStatus PerformThru(
void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
OSStatus err = AudioUnitRender(THIS->rioUnit, ioActionFlags, inTimeStamp, 1, inNumberFrames, ioData);
if (err)
printf("PerformThru: error %d\n", (int)err);
return err
}
From that callback I'm intending to send data to the peer (Not directly of course, but data will come from it)...
I do not see how I can play different output than the input, except maybe with 2 units, one recording, one playing, but it doesn't seems to be what Apple intended to (still accroding to the documentation).
And of course, I cannot find any documentation about it, audio unit is still pretty much un-documented...
Anyone would have an idea on what would be the best way to do it ?
I have not used VOIP or kAudioSessionCategory_PlayAndRecord. But if you want to record/transmit voice picked up from the mic and play back incoming data from network packages: Here is a good sample which included both mic and playback. Also if you have not read this doc from Apple, I would strongly recommend this.
In short: You need to create an AudioUnits instance. In it, configure two callbacks: one for mic and one for playback. The callback mic function will supply you the data that was picked up from the mic. You then can convert and transmit to other devices with whatever chosen network protocol. The playback callback function is where you supply the incoming data from other network devices to play back.
You can see this simple example. It describes how to use remote IO unit. After understanding this example, you should watch PJSIP's audio driver. These should help you implementing your own solution. Best of luck.

Most Effiecient Thread Implmentation IOS

I've been trying to find a thread implementation in IOS that suits my projects needs. So far I've failed to find an acceptable solution.
My Problem :
I need to read audio from up to 16 mp3 files on disk simultaneously.
What I have tried:
First off I tried using a NSTimer witch repeats. The timer was not fast enough and the audio would drop out when I played any more than 4 files.
Second I tried Using an NSThread with a priority of 1. The audio just about played correctly but the UI Became wholly unresponsive.
Finally I tried dispatching blocks using GCD in my callback whenever I needed more audio from a file. Again the audio would drop out but the UI was responsive.
In all three of the examples above I also tried dividing up the work load by creating 4 threads and having each thread handle 4 audio files each but this caused really bad synchronization problems with the audio.
Are there other thread options that I can try or do the above sum up what IOS has to offer?
Do you think that reading from 16 files from disk simultaneously is too much of a strain for the IOS system?
Is there a limit of how many threads IOS can handle?
To avoid making my question sound like a discussion I will summarize as follows.
What IOS thread technology is best suited for very frequent calling, quickly completing execution, that can be easily synchronized and will not impact on UI responsiveness.
Any anecdotal advice from solving a similar audio programming problem is also appreciated.
EDIT 1
This is some stripped down code I modelled on a suggestion from a so user. All I'm after solid advice on what setup is going to work best for me. Since my last post I tried NSThread and it does seem to leave me with audio dropouts. Also I tried using NSConditions so that my thread is wasting processing power when its not filling buffer but using these locks seems like a real bad idea for audio callbacks.
OSStatus channelMixerCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
AudioInfo = myaudio[inBusNumber];
if(myaudio.needsbufferfill==YES)
{
[refToSelf performSelector:#selector(GetAudioForItem:) onThread:engineDescribtion.producerthread withObject:myaudio waitUntilDone:false];
}
}
-(void) startthread
{
engineDescribtion.producerthread =[[NSThread alloc]initWithTarget:self selector:#selector(dosinglerunloop) object:nil];
[engineDescribtion.producerthread start];
}
-(void)dosinglerunloop
{
BOOL isstarted=YES;
NSAutoreleasePool *pool=[[NSAutoreleasePool alloc]init];
do {
[[NSRunLoop currentRunLoop]addPort:[NSMachPort port] forMode:NSDefaultRunLoopMode];
[[NSRunLoop currentRunLoop]runMode:NSDefaultRunLoopMode beforeDate:[NSDate distantFuture]];
} while (isstarted);
[pool release];
}
- (void)GetAudioForItem:(AudioInfo *)info
{
// use data in Audio Info to seek to
//corrent place in file
//and extract audio to buffers
}
Problem 0:
Your audio render callbacks should never lock. Example: Creating a single heap allocation will lock.
Your threads will all compete for the hardware. To keep the UI responsive, you should not have many highest priority threads (the audio playback should be the only one). Consider the number of cores, disks, etc you have available in your design.
If you still have issues once you have correctly fixed that: Loading short files into memory can offload some of the disk's demand to memory.
You should profile to determine what is actually the problem: It may be CPU or I/O. You may be simply missing your render deadlines and equating audio dropouts to "can't read fast enough". If you are using a lot of CPU, then Disk I/O may not be the problem. Decoding and performing sample rate conversion on 16 mp3 files can require relatively high CPU (as one example of the things you need to look for).
pthreads will be fastest, but will require some work to implement right. That really doesn't matter at this time because there seem to be a few high level issues yet and there are multiple APIs which should handle the task just fine.
Your program should be smart enough to detect when read buffers cannot be filled fast enough.
You are pre filling the buffers, correct?
Presumably, you are using a run loop?
Well, there's only one diskā€¦ So any solution that requires 16 simultaneous reads might be an issue. (Depending on if you're I/O bound or CPU bound.)
NSTimer is not going to get you consistent results.
I don't see any reason why NSThread would kill UI responsiveness, perhaps you had a bug.
I'm going with this system being disk-bound because 16 channels of MP3 is no problem CPU-wise on modern machines - how much rattling is coming from your box? I would probably be tempted to use just one thread to fill the empty buffers with the buffer sized to accommodate, (averageDiskLatency*(bytes/msec)*16*bodgeFactor) bytes of audio stream, (bodgeFactor means rounded up to 8K boundary and add a few 8K's). Whenever threads/callbacks/whatever empty a buffer and so start on the other one, they should queue the empty buffer to the disk read thread, (thread-safe producer-consumer queue), to get it filled up again. Probably, each buffer should include a 'fileControl' instance containing the the fileSpec, file handle, state variable for EOF etc, error string space and anything else needed for the read thread to work as well as the buffer space itself.
This design allows the disk to read nice, large chunks without being annoyingly preempted half-way through reads and being avoidably forced to move lumps of metal too often.
Rgds,
Martin
PS - If you haven't got one already, get an SSD - works wonders for multi-channel audio/video latency.

Resources