Calling MusicDeviceMIDIEvent from the audio unit's render thread - ios

There's one thing I don't understand about MusicDeviceMIDIEvent. In every single example I ever seen (searched Github and Apple examples) it was always used from the main thread. Now, in order to use the sample offset parameter the documentation states:
inOffsetSampleFrame:
If you are scheduling the MIDI Event from the audio unit's render thread, then you can supply a
sample offset that the audio unit may apply when applying that event in its next audio unit render.
This allows you to schedule to the sample, the time when a MIDI command is applied and is particularly
important when starting new notes. If you are not scheduling in the audio unit's render thread,
then you should set this value to 0
Still, even in the most simple case, in which you only have a sampler audio unit and an io unit, how can you schedule MIDI events from the audio unit's render thread since the sampler doesn't allow a render callback and even if it would (or if you use the io's callback just to tap in), it would feel hackish, since the render callback is not intended for schedule MIDI events?
How does one correctly calls this function from the audio unit's render thread?

A renderNotify callback is a perfect place to do scheduling from the render thread. You can even set the renderNotify on the MusicDevice itself. Here's what it might look like on an AUSampler.
OSStatus status = AudioUnitAddRenderNotify(sampler, renderNotify, sampler);
In this example I passed the sampler in as a reference via the inRefCon argument, and am just sending a note-on(144) to note 64 every 44100 samples, but in an application you would pass in a c struct to inRefCon with a reference to your midi device, and all the values you need to do your scheduling. Note the checking of the render flag for pre-render.
static OSStatus renderNotify(void * inRefCon,
AudioUnitRenderActionFlags * ioActionFlags,
const AudioTimeStamp * inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData) {
AudioUnit sampler = inRefCon;
if (ioActionFlags & kAudioUnitRenderAction_PreRender) {
for (int i = 0; i < inNumberFrames; i++) {
if (fmod(inTimeStamp->mSampleTime + i, 44000) == 0) {
MusicDeviceMIDIEvent(sampler,144, 64, 127, i); // i is the offset from render start, so use it for offset argument.
}
}
}
return noErr;
}

Related

Audio Unit increases render callback inNumberFrames when proximity sensor is covered

I'm using Linphone SDK for a VoIP iOS app. And I found the proximity sensor (the one that will dim your screen when you put the phone close to ear) affects the incoming voice badly.
I found The inBusNumber for input render callback will increase to 1024 when the proximity is covered, normally it's 256. When it happens it also cause about 180ms time gap that Audio Unit doesn't trigger this callback, which destroy Linphone's buffer strategy.
setup render callback:
AURenderCallbackStruct renderCallbackStruct;
renderCallbackStruct.inputProc = au_write_cb;
renderCallbackStruct.inputProcRefCon = card;
auresult=AudioUnitSetProperty (
card->io_unit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
outputBus,
&renderCallbackStruct,
sizeof (renderCallbackStruct)
);
In the render callback:
static OSStatus au_write_cb (
void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
//it changes to 1024 when proximity sensor is triggered
UInt32 inNumberFrames,
AudioBufferList *ioData
) {}
In my understanding the inNumberFrames will only change in circumstance of switching playback devices (such as switching earphone to bluetooth). Is there any way that I can fix this figure when the proximity sensor is triggered?
I also try to set kAudioUnitProperty_MaximumFramesPerSlice to 256 and setPreferredIOBufferDuration of audio session, but both don't work.
I download Apple official demo named Speakerbox, and I found their render callback's inNumberFrames persists to 256 no matter how I trigger the proximity sensor. I compared the Apple's code and mine but I can't find any difference that may cause this. Appreciate any help, thank you.
Your understanding is incorrect. iOS can change inNumberFrames for other reasons, such as for currently running app life cycle state(s), and for power management changes. An app's audio unit buffer management strategy needs to tolerate such changes in buffer size, such as by audio dropout/error concealment or resynchronization.
As for differences in iOS buffer size behavior, those might be modified by the app's choice of audio unit, audio session type and options, and background mode options.

Can't add render callback to output unit

I'm writing an app that should mix several sounds from disk and save resulting file to disk. I'm trying to use Audio Units.
I used Apple's MixerHost as a base for my app. It has Multichannel mixer connected to Remote I/O. When I'm trying to add render callback to remote IO I've got error -10861 "The attempted connection between two nodes cannot be made." when call AUGraphConnectNodeInput(...).
What I'm doing wrong? What's the right way to mix and record file to disk?
callback stub:
static OSStatus saveToDiskRenderCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
return noErr;
}
adding callback to Remote I/O Unit:
AURenderCallbackStruct saveToDiskCallbackStruct;
saveToDiskCallbackStruct.inputProc = &saveToDiskRenderCallback;
result = AUGraphSetNodeInputCallback (
processingGraph,
iONode,
0,
&saveToDiskCallbackStruct
);
error here:
result = AUGraphConnectNodeInput (
processingGraph,
mixerNode, // source node
0, // source node output bus number
iONode, // destination node
0 // desintation node input bus number
);
You are confused on how audio units works.
The node input callback (as set by AUGraphSetNodeInputCallback) and the node input connection (as set by AUGraphConnectNodeInput) are both on the same input side of your remote IO unit. It looks you believe that the input callback will be the output of your graph. This is wrong.
AUGraph offers two paths to feed the input of an AudioUnit:
Either from another upstream node (AUGraphConnectNodeInput)
or from a custom callback (AUGraphSetNodeInputCallback),
So you can't set them both simulatenously, it has no meaning.
Now two possibilities
1) Real time monitoring
This is not what you describe but this is the easier to get from where you are. So I assume you want to listen to the mix on the Remote I/O while it is being processed (in real time).
Then Read this
2) offline rendering
If you don't plan to listen in real time (which is what I understood first from your description), then the remote IO has nothing to do here since its purpose is to talk to a physical output. Then read that. It replaces the remote I/O unit with a Generic Output Unit. Be careful that the graph is not run in the same way.

Synchronising with Core Audio Thread

I am using the render callback of the ioUnit to store the audio data into a circular buffer:
OSStatus ioUnitRenderCallback(
void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
OSStatus err = noErr;
AMNAudioController *This = (__bridge AMNAudioController*)inRefCon;
err = AudioUnitRender(This.encoderMixerNode->unit,
ioActionFlags,
inTimeStamp,
inBusNumber,
inNumberFrames,
ioData);
// Copy the audio to the encoder buffer
TPCircularBufferCopyAudioBufferList(&(This->encoderBuffer), ioData, inTimeStamp, kTPCircularBufferCopyAll, NULL);
return err;
}
I then want to read the bytes out of the circular buffer, feed them to libLame and then to libShout.
I have tried starting a thread and using NSCondition to make it wait until data is available but this causes all sorts of issues due to using locks on the Core Audio callback.
What would be the recommended way to do this?
Thanks in advance.
More detail on how I implemented Adam's answer
I ended up taking Adam's advice and implemented it like so.
Producer
I use TPCircularBufferProduceBytes in the Core Audio Render callback to add the bytes to the circular buffer. In my case I have non-interleaved audio data so I ended up using two circular buffers.
Consumer
I spawn a new thread using pthread_create
Within the new thread create a new CFTimer and add it to the current
CFRunLoop (an interval of 0.005 seconds appears to work well)
I tell the current CFRunLoop to run
Within my timer callback I encode the audio and send it to the server (returning quickly if no data is buffered)
I also have a buffer size of 5MB which appears to work well (2MB was giving me overruns). This does seem a bit high :/
Use a repeating timer (NSTimer or CADisplayLink) to poll your lock-free circular buffer or FIFO. Skip doing work if there is not enough data in the buffer, and return (to the run loop). This works because you know the sample rate with high accuracy, and how much data you prefer or need to handle at a time, so can set the polling rate just slightly faster, to be on the safe side, but still be very close to the same efficiency as using conditional locks.
Using semaphores or locks (or anything else with unpredictable latency) in a real-time audio thread callback is not recommended.
You're on the right track, but you don't need NSCondition. You definitely don't want to block. The circular buffer implementation you're using is lock free and should do the trick. In the audio render callback, put the data into the buffer by calling TPCircularBufferProduceBytes. Then in the reader context (a timer callback is good, as hotpaw suggests), call TPCircularBufferTail to get the tail pointer (read address) and number of available bytes to read, and then call TPCircularBufferConsume to do the actual reading. Now you've done the transfer without taking any locks. Just make sure the buffer you allocate is large enough to handle the worst-case condition where your reader thread gets held off by the os for whatever reason, otherwise you can hit a buffer overrun condition and will lose data.

iOS - Accessing generated signal from Audio Unit render callback

I have an iOS/Objective-C program that uses a single audio unit to play a generated signal when a button is pressed. I'd like to add functionality such that:
a) When the button is first pressed, a signal is generated in some kind of numeric array.
b) The audio then begins, and the render callback accesses (and plays) that generated signal.
Given my current code, I feel like these additions will just be a few lines, but I'm having trouble with the syntax, which variable types to use, how to track the current sample, and so on. I've included the related code as it is now:
The button press:
- (IBAction)startPressed:(id)sender {
[self setupAudioPlayer];
[self createSignal];
[self playAudio];
}
A line from setupAudioPlayer:
input.inputProcRefCon=&mySignal; // mySignal is an instance var
The audio creation:
-(void)createSignal{
int beepLength=0.020*Fs; // Fs is sampling frequency
float beepFrequency=440; // Hz
// Declare some kind of numeric array "mySignal", which is an instance var.
mySignal=...?
// Generate audio signal (pure tone)
for (int i=1; i<=beepLength; i++) {
float t=i/Fs;
mySignal[i]=sinf(2*M_PI*beepFrequency*t);
}
}
The render callback:
OSStatus RenderTone(
void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
const int channel1 = 0;
Float32 *buffer = (Float32 *)ioData->mBuffers[channel1].mData;
// This is where things get hazy
Float32 *mySignal=(Float32 *)inRefCon;
for (UInt32 frame = 0; frame < inNumberFrames; frame++)
{
buffer[frame]=mySignal[?];
}
return noErr;
}
So, to summarize my questions: How should mySignal be defined? How do I access this instance variable from RenderTone (my 'hazy' code above is just a guess)? How can I track the current sample in RenderTone? Is there anything else missing/wonky in this approach?
Thanks for reading and for any help, really appreciated!
(I have seen sample code that passes a reference to the view controller's instance into the render callback, and then accesses the instance variables that way. However, perhaps mistakenly, I read elsewhere that this wasn't good form as it may involve too much computational overhead for a callback with such strict timing requirements.)
Since you're generating the frames from an algebraic function, why don't you simply follow Matt Gallagher's example? In brief: just move the function inside the render callback and transfer the parameters through the vc instance.
Generally speaking your choices are limited for passing data to a callback that has a pre-defined form. I'm probably the last person to counsel on good form in Objective C, but one of the few options is to use globals.
You could pass mySignal array (or else the frequency) as a global. Not the most 'elegant' object-oriented solution, but one that will work and avoid all the O.O. frou-frou overhead. Seems only appropriate to use a C-based solution, since the render callback is at base a C function.
As to "tracking," not quite sure what you mean, but in my own work with generating tones, I've initialized a remainingCycles global with the tone-length (in frame cycles = length in seconds * Fs or sampleRate whatever you want to call it) and decrementing each pass through the frame loop; when the number hits zero, you end the tone. (Of course, you could use an instance variable instead of a global.)
Maybe this violates the Canons of Object-Oriented Coding, but at the end of the day, you just need to get the job done.

iOS Audio unit get current time stamp

I'm using audio unit (iOS) to play music from file. How do I get the current time stamp of the music I m playing?
I found that there is a variable call "inTimeStamp" of type AudioTimeStamp in the playbackCallback function. Is it the right place i look for the current time stamp?
Here you are:
AudioTimeStamp ts;
UInt32 size = sizeof(ts);
AudioUnitGetProperty(THIS->audioPlayerUnit,
kAudioUnitProperty_CurrentPlayTime,
kAudioUnitScope_Global,
0, &ts, &size);
NSLog(#"TS%f", ts.mSampleTime);
A better way to get the seconds is to add
THIS->currentTime = ts.mSampleTime / THIS.streamFormatDescription.mSampleRate;
loretoparisi's answer is not suited for all scenes. If you're using kAudioUnitSubType_AudioFilePlayer, then try his answer; but if you're using other audio unit types such as RemoteIO unit, you need to set a global variable that stores the frame count that the audio unit has rendered, and update the value in every render cycle.
player.progress += inNumberFrames/player.canonicalFormat.mSampleRate;

Resources