This is a pretty "minutia" question regarding timing...
I'm using iOS's RemoteIO audio unit to do things. Just wonder how exactly system handles the timing: after calling AudioOutputUnitStart(), the unit should on "on", then render callbacks will be pulled by downstream units. Allow me to guess:
Possibility 1: the next render callback happens right after the execution of AudioOutputUnitStart(), then it goes on
Possibility 2: the system has its own render callback rhythm. After calling AudioOutputUnitStart(), the next render callback catches on one of the system's "next" tick, then start from there
1 or 2? or there's 3? Thanks in advance!
The audio latency seems to depend on the specific device model, audio session and options, requested sample rate and buffer size, and whether any other audio (background or recently closed app) is or has recently been playing or recording on the system. Whether or not the internal audio amplifier circuits (etc.) need to be powered up or are already turned on may make the biggest difference. Requesting certain sample rates seems to also cause extra time due to the buffering potentially needed by the OS resampling and mixer code.
So likely (2) or (3).
The best way to minimize latency when using RemoteIO is to request very short buffers (1 to 6 mS) in the audio session setup, start the audio session and Audio Unit way ahead of time (at app startup, view load, etc.), then fill the callback buffers with zeros (or discard recorded callback data) until you need sound.
Related
Inside my iOS 8.0. App I need to apply some custom audio processing on (non-realtime) audio playback. Typically, the audio comes from a device-local audio file.
Currently, I use MTAudioProcessingTap on a AVMutableAudioMix. Inside the process callback I then call my processing code. In certain cases this processing code may produce more samples than the amount of samples being passed in and I wonder what's the best way to handle this (think time stretching effect for example)
The process callback takes an incoming CMItemCount *numberFramesOut argument that signals the amount of outgoing frames. For in-place processing where the amount of incoming frames and outgoing frames is identical this is no problem. In the case where my processing generates more samples I need a way to get the playback going until my output buffers are emptied.
Is MTAudioProcessingTap the right choice here anyway?
MTAudioProcessingTap does not support changing the number of samples between the input and the output (to skip silences for instance).
You will need a custom audio unit graph for this.
A circular buffer/fifo is one of the most common methods to intermediate between different producer and consumer rates, as long as the long term rate is the same. If long term, you plan on producing more samples than are played, you may need to occasionally temporarily stop producing samples, while still playing, in order not to fill up all of the buffer or the systems memory.
I'm trying to realize an app which plays a sequence of tones in a loop.
Actually, I use OpenAL and my experiences with such framework are positive, as I can perform a sound pitch also.
Here's the scenario:
load a short sound (3 seconds) from a CAF file
play that sound in a loop and perform a sound shift also.
This works well, provided that the tact rate isn't too high - I mean a time of more than 10 milliseconds per tone.
Anyhow, my NSTimer (which embeds my sound sequence to play) should be configurable - and as soon as my tact rate increases (I mean less than 10 ms per tone), the sound is no more echoed correctly - even some tones are dropped in an obvious random way.
It seems that real time sound processing becomes an issue.
I'm still a novice in IOS programming, but I believe that Apple sets a limit concerning time consumption and/or semaphore.
Now my questions:
OpenAL is written in C - until now, I didn't understand the whole code and philosophy behind that framework. Is there a possibility to resolve my above mentioned problem making some modifications - I mean setting flags/values or overwriting certain methods?
If not, do you know another IOS sound framework more appropriate for such kind of real time sound processing?
Many thanks in advance!
I know that it deals with a quite extraordinary and difficult problem - maybe s.o. of you has resolved a similar one? Just to emphasize: sound pitch must be guaranteed!
It is not immediately clear from the explanation precisely what you're trying to achieve. Some code is expected.
However, your use of NSTimer to sequence audio playback is clearly problematic. It is neither intended as a reliable nor a high resolution timer.
NSTimer delivers events through a run-loop queue - probably your application's main queue - where they content with user interface events.
As the main thread is not a real-time thread, it may not even be scheduled to run for some time.
There may be quantisation effects on with the delay you requested, meaning your events effectively round to zero clock ticks and get scheduled immediately.
Perioidic timers have deleterious effects on battery life. iOS and MacOSX both take steps to reduce their impact by timer coalescing
The clock you should be using for sequencing events is the playback sample clock - which is available in the render handler of whatever framework you use. As well as being reliable this is efficient as well, as the render handler will be running periodically anyway, and in a real-time thread.
I'm doing an audio app where I need to manipulate audio buffers in real time for different sounds. I need a asynchronous clock running in the background that i can pull the time in various callbacks good to the millisecond for various audio manipulation purposes like the stop and start times of my audio playback, and when it records and stops recording.
I need it to be so such that the precision allows me to determine latency caused by the various processes and compensate for it in my code.
How would one implement such a clock in iOS?
If you are using the RemoteIO Audio Unit and requesting very short audio buffers from the Audio Session, then counting Remote callbacks and PCM samples within the callback buffer appears to be the most precise way to align audio with sub-millisecond resolution. Anything else will be off by the amount of latency between callbacks.
You can also use the timer in the mach_time.h header for fairly precise monotonic time measurement, but this won't be sync'd to audio playback rate, nor account for various latencies between subsystems.
If you have not already done so, I would definately advise you to look into the AVFoundation Framework, and take use of Core Audio. You can forget about the native NSTimer and NSDate options, considering your need of accuracy.
I'm writing an iOS application that will play audio instructions as one of it's features.
Every time the application wants to play audio it reads from a non-standard file and puts the resulting PCM data for that audio in a buffer in memory.
Even though I have that buffer with the PCM data, I'm having trouble getting the application to actually play the sound. After searching the iOS documentation, I started implementing an AudioUnit. The problem with this AudioUnit is the use of a render callback (as far as I know, the only way to output sound). From Apple's developer documentation:
… render callbacks have a strict performance requirement that you must
adhere to. A render callback lives on a real-time priority thread on
which subsequent render calls arrive asynchronously. The work you do
in the body of a render callback takes place in this time-constrained
environment. If your callback is still producing sample frames in
response to the previous render call when the next render call
arrives, you get a gap in the sound. For this reason you must not take
locks, allocate memory, access the file system or a network
connection, or otherwise perform time-consuming tasks in the body of a
render callback function
If I can't use locks inside the render callback method I can't be reading the buffer while writing in it. There is no opportunity to read the file and write to the buffer because the render callback will be accessing it constantly.
The only example I found actually generated the PCM data inside the render method, which I can't do.
Is this the only way of using AudioUnits (with an asynchronous render callback)?
Is there an alternative to playback PCM data from memory?
Using the RemoteIO Audio Unit might require having a separate data queue (fifo or circular buffer), outside the audio unit callback, which can pre-buffer up enough audio data from a file read, ahead of audio unit render callback, to meet worse case latencies. Then the render callback only needs to do a quick copy of the audio data, and then the update of a write-only flag that indicates that audio data was consumed.
An alternative built into iOS is to use the Audio Queue API, which does the pre-buffering for you. It allows your app to fill a number of larger audio buffers in the main run loop ahead of time. You still have to pre-buffer enough data to allow for the max of file, network, lock or other latencies.
Another strategy is to have alternative audio data to feed the real-time render callback if the file or network read didn't keep up, such as quickly creating an audio buffer that tapers to silence (and then un-tapering when real data starts arriving again).
I'm been happily synthesizing audio (at 44.1khz) and sending it out through the RemoteIO audio unit. It's come to my attention that my app's audio is "garbled" when going out via HDMI to a certain model of TV. It looks to me like the problem is related to the fact that this TV is looking for audio data at 48khz.
Here are some questions:
Does RemoteIO adopt the sample rate of whichever device it's outputting to? If I'm sending audio via HDMI to a device that asks for 48kz, do my RemoteIO callback buffers become 48khz?
Is there some tidy way to just force RemoteIO to still think in terms of 44.1khz, and be smart enough to perform any necessary sample rate conversions on its own, before it hands data off to the device?
If RemoteIO does indeed just defer to the device it's connected to, then presumably I need to do some sample rate conversion between my synthesis engine and remote IO. Is AudioConverterConvertComplexBuffer the best way to do this?
Fixed my problem. I was incorrectly assuming that the number of frames requested by the render callback would be a power of two. Changed my code to accommodate any arbitrary number of frames and all seems to work fine now.
If you want sample rate conversion, try using the Audio Queue API, or do the conversion within your own app using some DSP code.
Whether the RemoteIO buffer size or sample rate can be configured or not might depend on iOS device model, OS version, audio routes, background modes, etc., so an app must accomodate different buffer sizes and sample rates when using RemoteIO.