I am interested in adding effects to songs played from the iTunes library. I have constructed an auGraph as follows: AUFilePlayer -> Effects Unit -> Mixer -> RemoteIO. There is much emphasis on the use of data buffers and render callback when playing large audio files as part of efficient memory management. I have found in scattered sources that the AUFilePlayer (>iOS 5) somewhat reduces the need for a buffer. Given my setup using an AUFilePlayer, should my design still include a ring buffer and render callback?
In short, No. There is no need to add a buffer.
AUFilePlayer internally loads audio into a buffer and pulls from it as the graph requests audio.
Max
Related
I need to analyse chunks of audio data of (approximately) 1 second with a sample rate of 8kHz. Although the audio will be recorded in real time, it will only be used for detecting specific events. Hence, there are no strict latency requirements. What would be the best framework to use in this case?
I already started learning Core Audio and I worked through the book Learning Core Audio. With the minimal amount of Swift documentation available on the internet I was able to set up an AUGraph for iOS to record audio with the remote I/O audio unit and to get acces to the raw samples with the output render callback. Unfortunately, I got stuck to create chunks of 1 seconds of audio samples to perform the audio analysis. Could a custom AudioBufferList be used for this? Or could a large ringbuffer be implemented on the remote I/O audio unit (like it is required in case of a HAL audio unit)?
I also tried to adopt AVFoundation with AVAssetReader to obtain the audio chunks. Although I was able to obtain samples of a recorded audio signal, I did not succes in creating a buffer of 1 second (and I even don’t know whether it would be possible to do this in realtime). Would AVFoundation be a good choice in this situation anyhow?
I would appreciate any advice on this.
A main problem for me is the fact that I try to use Swift but that there is not much example code available and that there is even less documentation. I feel that it would be better to switch to Objective-C for audio programming, and to stop trying to get everything in Swift. I am curious whether this would be a better time investment?
For analyzing 1 second windows of audio samples, the simplest solution would be to use the Audio Queue API with a lock-free ring buffer (say around 2 seconds long) to record samples. You can use a repeating nstimer task to poll how full the buffer is, and emit 1 second chunks to a processing task when they become available.
Core Audio and the RemoteIO Audio Unit is for if you need much shorter data windows with latency requirements on the order a few milliseconds.
Core Audio is a C API.
Objective-C is an extension of C. I find that Objective-C is much nicer for working with core audio than swift.
I created a cross platform c lockless ring buffer. There is sample code that demonstrates setting up the ring, setting up the mic, playing audio, and reading and writing from the ring.
The ring records that last N number of seconds that you specify. Old data is overwritten by new data. So you specify that you want the latest 3 seconds recorded. The sample I show plays a sine wave while recording through the microphone. Every 7 seconds, it grabs the last 2 seconds of recorded audio.
Here is the complete sample code on github.
Inside my iOS 8.0. App I need to apply some custom audio processing on (non-realtime) audio playback. Typically, the audio comes from a device-local audio file.
Currently, I use MTAudioProcessingTap on a AVMutableAudioMix. Inside the process callback I then call my processing code. In certain cases this processing code may produce more samples than the amount of samples being passed in and I wonder what's the best way to handle this (think time stretching effect for example)
The process callback takes an incoming CMItemCount *numberFramesOut argument that signals the amount of outgoing frames. For in-place processing where the amount of incoming frames and outgoing frames is identical this is no problem. In the case where my processing generates more samples I need a way to get the playback going until my output buffers are emptied.
Is MTAudioProcessingTap the right choice here anyway?
MTAudioProcessingTap does not support changing the number of samples between the input and the output (to skip silences for instance).
You will need a custom audio unit graph for this.
A circular buffer/fifo is one of the most common methods to intermediate between different producer and consumer rates, as long as the long term rate is the same. If long term, you plan on producing more samples than are played, you may need to occasionally temporarily stop producing samples, while still playing, in order not to fill up all of the buffer or the systems memory.
I have run through an audio units tutorial for a sine wave generator and done a bit of reading, and I understand basically how it is working. What I would actually like to do for my app, is play a short sound file in response to some external event. These sounds would be about 1-2 seconds in duration and occur at a rate of about about 1-2 per second.
Basically where I am at right now is trying to figure out how to play an actual audio file using my audio unit, rather than generating a sine wave. So basically my question is, how do I get an audio unit to play an audio file?
Do I simply read bytes from the audio file into the buffer in the render callback?
(if so what class do I need to deal with to open / convert / decompress / read the audio file)
or is there some simpler method where I could maybe just hand off the entire buffer and tell it to play?
Any names of specific classes or APIs I will need to look at to accomplish this would be very helpful.
OK, check this:
http://developer.apple.com/library/ios/samplecode/MixerHost/Introduction/Intro.html
EDIT: That is a sample project. This page has detailed instructions with inline code to setup common configurations: http://developer.apple.com/library/ios/ipad/#DOCUMENTATION/MusicAudio/Conceptual/AudioUnitHostingGuide_iOS/ConstructingAudioUnitApps/ConstructingAudioUnitApps.html#//apple_ref/doc/uid/TP40009492-CH16-SW1
If you don't mind being tied to IOS 5+, you should look into AUFilePlayer. It is much easer then using the callbacks and you don't have to worry about setting up your own ring buffer (something that you would need to do if you want to avoid loading all of your audio data into memory on start-up)
I've searched around but haven't found any good examples or tutorials of saving audio out of a RemoteIO Audio Unit.
My setup: Using the MusicPlayer API, I have several AUSamplers -> MixerUnit -> RemoteIO
Audio playback works great. I would like to add functionality to save the audio output to a file. Would I do this in a render callback on the RemoteIO?
Any tips or pointers to example code much appreciated!
Due to the tight latency requirements of Audio Unit callbacks, one should not to do any synchronous file access (or any other calls that could potentially block, involve memory management or OS locking actions) inside the RemoteIO callback. Instead, just copy the audio data out to another buffer (a larger circular buffer for example), and set some state indicating how much data has been copied. Then, in another thread, when the amount of data is sufficient, write the contents of that buffer out to a file. This could be a raw PCM file, which can later be converted by AVAssetReader/Writer into another audio file type.
I'm writing an iOS application that will play audio instructions as one of it's features.
Every time the application wants to play audio it reads from a non-standard file and puts the resulting PCM data for that audio in a buffer in memory.
Even though I have that buffer with the PCM data, I'm having trouble getting the application to actually play the sound. After searching the iOS documentation, I started implementing an AudioUnit. The problem with this AudioUnit is the use of a render callback (as far as I know, the only way to output sound). From Apple's developer documentation:
… render callbacks have a strict performance requirement that you must
adhere to. A render callback lives on a real-time priority thread on
which subsequent render calls arrive asynchronously. The work you do
in the body of a render callback takes place in this time-constrained
environment. If your callback is still producing sample frames in
response to the previous render call when the next render call
arrives, you get a gap in the sound. For this reason you must not take
locks, allocate memory, access the file system or a network
connection, or otherwise perform time-consuming tasks in the body of a
render callback function
If I can't use locks inside the render callback method I can't be reading the buffer while writing in it. There is no opportunity to read the file and write to the buffer because the render callback will be accessing it constantly.
The only example I found actually generated the PCM data inside the render method, which I can't do.
Is this the only way of using AudioUnits (with an asynchronous render callback)?
Is there an alternative to playback PCM data from memory?
Using the RemoteIO Audio Unit might require having a separate data queue (fifo or circular buffer), outside the audio unit callback, which can pre-buffer up enough audio data from a file read, ahead of audio unit render callback, to meet worse case latencies. Then the render callback only needs to do a quick copy of the audio data, and then the update of a write-only flag that indicates that audio data was consumed.
An alternative built into iOS is to use the Audio Queue API, which does the pre-buffering for you. It allows your app to fill a number of larger audio buffers in the main run loop ahead of time. You still have to pre-buffer enough data to allow for the max of file, network, lock or other latencies.
Another strategy is to have alternative audio data to feed the real-time render callback if the file or network read didn't keep up, such as quickly creating an audio buffer that tapers to silence (and then un-tapering when real data starts arriving again).