I have an audio callback that I use to access a bufferList and analyse the audio.
I need to record this audio too. Firstly would it be wise to do the recording in the same callback?
e.g. memcpy(void *dest, ioData->mBuffers[0].mData, int byteCount);
Or should the recording have its own callback?
Either way, is this memcpy the correct way to do this and how would I write this audio to a file?
Should the totalByteCount be used with pointer arithmetic on the void * dest once the audio input completes and pass the data to a file writer?
What is the best way to record audio in a core-audio render callback?
I think you can have two different callbacks each for both input and output audio stream. Normally when you open a particular stream it could be input or output you specify the callback too. In the callback you can do all your audio processing provided that you can meeting the callback deadline otherwise there are chances that you may end up missing audio samples. A better way would be to use some kind of circular buffer and it the callback you just fill the buffer. You can do all the other processing in main thread (along with recording).
I'm not sure which audio framework you are using. I've used portaudio in my project and it worked fine. Portaudio also provide a lock free circular buffer which can be used inside callback without need for thread locking mechanism.
Following links might help you.
http://portaudio.com/docs/v19-doxydocs/paex__record_8c.html.
http://portaudio.com/docs/v19-doxydocs/paex_ocean_shore_8c.html
Related
I am using a VoiceProcessingIO audio unit for voip calls. However, when I set the loud speaker (setting the kAudioSessionOverrideAudioRoute_Speaker audio session property), the PCM data received in the input callback by calling AudioUnitRender has a very low volume.
For a voip call, it is actually fine. The interlocutor hears it fainter, but he hears it. However, I would like to save to disk a good quality version of the input audio, possibly a raw audio from the mic.
Is it actually possible? In my tests I have not be able to do it. When VoiceProcessingIO is in use, the audio from the input-callback is just very low. Perhaps, I can get the unprocessed audio from some other source? Note, VoiceProcessingIO must still be used during the voip call.
The same question on Apple's forum is thread-655091, it has been asked 1 year ago and it has no answers.
Closest questions on SO I found are Two audio units? and Effect before render callback?, but they are more concerned about the output of VoiceProcessingIO rather than the input.
An idea would be to add a parallel "raw" RemoteIO unit to get the audio from the mic, but both in Two audio units? and in apple-forum-110816, developers say it will not be possible to add another RemoteIO in parallel to the VoiceProcessingIO, because having set
the audio session category as PlayAndRecord and the audio mode as VoiceChat, RemoteIO will not function as usual. I have not had a chance to try it, but it seems possible.
Are there other strategies? Are there some "pre-render input callbacks" called before VoiceProcessingIO unit kicks in and processes the raw data from the mic?
Is it possible to install some TAP between the mic and the VoiceProcessingIO unit?
AFAIK, there is no public API that allows getting both processed and unprocessed input from the microphone on an iOS device.
If you need processed input (voice processing for echo cancellation, etc.), then your best bet is to just add gain to the audio data for your other needs (via some DSP library, etc.), since it is float data.
Are there direct calls to the audio unit so that I don't have to depend on the system callbacks-input and render callbacks(I can mimic this with a timer). For example, like there is AudioUnitRender to pull data from the audio unit, is there another API to push data to the audio unit?
While I'm not aware of a specific push-like call in the CoreAudio API, you can easily accomplish this by doing your DSP processing in a separate C function which takes floating point buffers passed into it. This way, the render callback can do the hard work there, and you can also call the function manually if you need to do push-based processing.
The audio units behavior is the pull model. The output request data to play and if it has it, then it plays.
The common way to start recording is to call Render within the output callback. The Render will get you the data from the recorder which can be played or just save your recorded data to somewhere else and set the player to play silence.
I've searched around but haven't found any good examples or tutorials of saving audio out of a RemoteIO Audio Unit.
My setup: Using the MusicPlayer API, I have several AUSamplers -> MixerUnit -> RemoteIO
Audio playback works great. I would like to add functionality to save the audio output to a file. Would I do this in a render callback on the RemoteIO?
Any tips or pointers to example code much appreciated!
Due to the tight latency requirements of Audio Unit callbacks, one should not to do any synchronous file access (or any other calls that could potentially block, involve memory management or OS locking actions) inside the RemoteIO callback. Instead, just copy the audio data out to another buffer (a larger circular buffer for example), and set some state indicating how much data has been copied. Then, in another thread, when the amount of data is sufficient, write the contents of that buffer out to a file. This could be a raw PCM file, which can later be converted by AVAssetReader/Writer into another audio file type.
What I want to do is to take the output samples of an AVAsset corresponding to an audio file (no video involved) and send them to an audio effect class that takes in a block of samples, and I want to be able to this in real time.
I am currently looking at the AVfoundation class reference and programming guide, but I can't see a way of redirect the output of a player item and send it to my effect class, and from there, send the transformed samples to an Audio output (using AVAssetReaderAudioMixOutput?) and hear it from there. I see that the AVAssetReader class gives me a way to get a block of samples using
[myAVAssetReader addOutput:myAVAssetReaderTrackOutput];
[myAVAssetReaderTrackOutput copyNextSampleBuffer];
but Apple documentation specifies that the AVAssetReader class is not made and should not be used for real-time situations. Does anybody have a suggestion on where to look, or if I am having the right approach?
The MTAudioProcessingTap is perfect for this. By leveraging an AVPlayer, you can avoid having to block the samples yourself with the AVAssetReaderOutput and then render them yourself in an Audio Queue or with an Audio Unit.
Instead, attach an MTAudioProcessingTap to the inputParameters of your AVAsset's audioMix, and you'll be given samples in blocks which are easy to then throw into an effect unit.
Another benefit from this is that it will work with AVAssets derived from URLs that can't always be opened by other Apple APIs (like Audio File Services), such as the user's iPod library. Additionally, you get all of the functionality like tolerance of audio interruptions that the AVPlayer provides for free, which you would otherwise have to implement by hand if you went with an AVAssetReader solution.
To set up a tap you have to set up some callbacks that the system invokes as appropriate during playback. Full code for such processing can be found at this tutorial here.
There's a new MTAudioProcessingTap object in iOS 6 and Mac OS 10.8 . Check out the Session 517 video from WWDC 2012 - they've demonstrated exactly what you want to do.
WWDC Link
AVAssetReader is not ideal for realtime usage because it handles the decoding for you, and in various cases copyNextSampleBuffer can block for random amounts of time.
That being said, AVAssetReader can be used wonderfully well in a producer thread feeding a circular buffer. It depends on your required usage, but I've had good success using this method to feed a RemoteIO output, and doing my effects/signal processing in the RemoteIO callback.
I'm writing an iOS application that will play audio instructions as one of it's features.
Every time the application wants to play audio it reads from a non-standard file and puts the resulting PCM data for that audio in a buffer in memory.
Even though I have that buffer with the PCM data, I'm having trouble getting the application to actually play the sound. After searching the iOS documentation, I started implementing an AudioUnit. The problem with this AudioUnit is the use of a render callback (as far as I know, the only way to output sound). From Apple's developer documentation:
… render callbacks have a strict performance requirement that you must
adhere to. A render callback lives on a real-time priority thread on
which subsequent render calls arrive asynchronously. The work you do
in the body of a render callback takes place in this time-constrained
environment. If your callback is still producing sample frames in
response to the previous render call when the next render call
arrives, you get a gap in the sound. For this reason you must not take
locks, allocate memory, access the file system or a network
connection, or otherwise perform time-consuming tasks in the body of a
render callback function
If I can't use locks inside the render callback method I can't be reading the buffer while writing in it. There is no opportunity to read the file and write to the buffer because the render callback will be accessing it constantly.
The only example I found actually generated the PCM data inside the render method, which I can't do.
Is this the only way of using AudioUnits (with an asynchronous render callback)?
Is there an alternative to playback PCM data from memory?
Using the RemoteIO Audio Unit might require having a separate data queue (fifo or circular buffer), outside the audio unit callback, which can pre-buffer up enough audio data from a file read, ahead of audio unit render callback, to meet worse case latencies. Then the render callback only needs to do a quick copy of the audio data, and then the update of a write-only flag that indicates that audio data was consumed.
An alternative built into iOS is to use the Audio Queue API, which does the pre-buffering for you. It allows your app to fill a number of larger audio buffers in the main run loop ahead of time. You still have to pre-buffer enough data to allow for the max of file, network, lock or other latencies.
Another strategy is to have alternative audio data to feed the real-time render callback if the file or network read didn't keep up, such as quickly creating an audio buffer that tapers to silence (and then un-tapering when real data starts arriving again).