I'd like to "intercept" audio data on its way to the iOS device's speaker. I believe this can be done using remoteIO audio units and callbacks. In the playbackCallback below does ioData actually contain any audio data?
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) { ... }
I'm confused because logging info about ioData seems to imply it contains audio data...
// if (ioData->mNumberBuffers > 0)
AudioBuffer buffer = ioData->mBuffers[0];
NSLog(#"buffer.mNumberChannels: %ld", buffer.mNumberChannels); // prints 2
NSLog(#"buffer.mDataByteSize : %ld", buffer.mDataByteSize); // prints 4096
However creating a CMSampleBufferRef from the contents of ioData and writing it to a CoreAudioFile using an AVAssetWriter yields a silent file. The length of the output file seems fine (a few seconds) but opening the file in Audacity for example shows a flatline.
After reading numerous SO posts and experimenting with lots of remoteIO audio unit sample code I'm starting to wonder if ioData above contains pre-sized but empty buffers that should be populated in the playbackCallback.
The ioData buffers in a playCallback are where the callback should put the audio samples you want to play. The buffers do not contain other audio intercepted on its way to the speaker.
Related
aurioTouch sample codes runs OK on iOS15 devices, but on iOS16, inNumberFrames of performRender always becomes 1.
On iOS15, inNumberFrames is usually 512 or 1024. Sometimes smaller or bigger, but not 1.
// Render callback function
static OSStatus performRender (void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
I know aurioTouch is old sample code, but it runs OK on iOS15, so I suppose there should be difference between iOS15 and iOS16.
Any help on this issue would be appreciated.
This is what I would like to do:
Get audio from the microphone
Encode it in AAC, G.711 or G.726
Write the encoded frames to a socket.
And this is how I'm trying to get there:
I'm getting audio (PCM) from the microphone using TheAmazingAudioEngine and putting it in a buffer;
Using TPAACAudioConverter I'm reading audio from my buffer and writing to a temp file (AAC);
In the processing thread of TPAACAudioConverter I replaced this:
OSStatus status = ExtAudioFileWrite(destinationFile, numFrames, &fillBufList);
with this:
OSStatus status = ExtAudioFileWrite(destinationFile, numFrames, &fillBufList);
UInt32 framesWritten = numFrames;
totalFramesWritten += framesWritten;
AudioBufferList readData;
readData.mNumberBuffers = 1;
ExtAudioFileSeek(destinationFile, totalFramesWritten - framesWritten);
OSStatus readStatus = ExtAudioFileRead(destinationFile, &numFrames, &readData);
ExtAudioFileSeek(destinationFile, totalFramesWritten);
NSLog(#"Bytes read=%d", numFrames);
but what I get is 0 numFrames read from file.
Any idea on what I may be doing wrong or any suggestion on alternative paths to achieve what I need?
The issue is that whatever ExtAudioFile does under the hood doesn't allow for seeking on a file that is open for writing. If you look at the documentation for ExtAudioFileSeek it says "This function's behavior with files open for writing is currently undefined".
You can solve this by using the more extensible (and difficult) Audio File Services and the Audio Converter Services directly instead of the convenient Extended audio file services.
I abandoned this approach and reused the AQRecorder class from the SpeakHere example by Apple.
The project is available here https://github.com/robovm/apple-ios-samples/tree/master/SpeakHere.
I am working on a project that requires almost live (we can live with some core audio latency) audio monitoring from a bluetooth microphone to the iPhone's default
I've tried both:
UInt32 allowBluetoothInput = 1;
OSStatus stat = AudioSessionSetProperty (kAudioSessionProperty_OverrideCategoryEnableBluetoothInput, sizeof (allowBluetoothInput), &allowBluetoothInput);
UInt32 audioRouteOverride = kAudioSessionOverrideAudioRoute_Speaker;
stat= AudioSessionSetProperty (kAudioSessionProperty_OverrideAudioRoute, sizeof (audioRouteOverride), &audioRouteOverride);
and
UInt32 overrideAudioRoute = kAudioSessionOverrideAudioRoute_None;
AudioSessionSetProperty (kAudioSessionProperty_OverrideAudioRoute, sizeof (overrideAudioRoute), &overrideAudioRoute);
UInt32 doChangeDefaultRoute = 1;
AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryDefaultToSpeaker, sizeof(doChangeDefaultRoute), &doChangeDefaultRoute);
but neither yields the correct behavior
Is there any property I can set to get this to work?
You are talking about a speak through application? I did some R&D on this awhile back and ran into the issues you are talking about. At the present it seems that this is not possible on iOS. You cannot configure your iPhone to record from the microphone and route that audio recording out to the speaker for instant playback. The best you can do is record it, stop it and send it. Recording and playing through the speaker cannot be configured in an audio session. Spending time researching this (like I did) will be upsetting. It has nothing to do with core audio hacking it has to do with what is allowed by the system
I have been trying to play audio which is received as raw data in didOutputSampleBuffer delegate. What should be the proper way to process the raw data?
Look at the following sample code from Apple: AVCaptureTtoAudioUnitOSX
There you can see how to properly process the raw audio data and pass it to the AudioUnit.
The basic principle is as follows:
Get the SampleBuffer's AudioStreamBasicDescription for info on format
First get the CMFormatDescriptionRef with CMSampleBufferGetFormatDescription
Then get the AudioStreamBasicDescription with CMAudioFormatDescriptionGetStreamBasicDescription
Now you can get info on sample rate, bits per channel, channels per frame and frames per packet
Get the AudioBufferList with the actual audio data
Either use CoreAudio's Public Utility or check this mailing list entry for a correct way of doing so
The function is called CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer. Third parameter is the bufferListOut which is the AudioBufferList you want and will pass on to work with e.g. the AudioUnit or whatever your need is.
Getting the actual raw data
The AudioBufferList contains AudioBuffers each of which contain the data
struct AudioBuffer {
UInt32 mNumberChannels;
UInt32 mDataByteSize;
void *mData;
};
This should get you going. Look at the sample code from Apple for more info.
just started developing a small test iOS 5.0 app to see what is possible on the platform.
Some ressources were unvaluable, for example Chris Adamson's blog, and David Zicarelli's audioGraph example (based upon Apple's MixerHost with a bunch of new features).
What I'm trying to do is to setup something like this, using the new FilePlayer AudioUnit from the iOS 5.x SDK.
(AUFilePlayer bus0) -> (some custom process) -> (bus0 MultiChannelMixer bus0) -> (bus0 Remote I/O)
I started with audioGraph, removed what I didn't want, and ended up with the above chain.
I could see that AUFilePlayer preferred output is a 8.24 streams, so the mixer is also setup that way (8.24 on input scope). My process will handle any conversion it will need.
The "custom process" callback is registered for the mixer on bus 0. Once the app is launched, it gets called regularly, which I could verify by logging.
static OSStatus simpleRenderCallback (
void *inRefCon,
AudioUnitRenderActionFlags
*ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData
) {
MixerHostAudio* THIS = (MixerHostAudio*) inRefCon; //the MixerHostAudio instance
AudioUnit fpUnit= THIS.auFilePlayerUnit_1;
OSStatus renderErr = noErr;
renderErr= AudioUnitRender(AUFilePlayerUnit, ioActionFlags, inTimeStamp, 0, inNumberFrames, ioData);
if (renderErr < 0) {
NSLog(#"error:%ld",renderErr);
return renderErr;
}
return noErr;
}
The issue is that I'm always getting a renderErr = -50 each time the AudioUnitRender gets called in my callback.
I'm running with the simulator for now, the Mac soundcard is set to 44,100Hz, and I could see that the inNumberFrames is always equal to 512.
Where does the problem come from? -50 means "bad param" in CoreAudio, but that's not enough to know what's wrong.
Thanks!