I'm working on an application for tvOS platform for playing back audio using WebRTC (https://webrtc.org/). WebRTC uses AudioUnit for audio playout (https://chromium.googlesource.com/external/webrtc/+/7a82467d0db0d61f466a1da54b94f6a136726a3c/sdk/objc/native/src/audio/voice_processing_audio_unit.mm). It works perfectly on iOS, but produces errors on tvOS.
First of all I've disabled audio capturing at all. The first error happens when creating a Voice Processing IO audio unit:
// Create an audio component description to identify the Voice Processing
// I/O audio unit.
AudioComponentDescription vpio_unit_description;
vpio_unit_description.componentType = kAudioUnitType_Output;
vpio_unit_description.componentSubType = kAudioUnitSubType_VoiceProcessingIO;
vpio_unit_description.componentManufacturer = kAudioUnitManufacturer_Apple;
vpio_unit_description.componentFlags = 0;
vpio_unit_description.componentFlagsMask = 0;
// Obtain an audio unit instance given the description.
AudioComponent found_vpio_unit_ref =
AudioComponentFindNext(nullptr, &vpio_unit_description);
// Create a Voice Processing IO audio unit.
OSStatus result = noErr;
result = AudioComponentInstanceNew(found_vpio_unit_ref, &vpio_unit_);
if (result != noErr) {
vpio_unit_ = nullptr;
RTCLogError(#"AudioComponentInstanceNew failed. Error=%ld.", (long)result);
return false;
}
AudioComponentInstanceNew returns -3000 OSStatus (I assume it means an invalid component ID). This issue can be fixed by replacing kAudioUnitSubType_VoiceProcessingIO → kAudioUnitSubType_GenericOutput (I'm not sure this is a correct replacement, but the error is gone).
After that WebRTC is trying to enable output
// Enable output on the output scope of the output element.
UInt32 enable_output = 1;
result = AudioUnitSetProperty(vpio_unit_, kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Output, kOutputBus,
&enable_output, sizeof(enable_output));
if (result != noErr) {
DisposeAudioUnit();
RTCLogError(#"Failed to enable output on output scope of output element. "
"Error=%ld.",
(long)result);
return false;
}
and this doesn't work as well: it returns -10879 OSStatus (I assume it means an invalid property). I think the problem is in providing kAudioOutputUnitProperty_EnableIO property, but have no idea why should be utilized instead.
Any ideas pr advices are very much appreciated. Thanks in advance.
Related
I need to generate a signal and play it with iPhone's speakers or a headset.
To do so I generate an interleaved signal. Then i need to instantiate an AudioUnit inherited class object with the next info: 2 channels, 44100 kHz sample rate, some buffer size to store a few frames.
Then I need to write a callback method which will take a chink of my signal and pit it into iPhone's output buffer.
The problem is that I have no idea how to write an AudioUnit inherited class. I can't understand Apple's documentation regarding it, and all the examples I could find either read from file and play it with huge lag or use depricated constructions.
I start to think I am stupid or something. Please, help...
To play audio to the iPhone's hardware with an AudioUnit, you don't derive from the AudioUnit as CoreAudio is a c framework - instead you give it a render callback in which you feed the unit your audio samples. The following code sample shows you how. You need to replace the asserts with real error handling and you'll probably want to change or at least inspect the audio unit's sample format using the kAudioUnitProperty_StreamFormat selector. My format happens to be 48kHz floating point interleaved stereo.
static OSStatus
renderCallback(
void* inRefCon,
AudioUnitRenderActionFlags* ioActionFlags,
const AudioTimeStamp* inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList* ioData)
{
// inRefCon contains your cookie
// write inNumberFrames to ioData->mBuffers[i].mData here
return noErr;
}
AudioUnit
createAudioUnit() {
AudioUnit au;
OSStatus err;
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
AudioComponent comp = AudioComponentFindNext(NULL, &desc);
assert(0 != comp);
err = AudioComponentInstanceNew(comp, &au);
assert(0 == err);
AURenderCallbackStruct input;
input.inputProc = renderCallback;
input.inputProcRefCon = 0; // put your cookie here
err = AudioUnitSetProperty(au, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Input, 0, &input, sizeof(input));
assert(0 == err);
err = AudioUnitInitialize(au);
assert(0 == err);
err = AudioOutputUnitStart(au);
assert(0 == err);
return au;
}
I am using RemoteIO Audio Unit for audio playback in my app with kAudioUnitProperty_ScheduledFileIDs.
Audio files are in PCM format. How can I implement a render callback function for this case, so I could manually modify buffer samples?
Here is my code:
static AudioComponentInstance audioUnit;
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
AudioComponent comp = AudioComponentFindNext(NULL, &desc);
CheckError(AudioComponentInstanceNew(comp, &audioUnit), "error AudioComponentInstanceNew");
NSURL *playerFile = [[NSBundle mainBundle] URLForResource:#"short" withExtension:#"wav"];
AudioFileID audioFileID;
CheckError(AudioFileOpenURL((__bridge CFURLRef)playerFile, kAudioFileReadPermission, 0, &audioFileID), "error AudioFileOpenURL");
// Determine file properties
UInt64 packetCount;
UInt32 size = sizeof(packetCount);
CheckError(AudioFileGetProperty(audioFileID, kAudioFilePropertyAudioDataPacketCount, &size, &packetCount),
"AudioFileGetProperty(kAudioFilePropertyAudioDataPacketCount)");
AudioStreamBasicDescription dataFormat;
size = sizeof(dataFormat);
CheckError(AudioFileGetProperty(audioFileID, kAudioFilePropertyDataFormat, &size, &dataFormat),
"AudioFileGetProperty(kAudioFilePropertyDataFormat)");
// Assign the region to play
ScheduledAudioFileRegion region;
memset (®ion.mTimeStamp, 0, sizeof(region.mTimeStamp));
region.mTimeStamp.mFlags = kAudioTimeStampSampleTimeValid;
region.mTimeStamp.mSampleTime = 0;
region.mCompletionProc = NULL;
region.mCompletionProcUserData = NULL;
region.mAudioFile = audioFileID;
region.mLoopCount = 0;
region.mStartFrame = 0;
region.mFramesToPlay = (UInt32)packetCount * dataFormat.mFramesPerPacket;
CheckError(AudioUnitSetProperty(audioUnit, kAudioUnitProperty_ScheduledFileRegion, kAudioUnitScope_Global, 0, ®ion, sizeof(region)),
"AudioUnitSetProperty(kAudioUnitProperty_ScheduledFileRegion)");
// Prime the player by reading some frames from disk
UInt32 defaultNumberOfFrames = 0;
CheckError(AudioUnitSetProperty(audioUnit, kAudioUnitProperty_ScheduledFilePrime, kAudioUnitScope_Global, 0, &defaultNumberOfFrames, sizeof(defaultNumberOfFrames)),
"AudioUnitSetProperty(kAudioUnitProperty_ScheduledFilePrime)");
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = MyCallback;
callbackStruct.inputProcRefCon = (__bridge void * _Nullable)(self);
CheckError(AudioUnitSetProperty(audioUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Input, 0, &callbackStruct, sizeof(callbackStruct)), "error AudioUnitSetProperty[kAudioUnitProperty_setRenderCallback]");
CheckError(AudioUnitInitialize(audioUnit), "error AudioUnitInitialize");
Callback function:
static OSStatus MyCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData){
printf("my callback");
return noErr;
}
Audio Unit start playback on button press:
- (IBAction)playSound:(id)sender {
CheckError(AudioOutputUnitStart(audioUnit), "error AudioOutputUnitStart");
}
This code fails during compiling with kAudioUnitErr_InvalidProperty(-10879) error. The goal is to modify buffer samples that has been read from the AudioFileID and send the result to the speakers.
Seeing as how you are just getting familiar with core audio, I suggest you first get your remoteIO callback working independently of your file player. Just remove all of your file player related code and try to get that working first.
Then, once you have that working, move on to incorporating your file player.
As far as what I can see that's wrong, I think you are confusing the Audio File Services API with an audio unit. This API is used to read a file into a buffer which you would manually feed to to remoteIO, if you do want to go this route, use the Extended Audio File Services API, it's a LOT easier. The kAudioUnitProperty_ScheduledFileRegion property is supposed to be called on a file player audio unit. To get one of those, you would need to create it the same way as your remmoteIO with the exception that AudioComponentDescription's componentSubType and componentType are kAudioUnitSubType_AudioFilePlayer and kAudioUnitType_Generator respectively. Then, once you have that unit you would need to connect it to the remoteIO using the kAudioUnitProperty_MakeConnection property.
But seriously, start with just getting your remoteIO callback working, then try making a file player audio unit and connecting it (without the callback), then go from there.
Ask very specific questions about each of these steps independently, posting code you have tried that's not working, and you'll get a ton of help.
I'm working on an iOS app that does two things at the same time:
It captures audio and video and relays them to a server to provide video chat functionality.
It captures local audio and video and encodes them into an mp4 file to be saved for posterity.
Unfortunately, when we configure the app with an audio unit to enable echo cancellation, the recording functionality breaks: the AVAssetWriterInput instance we're using to encode audio rejects incoming samples. When we don't set up the audio unit, recording works, but we have terrible echo.
To enable echo cancellation, we configure an audio unit like this (paraphrasing for the sake of brevity):
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_VoiceProcessingIO;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
AudioComponent comp = AudioComponentFindNext(NULL, &desc);
OSStatus status = AudioComponentInstanceNew(comp, &_audioUnit);
status = AudioUnitInitialize(_audioUnit);
This works fine for video chat, but it breaks the recording functionality, which is set up like this (again, paraphrasing—the actual implementation is spread out over several methods).
_captureSession = [[AVCaptureSession alloc] init];
// Need to use the existing audio session & configuration to ensure we get echo cancellation
_captureSession.usesApplicationAudioSession = YES;
_captureSession.automaticallyConfiguresApplicationAudioSession = NO;
[_captureSession beginConfiguration];
AVCaptureDeviceInput *audioInput = [[AVCaptureDeviceInput alloc] initWithDevice:[self audioCaptureDevice] error:NULL];
[_captureSession addInput:audioInput];
_audioDataOutput = [[AVCaptureAudioDataOutput alloc] init];
[_audioDataOutput setSampleBufferDelegate:self queue:_cameraProcessingQueue];
[_captureSession addOutput:_audioDataOutput];
[_captureSession commitConfiguration];
And the relevant portion of captureOutput looks something like this:
NSLog(#"Audio format, channels: %d, sample rate: %f, format id: %d, bits per channel: %d", basicFormat->mChannelsPerFrame, basicFormat->mSampleRate, basicFormat->mFormatID, basicFormat->mBitsPerChannel);
if (_assetWriter.status == AVAssetWriterStatusWriting) {
if (_audioEncoder.readyForMoreMediaData) {
if (![_audioEncoder appendSampleBuffer:sampleBuffer]) {
NSLog(#"Audio encoder couldn't append sample buffer");
}
}
}
What happens is the call to appendSampleBuffer fails, but—and this is the strange part—only if I don't have earphones plugged into my phone. Examining the logs produced when this happens, I found that without earphones connected, the number of channels reported in the log message was 3, whereas with earphones connected, the number of channels was 1. This explains why the encode operation was failing, since the encoder was configured to expect just a single channel.
What I don't understand is why I'm getting three channels here. If I comment out the code that initializes the audio unit, I only get a single channel and recording works fine, but echo cancellation doesn't work. Furthermore, if I remove these lines
// Need to use the existing audio session & configuration to ensure we get echo cancellation
_captureSession.usesApplicationAudioSession = YES;
_captureSession.automaticallyConfiguresApplicationAudioSession = NO;
recording works (I only get a single channel with or without headphones), but again, we lose echo cancellation.
So, the crux of my question is: why am I getting three channels of audio when I configure an audio unit to provide echo cancellation? Furthermore, is there any way to prevent this from happening or to work around this behavior using AVCaptureSession?
I've considered piping the microphone audio directly from the low-level audio unit callback into the encoder, as well as to the chat pipeline, but it seems like conjuring up the necessary Core Media buffers to do so would be a bit of work that I'd like to avoid if possible.
Note that the chat and recording functions were written by different people—neither of them me—which is the reason this code isn't more integrated. If possible, I'd like to avoid having to refactor the whole mess.
Ultimately, I was able to work around this issue by gathering audio samples from the microphone via the I/O audio unit, repackaging these samples into a CMSampleBuffer, and passing the newly constructed CMSampleBuffer into the encoder.
The code to do the conversion looks like this (abbreviated for the sake of brevity).
// Create a CMSampleBufferRef from the list of samples, which we'll own
AudioStreamBasicDescription monoStreamFormat;
memset(&monoStreamFormat, 0, sizeof(monoStreamFormat));
monoStreamFormat.mSampleRate = 48000;
monoStreamFormat.mFormatID = kAudioFormatLinearPCM;
monoStreamFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagsNativeEndian | kAudioFormatFlagIsPacked | kAudioFormatFlagIsNonInterleaved;
monoStreamFormat.mBytesPerPacket = 2;
monoStreamFormat.mFramesPerPacket = 1;
monoStreamFormat.mBytesPerFrame = 2;
monoStreamFormat.mChannelsPerFrame = 1;
monoStreamFormat.mBitsPerChannel = 16;
CMFormatDescriptionRef format = NULL;
OSStatus status = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &monoStreamFormat, 0, NULL, 0, NULL, NULL, &format);
// Convert the AudioTimestamp to a CMTime and create a CMTimingInfo for this set of samples
uint64_t timeNS = (uint64_t)(hostTime * _hostTimeToNSFactor);
CMTime presentationTime = CMTimeMake(timeNS, 1000000000);
CMSampleTimingInfo timing = { CMTimeMake(1, 48000), presentationTime, kCMTimeInvalid };
CMSampleBufferRef sampleBuffer = NULL;
status = CMSampleBufferCreate(kCFAllocatorDefault, NULL, false, NULL, NULL, format, numSamples, 1, &timing, 0, NULL, &sampleBuffer);
// add the samples to the buffer
status = CMSampleBufferSetDataBufferFromAudioBufferList(sampleBuffer,
kCFAllocatorDefault,
kCFAllocatorDefault,
0,
samples);
// Pass the buffer into the encoder...
Please note that I've removed error handling and cleanup of the allocated objects.
I am writing an HOST app that uses Core Audio's new iOS 7 Inter App Audio technology to pull audio from a single NODE "generator" app and route it into my host app. I am using the Audio Components Services and Audio Unit Component Services C frameworks to achieve this.
What I want to achieve is to establish a connection to an external node app who can generate sound. I want that sound to be routed into my host app and for my host app to be able to directly access the audio packet data as a stream of raw audio data.
I have written the code inside my HOST app that does the following sequentially:
Sets up and activates an audio session with the correct session category.
Refreshes a list of interapp audio compatible apps who are of typekAudioUnitType_RemoteGenerator or kAudioUnitType_RemoteInstrument (I'm not interested in effects apps).
Pulls out the last object from that list and attempts to establish a connection using AudioComponentInstanceNew()
Sets the Audio Stream Basic Description that my host app needs the audio format in.
Sets up audio unit properties and callbacks as well as an audio unit render callback on the output scope (bus).
Initializes the audio unit.
So far so good, I have been able to successfully establish a connection, but my problem is that my render callback is not being called at all. What I am having trouble understanding is how exactly to pull the audio from the node application? I have read that I need to call AudioUnitRender() in order to initiate a rendering cycle on the node app, but how exactly does this need to be setup in my situation? I have seen other examples where AudioUnitRender() is called from inside the rendering callback, but this isnt going to work for me because my render callback isn't being called currently. Do I need to setup up my own audio processing thread and periodically call AudioUnitRender() on my 'node'?
The following is the code described above inside my HOST app.
static OSStatus MyAURenderCallback (void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
//Do something here with the audio data?
//This method is never being called?
//Do I need to puts AudioUnitRender() in here?
}
- (void)start
{
[self configureAudioSession];
[self refreshAUList];
}
- (void)configureAudioSession
{
NSError *audioSessionError = nil;
AVAudioSession *mySession = [AVAudioSession sharedInstance];
[mySession setPreferredSampleRate: _graphSampleRate error: &audioSessionError];
[mySession setCategory: AVAudioSessionCategoryPlayAndRecord error: &audioSessionError];
[mySession setActive: YES error: &audioSessionError];
self.graphSampleRate = [mySession sampleRate];
}
- (void)refreshAUList
{
_audioUnits = #[].mutableCopy;
AudioComponentDescription searchDesc = { 0, 0, 0, 0, 0 }, foundDesc;
AudioComponent comp = NULL;
while (true) {
comp = AudioComponentFindNext(comp, &searchDesc);
if (comp == NULL) break;
if (AudioComponentGetDescription(comp, &foundDesc) != noErr) continue;
if (foundDesc.componentType == kAudioUnitType_RemoteGenerator || foundDesc.componentType == kAudioUnitType_RemoteInstrument) {
RemoteAU *rau = [[RemoteAU alloc] init];
rau->_desc = foundDesc;
rau->_comp = comp;
AudioComponentCopyName(comp, &rau->_name);
rau->_image = AudioComponentGetIcon(comp, 48);
rau->_lastActiveTime = AudioComponentGetLastActiveTime(comp);
[_audioUnits addObject:rau];
}
}
[self connect];
}
- (void)connect {
if ([_audioUnits count] <= 0) {
return;
}
RemoteAU *rau = [_audioUnits lastObject];
AudioUnit myAudioUnit;
//Node application will get launched in background
Check(AudioComponentInstanceNew(rau->_comp, &myAudioUnit));
AudioStreamBasicDescription format = {0};
format.mChannelsPerFrame = 2;
format.mSampleRate = [[AVAudioSession sharedInstance] sampleRate];
format.mFormatID = kAudioFormatMPEG4AAC;
UInt32 propSize = sizeof(format);
Check(AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &propSize, &format));
//Output format from node to host
Check(AudioUnitSetProperty(myAudioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 0, &format, sizeof(format)));
//Setup a render callback to the output scope of the audio unit representing the node app
AURenderCallbackStruct callbackStruct = {0};
callbackStruct.inputProc = MyAURenderCallback;
callbackStruct.inputProcRefCon = (__bridge void *)(self);
Check(AudioUnitSetProperty(myAudioUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Output, 0, &callbackStruct, sizeof(callbackStruct)));
//setup call backs
Check(AudioUnitAddPropertyListener(myAudioUnit, kAudioUnitProperty_IsInterAppConnected, IsInterappConnected, NULL));
Check(AudioUnitAddPropertyListener(myAudioUnit, kAudioOutputUnitProperty_HostTransportState, AudioUnitPropertyChangeDispatcher, NULL));
//intialize the audio unit representing the node application
Check(AudioUnitInitialize(myAudioUnit));
}
I'm getting a -10879 (kAudioUnitErr_InvalidProperty) when I try to set the ReverbRoomType property on a Reverb2 unit in iOS.Here's how I create it:
AudioComponentDescription outputcd = {0};
outputcd.componentType = kAudioUnitType_Effect;
outputcd.componentSubType = kAudioUnitSubType_Reverb2;
outputcd.componentManufacturer = kAudioUnitManufacturer_Apple;
AUNode reverbNode;
CheckError(AUGraphAddNode(self.auGraph, &outputcd, &reverbNode),
"AUGraphNode[kAudioUnitSubType_Reverb2] failed");
CheckError(AUGraphNodeInfo(_auGraph, reverbNode, NULL, &_auReverb),
"AUGraphNodeInfo failed [reverbNode]");
And here's setting the room type (which fails the "CheckError()" call, which tests the return against noErr and logs the error before exiting). This is the call that fails with kAudioUnitErr_InvalidProperty:
UInt32 roomType = kReverbRoomType_LargeHall;
CheckError(AudioUnitSetProperty(_auReverb, kAudioUnitProperty_ReverbRoomType,
kAudioUnitScope_Global, 0, &roomType, sizeof(UInt32)),
"AudioUnitSetProperty[kAudioUnitProperty_ReverbRoomType] failed");
I've only tried 2 different values for the property, but the error makes me think that the property constant itself is wrong. Is this not settable on iOS 5? Changing from kAudioUnitProperty_ReverbRoomType to kAudioUnitProperty_ReverbPreset doesn't help either.
Thanks in advance for any suggestions.
The kAudioUnitProperty_ReverbRoomType and kAudioUnitProperty_ReverbPreset properties apply to the 3D Mixer AU, not the Reverb2 AU.