As a learning exercise, I'm using an AudioQueue to generate and play a 300 Hz sine wave. (I understand there are a variety of tools to generate and play audio, but yes, this is just to build up my Core Audio chops and this task is all about the AudioQueue.)
The wave plays, but with distortion. Recording and plotting the sound shows that there is some distortion at the boundary between buffers (every half second), in addition to other short bursts of distortion here and there. I've included my code below. If anyone could shine some light on the problem, that would be amazing—thanks for reading!
EDIT: Found the problem. It should read bufferByteSize=numPacketsForTime*asbd.mBytesPerPacket;
static void MyAQOutputCallback(void *inUserData,
AudioQueueRef inAQ,
AudioQueueBufferRef inCompleteAQBuffer){
int i;
MyWave *inData=(MyWave*)inUserData;
// synth params
int phaseL =inData->sampleCount;
float FL = (2.0 * 3.14159265 * 300.0) / 44100.0;
float amp = 0.5;
int frameCount=22050;
// Get the info struct and a pointer to our output data
short *coreAudioBuffer = (short*) inCompleteAQBuffer->mAudioData;
// Need to set this
inCompleteAQBuffer->mAudioDataByteSize = 2*frameCount; // two shorts per frame, one frame per packet
// For each frame/packet (the same in our example)
for(i=0;i<frameCount;i++) {
// Render the sine waves - signed interleaved shorts (-32767 -> 32767), 16 bit stereo
float sampleL = (amp * sin(FL * (float)phaseL));
short sampleIL = (int)(sampleL * 32767.0);
coreAudioBuffer[i ] = sampleIL;
phaseL++;
}
// "Enqueue" the buffer
AudioQueueEnqueueBuffer(inAQ, inCompleteAQBuffer, 0, NULL);
inData->sampleCount=phaseL;
}
int main(int argc, const char * argv[])
{
// Open an audio file
MyWave thisWave={0};
// Set up format
AudioStreamBasicDescription asbd;
memset(&asbd,0,sizeof(asbd));
asbd.mSampleRate=SAMPLE_RATE;
asbd.mFormatID=kAudioFormatLinearPCM;
asbd.mFormatFlags=kLinearPCMFormatFlagIsSignedInteger|kAudioFormatFlagIsPacked;
asbd.mBitsPerChannel=16;
asbd.mChannelsPerFrame=1;
asbd.mFramesPerPacket=1;
asbd.mBytesPerFrame=2;
asbd.mBytesPerPacket=2;
// Set up queue
AudioQueueRef queue;
CheckError(AudioQueueNewOutput(&asbd,
MyAQOutputCallback,
&thisWave,
NULL,
NULL,
0,
&queue),
"AudioQueueNewOutput failed");
UInt32 bufferByteSize;
Float64 numPacketsForTime=asbd.mSampleRate/asbd.mFramesPerPacket*0.5;
bufferByteSize=numPacketsForTime;
AudioQueueBufferRef buffers[kNumberPlaybackBuffers];
int i;
for (i=0;i<kNumberPlaybackBuffers;++i){
CheckError(AudioQueueAllocateBuffer(queue,
bufferByteSize,
&buffers[i]),
"AudioQueueAllocateBuffer failed");
MyAQOutputCallback(&thisWave, queue, buffers[i]);
}
// Start queue
CheckError(AudioQueueStart(queue,
NULL),
"AudioQueueStart failed");
printf("Playing...\n");
do
{
CFRunLoopRunInMode(kCFRunLoopDefaultMode,
0.25,
false);
}while (1==1);
CFRunLoopRunInMode(kCFRunLoopDefaultMode, 2, false);
// Clean up queue when finished
CheckError(AudioQueueStop(queue,
TRUE),
"AudioQueueStop failed");
AudioQueueDispose(queue, TRUE);
return 0;
}
Problem found, it should read:
bufferByteSize = numPacketsForTime*asbd.mBytesPerPacket;
I'll leave this up here, as somebody may find the code useful!
Related
I'm developing an app for transfer audio data between 2 iOS devices.
At the moment I'm using NSInputStream for receive the incoming audio and the data into a queue (AudioQueueNewInput).
The callback of the AudioQueue is
void TDAudioQueueInputCallback(
void* inUserData,
AudioQueueRef inAudioQueue,
AudioQueueBufferRef inBuffer,
const AudioTimeStamp* inStartTime,
UInt32 inNumPackets,
const AudioStreamPacketDescription* inPacketDesc)
{
NSLog(#"recordCallback %u", (unsigned int)inBuffer->mAudioDataByteSize);
TDAudioQueue *audioQueue = (__bridge TDAudioQueue *)inUserData;
AudioBufferList *audioBufferList = [audioQueue getBufferListFromQueueBuffer:inBuffer];
[audioQueue sendAudioBufferToRecord:audioBufferList];
[audioQueue didFreeAudioQueueBuffer:inBuffer];
}
getBufferListFromQueueBuffer method
-(AudioBufferList *) getBufferListFromQueueBuffer: (AudioQueueBufferRef ) data
{
if (data->mAudioDataByteSize > 0)
{
NSUInteger len = data->mAudioDataByteSize;
//I guess you can use Byte*, void* or Float32*. I am not sure if that makes any difference.
Byte * byteData = (Byte*) malloc (len);
memcpy (byteData, data->mAudioData, len);
if (byteData)
{
AudioBufferList * theDataBuffer =(AudioBufferList*)malloc(sizeof(AudioBufferList) * 1);
theDataBuffer->mNumberBuffers = 1;
theDataBuffer->mBuffers[0].mDataByteSize = len;
theDataBuffer->mBuffers[0].mNumberChannels = 1;
theDataBuffer->mBuffers[0].mData = byteData;
// Read the data into an AudioBufferList
return theDataBuffer;
}
}
return nil;
}
After this I'm saving the audio into a file.
The problem is I can hear the first 2 or 3 seconds but then I start to get this warning and my audio file gets silence.
The warning is:
[AQConverterThread] >aq> 995: Input ring buffer returning 0x400 frames of silence
I have a project where I need to decode h264 video from a live network stream and eventually end up with a texture I can display in another framework (Unity3D) on iOS devices. I can successfully decode the video using VTDecompressionSession and then grab the texture with CVMetalTextureCacheCreateTextureFromImage (or the OpenGL variant). It works great when I use a low-latency encoder and the image buffers come out in display order, however, when I use the regular encoder the image buffers do not come out in display order and reordering the image buffers is apparently far more difficult that I expected.
The first attempt was to set the VTDecodeFrameFlags with kVTDecodeFrame_EnableAsynchronousDecompression and kVTDecodeFrame_EnableTemporalProcessing... However, it turns out that VTDecompressionSession can choose to ignore the flag and do whatever it wants... and in my case, it chooses to ignore the flag and still outputs the buffer in encoder order (not display order). Essentially useless.
The next attempt was to associate the image buffers with the presentation time stamp and then throw them into a vector which would allow me to grab the image buffer I needed when I create the texture. The problem seems to be that the image buffer that goes into the VTDecompressionSession, which is associated with a time stamp, is no longer the same buffer that comes out, essentially making the time stamp useless.
For example, going into the decoder...
VTDecodeFrameFlags flags = kVTDecodeFrame_EnableAsynchronousDecompression;
VTDecodeInfoFlags flagOut;
// Presentation time stamp to be passed with the buffer
NSNumber *nsPts = [NSNumber numberWithDouble:pts];
VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flags,
(void*)CFBridgingRetain(nsPts), &flagOut);
On the callback side...
void decompressionSessionDecodeFrameCallback(void *decompressionOutputRefCon, void *sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CVImageBufferRef imageBuffer, CMTime presentationTimeStamp, CMTime presentationDuration)
{
// The presentation time stamp...
// No longer seems to be associated with the buffer that it went in with!
NSNumber* pts = CFBridgingRelease(sourceFrameRefCon);
}
When ordered, the time stamps on the callback side increase monotonically at the expected rate, but the buffers are not in the right order. Does anyone see where I am making an error here? Or know how to determine the order of the buffers on the callback side? At this point I have tried just about everything I can think of.
In my case, the problem wasn't with VTDecompressionSession, it was a problem with the demuxer getting the wrong PTS. While I couldn't get VTDecompressionSession to put out the frames in temporal (display) order with the kVTDecodeFrame_EnableAsynchronousDecompression and kVTDecodeFrame_EnableTemporalProcessing flags, I could sort the frames myself based on PTS with a small vector.
First, make sure you associate all of your timing information with your CMSampleBuffer along with the block buffer so you receive it in the VTDecompressionSession callback.
// Wrap our CMBlockBuffer in a CMSampleBuffer...
CMSampleBufferRef sampleBuffer;
CMTime duration = ...;
CMTime presentationTimeStamp = ...;
CMTime decompressTimeStamp = ...;
CMSampleTimingInfo timingInfo{duration, presentationTimeStamp, decompressTimeStamp};
_sampleTimingArray[0] = timingInfo;
_sampleSizeArray[0] = nalLength;
// Wrap the CMBlockBuffer...
status = CMSampleBufferCreate(kCFAllocatorDefault, blockBuffer, true, NULL, NULL, _formatDescription, 1, 1, _sampleTimingArray, 1, _sampleSizeArray, &sampleBuffer);
Then, decode the frame... It is worth trying to get the frames out in display order with the flags.
VTDecodeFrameFlags flags = kVTDecodeFrame_EnableAsynchronousDecompression | kVTDecodeFrame_EnableTemporalProcessing;
VTDecodeInfoFlags flagOut;
VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flags,
(void*)CFBridgingRetain(NULL), &flagOut);
On the callback side of things, we need a way of sorting the CVImageBufferRefs we receive. I use a struct that contains the CVImageBufferRef and the PTS. Then a vector with a size of two that will do the actual sorting.
struct Buffer
{
CVImageBufferRef imageBuffer = NULL;
double pts = 0;
};
std::vector <Buffer> _buffer;
We also need a way to sort the Buffers. Always writing to and reading from the index with the lowest PTS works well.
-(int) getMinIndex
{
if(_buffer[0].pts > _buffer[1].pts)
{
return 1;
}
return 0;
}
In the callback, we need to fill the vector with Buffers...
void decompressionSessionDecodeFrameCallback(void *decompressionOutputRefCon, void *sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CVImageBufferRef imageBuffer, CMTime presentationTimeStamp, CMTime presentationDuration)
{
StreamManager *streamManager = (__bridge StreamManager *)decompressionOutputRefCon;
#synchronized(streamManager)
{
if (status != noErr)
{
NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:status userInfo:nil];
NSLog(#"Decompressed error: %#", error);
}
else
{
// Get the PTS
double pts = CMTimeGetSeconds(presentationTimeStamp);
// Fill our buffer initially
if(!streamManager->_bufferReady)
{
Buffer buffer;
buffer.pts = pts;
buffer.imageBuffer = imageBuffer;
CVBufferRetain(buffer.imageBuffer);
streamManager->_buffer[streamManager->_bufferIndex++] = buffer;
}
else
{
// Push new buffers to the index with the lowest PTS
int index = [streamManager getMinIndex];
// Release the old CVImageBufferRef
CVBufferRelease(streamManager->_buffer[index].imageBuffer);
Buffer buffer;
buffer.pts = pts;
buffer.imageBuffer = imageBuffer;
// Retain the new CVImageBufferRef
CVBufferRetain(buffer.imageBuffer);
streamManager->_buffer[index] = buffer;
}
// Wrap around the buffer when initialized
// _bufferWindow = 2
if(streamManager->_bufferIndex == streamManager->_bufferWindow)
{
streamManager->_bufferReady = YES;
streamManager->_bufferIndex = 0;
}
}
}
}
Finally we need to drain the Buffers in temporal (display) order...
- (void)drainBuffer
{
#synchronized(self)
{
if(_bufferReady)
{
// Drain buffers from the index with the lowest PTS
int index = [self getMinIndex];
Buffer buffer = _buffer[index];
// Do something useful with the buffer now in display order
}
}
}
I would like to improve upon that answer a bit. While the outlined solution works, it requires knowledge of the number of frames needed to produce an output frame. The example uses a buffer size of 2, but in my case I needed a buffer size of 3.
To avoid having to specify this in advance one can make use of the fact, that frames (in display order) align exactly in terms of pts/duration. I.e. the end of one frame is exactly the beginning of the next. Thus one can simply accumulate frames until there is no "gap" at the beginning, then pop the first frame, and so on. Also one can take the pts of the first frame (which is always an I-frame) as the initial "head" (as it does not have to be zero...).
Here is some code that does this:
#include <CoreVideo/CVImageBuffer.h>
#include <boost/container/flat_set.hpp>
inline bool operator<(const CMTime& left, const CMTime& right)
{
return CMTimeCompare(left, right) < 0;
}
inline bool operator==(const CMTime& left, const CMTime& right)
{
return CMTimeCompare(left, right) == 0;
}
inline CMTime operator+(const CMTime& left, const CMTime& right)
{
return CMTimeAdd(left, right);
}
class reorder_buffer_t
{
public:
struct entry_t
{
CFGuard<CVImageBufferRef> image;
CMTime pts;
CMTime duration;
bool operator<(const entry_t& other) const
{
return pts < other.pts;
}
};
private:
typedef boost::container::flat_set<entry_t> buffer_t;
public:
reorder_buffer_t()
{
}
void push(entry_t entry)
{
if (!_head)
_head = entry.pts;
_buffer.insert(std::move(entry));
}
bool empty() const
{
return _buffer.empty();
}
bool ready() const
{
return !empty() && _buffer.begin()->pts == _head;
}
entry_t pop()
{
assert(ready());
auto entry = *_buffer.begin();
_buffer.erase(_buffer.begin());
_head = entry.pts + entry.duration;
return entry;
}
void clear()
{
_buffer.clear();
_head = boost::none;
}
private:
boost::optional<CMTime> _head;
buffer_t _buffer;
};
Here's a solution that works with any required buffer size, and also does not need any 3rd party libraries. My C++ code might not be the best, but it works.
We create a Buffer struct to identify the buffers by pts:
struct Buffer
{
CVImageBufferRef imageBuffer = NULL;
uint64_t pts = 0;
};
In our decoder, we need to keep track of the buffers, and what pts we want to release next:
#property (nonatomic) std::vector <Buffer> buffers;
#property (nonatomic, assign) uint64_t nextExpectedPts;
Now we are ready to handle the buffers coming in. In my case the buffers were provided asynchronously. Make sure you provide the correct duration and presentation timestamp values to the decompressionsession to be able to sort them properly:
-(void)handleImageBuffer:(CVImageBufferRef)imageBuffer pts:(CMTime)presentationTimeStamp duration:(uint64_t)duration {
//Situation 1, we can directly pass over this buffer
if (self.nextExpectedPts == presentationTimeStamp.value || duration == 0) {
[self sendImageBuffer:imageBuffer duration:duration];
return;
}
//Situation 2, we got this buffer too fast. We will store it, but first we check if we have already stored the expected buffer
Buffer futureBuffer = [self bufferWithImageBuffer:imageBuffer pts:presentationTimeStamp.value];
int smallestPtsInBufferIndex = [self getSmallestPtsBufferIndex];
if (smallestPtsInBufferIndex >= 0 && self.nextExpectedPts == self.buffers[smallestPtsInBufferIndex].pts) {
//We found the next buffer, lets store the current buffer and return this one
Buffer bufferWithSmallestPts = self.buffers[smallestPtsInBufferIndex];
[self sendImageBuffer:bufferWithSmallestPts.imageBuffer duration:duration];
CVBufferRelease(bufferWithSmallestPts.imageBuffer);
[self setBuffer:futureBuffer atIndex:smallestPtsInBufferIndex];
} else {
//We dont have the next buffer yet, lets store this one to a new slot
[self setBuffer:futureBuffer atIndex:self.buffers.size()];
}
}
-(Buffer)bufferWithImageBuffer:(CVImageBufferRef)imageBuffer pts:(uint64_t)pts {
Buffer futureBuffer = Buffer();
futureBuffer.pts = pts;
futureBuffer.imageBuffer = imageBuffer;
CVBufferRetain(futureBuffer.imageBuffer);
return futureBuffer;
}
- (void)sendImageBuffer:(CVImageBufferRef)imageBuffer duration:(uint64_t)duration {
//Send your buffer to wherever you need it here
self.nextExpectedPts += duration;
}
-(int) getSmallestPtsBufferIndex
{
int minIndex = -1;
uint64_t minPts = 0;
for(int i=0;i<_buffers.size();i++) {
if (_buffers[i].pts < minPts || minPts == 0) {
minPts = _buffers[i].pts;
minIndex = i;
}
}
return minIndex;
}
- (void)setBuffer:(Buffer)buffer atIndex:(int)index {
if (_buffers.size() <= index) {
_buffers.push_back(buffer);
} else {
_buffers[index] = buffer;
}
}
Do not forget to release all the buffers in the vector when deallocating your decoder, and if you're working with a looping file for example, keep track of when the file has fully looped to reset the nextExpectedPts and such.
I am trying to take a video file read it in using AVAssetReader and pass the audio off to CoreAudio for processing (adding effects and stuff) before saving it back out to disk using AVAssetWriter. I would like to point out that if i set the componentSubType on AudioComponentDescription of my output node as RemoteIO, things play correctly though the speakers. This makes me confident that my AUGraph is properly setup as I can hear things working. I am setting the subType to GenericOutput though so I can do the rendering myself and get back the adjusted audio.
I am reading in the audio and i pass the CMSampleBufferRef off to copyBuffer. This puts the audio into a circular buffer that will be read in later.
- (void)copyBuffer:(CMSampleBufferRef)buf {
if (_readyForMoreBytes == NO)
{
return;
}
AudioBufferList abl;
CMBlockBufferRef blockBuffer;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(buf, NULL, &abl, sizeof(abl), NULL, NULL, kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &blockBuffer);
UInt32 size = (unsigned int)CMSampleBufferGetTotalSampleSize(buf);
BOOL bytesCopied = TPCircularBufferProduceBytes(&circularBuffer, abl.mBuffers[0].mData, size);
if (!bytesCopied){
/
_readyForMoreBytes = NO;
if (size > kRescueBufferSize){
NSLog(#"Unable to allocate enought space for rescue buffer, dropping audio frame");
} else {
if (rescueBuffer == nil) {
rescueBuffer = malloc(kRescueBufferSize);
}
rescueBufferSize = size;
memcpy(rescueBuffer, abl.mBuffers[0].mData, size);
}
}
CFRelease(blockBuffer);
if (!self.hasBuffer && bytesCopied > 0)
{
self.hasBuffer = YES;
}
}
Next I call processOutput. This will do a manual reder on the outputUnit. When AudioUnitRender is called it invokes the playbackCallback below, which is what is hooked up as input callback on my first node. playbackCallback pulls the data off the circular buffer and feeds it into the audioBufferList passed in. Like I said before if the output is set as RemoteIO this will cause the audio to correctly be played on the speakers. When AudioUnitRender finishes, it returns noErr and the bufferList object contains valid data. When I call CMSampleBufferSetDataBufferFromAudioBufferList though I get kCMSampleBufferError_RequiredParameterMissing (-12731).
-(CMSampleBufferRef)processOutput
{
if(self.offline == NO)
{
return NULL;
}
AudioUnitRenderActionFlags flags = 0;
AudioTimeStamp inTimeStamp;
memset(&inTimeStamp, 0, sizeof(AudioTimeStamp));
inTimeStamp.mFlags = kAudioTimeStampSampleTimeValid;
UInt32 busNumber = 0;
UInt32 numberFrames = 512;
inTimeStamp.mSampleTime = 0;
UInt32 channelCount = 2;
AudioBufferList *bufferList = (AudioBufferList*)malloc(sizeof(AudioBufferList)+sizeof(AudioBuffer)*(channelCount-1));
bufferList->mNumberBuffers = channelCount;
for (int j=0; j<channelCount; j++)
{
AudioBuffer buffer = {0};
buffer.mNumberChannels = 1;
buffer.mDataByteSize = numberFrames*sizeof(SInt32);
buffer.mData = calloc(numberFrames,sizeof(SInt32));
bufferList->mBuffers[j] = buffer;
}
CheckError(AudioUnitRender(outputUnit, &flags, &inTimeStamp, busNumber, numberFrames, bufferList), #"AudioUnitRender outputUnit");
CMSampleBufferRef sampleBufferRef = NULL;
CMFormatDescriptionRef format = NULL;
CMSampleTimingInfo timing = { CMTimeMake(1, 44100), kCMTimeZero, kCMTimeInvalid };
AudioStreamBasicDescription audioFormat = self.audioFormat;
CheckError(CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &audioFormat, 0, NULL, 0, NULL, NULL, &format), #"CMAudioFormatDescriptionCreate");
CheckError(CMSampleBufferCreate(kCFAllocatorDefault, NULL, false, NULL, NULL, format, numberFrames, 1, &timing, 0, NULL, &sampleBufferRef), #"CMSampleBufferCreate");
CheckError(CMSampleBufferSetDataBufferFromAudioBufferList(sampleBufferRef, kCFAllocatorDefault, kCFAllocatorDefault, 0, bufferList), #"CMSampleBufferSetDataBufferFromAudioBufferList");
return sampleBufferRef;
}
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
int numberOfChannels = ioData->mBuffers[0].mNumberChannels;
SInt16 *outSample = (SInt16 *)ioData->mBuffers[0].mData;
/
memset(outSample, 0, ioData->mBuffers[0].mDataByteSize);
MyAudioPlayer *p = (__bridge MyAudioPlayer *)inRefCon;
if (p.hasBuffer){
int32_t availableBytes;
SInt16 *bufferTail = TPCircularBufferTail([p getBuffer], &availableBytes);
int32_t requestedBytesSize = inNumberFrames * kUnitSize * numberOfChannels;
int bytesToRead = MIN(availableBytes, requestedBytesSize);
memcpy(outSample, bufferTail, bytesToRead);
TPCircularBufferConsume([p getBuffer], bytesToRead);
if (availableBytes <= requestedBytesSize*2){
[p setReadyForMoreBytes];
}
if (availableBytes <= requestedBytesSize) {
p.hasBuffer = NO;
}
}
return noErr;
}
The CMSampleBufferRef I pass in looks valid (below is a dump of the object from the debugger)
CMSampleBuffer 0x7f87d2a03120 retainCount: 1 allocator: 0x103333180
invalid = NO
dataReady = NO
makeDataReadyCallback = 0x0
makeDataReadyRefcon = 0x0
formatDescription = <CMAudioFormatDescription 0x7f87d2a02b20 [0x103333180]> {
mediaType:'soun'
mediaSubType:'lpcm'
mediaSpecific: {
ASBD: {
mSampleRate: 44100.000000
mFormatID: 'lpcm'
mFormatFlags: 0xc2c
mBytesPerPacket: 2
mFramesPerPacket: 1
mBytesPerFrame: 2
mChannelsPerFrame: 1
mBitsPerChannel: 16 }
cookie: {(null)}
ACL: {(null)}
}
extensions: {(null)}
}
sbufToTrackReadiness = 0x0
numSamples = 512
sampleTimingArray[1] = {
{PTS = {0/1 = 0.000}, DTS = {INVALID}, duration = {1/44100 = 0.000}},
}
dataBuffer = 0x0
The buffer list looks like this
Printing description of bufferList:
(AudioBufferList *) bufferList = 0x00007f87d280b0a0
Printing description of bufferList->mNumberBuffers:
(UInt32) mNumberBuffers = 2
Printing description of bufferList->mBuffers:
(AudioBuffer [1]) mBuffers = {
[0] = (mNumberChannels = 1, mDataByteSize = 2048, mData = 0x00007f87d3008c00)
}
Really at a loss here, hoping someone can help. Thanks,
In case it matters i am debugging this in ios 8.3 simulator and the audio is coming from a mp4 that i shot on my iphone 6 then saved to my laptop.
I have read the following issues, however still to no avail, things are not working.
How to convert AudioBufferList to CMSampleBuffer?
Converting an AudioBufferList to a CMSampleBuffer Produces Unexpected Results
CMSampleBufferSetDataBufferFromAudioBufferList returning error 12731
core audio offline rendering GenericOutput
UPDATE
I poked around some more and notice that when my AudioBufferList right before AudioUnitRender runs looks like this:
bufferList->mNumberBuffers = 2,
bufferList->mBuffers[0].mNumberChannels = 1,
bufferList->mBuffers[0].mDataByteSize = 2048
mDataByteSize is numberFrames*sizeof(SInt32), which is 512 * 4. When I look at the AudioBufferList passed in playbackCallback, the list looks like this:
bufferList->mNumberBuffers = 1,
bufferList->mBuffers[0].mNumberChannels = 1,
bufferList->mBuffers[0].mDataByteSize = 1024
not really sure where that other buffer is going, or the other 1024 byte size...
if when i get finished calling Redner if I do something like this
AudioBufferList newbuff;
newbuff.mNumberBuffers = 1;
newbuff.mBuffers[0] = bufferList->mBuffers[0];
newbuff.mBuffers[0].mDataByteSize = 1024;
and pass newbuff off to CMSampleBufferSetDataBufferFromAudioBufferList the error goes away.
If I try setting the size of BufferList to have 1 mNumberBuffers or its mDataByteSize to be numberFrames*sizeof(SInt16) I get a -50 when calling AudioUnitRender
UPDATE 2
I hooked up a render callback so I can inspect the output when I play the sound over the speakers. I noticed that the output that goes to the speakers also has a AudioBufferList with 2 buffers, and the mDataByteSize during the input callback is 1024 and in the render callback its 2048, which is the same as I have been seeing when manually calling AudioUnitRender. When I inspect the data in the rendered AudioBufferList I notice that the bytes in the 2 buffers are the same, which means I can just ignore the second buffer. But I am not sure how to handle the fact that the data is 2048 in size after being rendered instead of 1024 as it's being taken in. Any ideas on why that could be happening? Is it in more of a raw form after going through the audio graph and that is why the size is doubling?
Sounds like the issue you're dealing with is because of a discrepancy in the number of channels. The reason you're seeing data in blocks of 2048 instead of 1024 is because it is feeding you back two channels (stereo). Check to make sure all of your audio units are properly configured to use mono throughout the entire audio graph, including the Pitch Unit and any audio format descriptions.
One thing to especially beware of is that calls to AudioUnitSetProperty can fail - so be sure to wrap those in CheckError() as well.
So, basically I want to play some audio files (mp3 and caf mostly). But the callback never gets called. Only when I call them to prime the queue.
Here's my data struct:
struct AQPlayerState
{
CAStreamBasicDescription mDataFormat;
AudioQueueRef mQueue;
AudioQueueBufferRef mBuffers[kBufferNum];
AudioFileID mAudioFile;
UInt32 bufferByteSize;
SInt64 mCurrentPacket;
UInt32 mNumPacketsToRead;
AudioStreamPacketDescription *mPacketDescs;
bool mIsRunning;
};
Here's my callback function:
static void HandleOutputBuffer (void *aqData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer)
{
NSLog(#"HandleOutput");
AQPlayerState *pAqData = (AQPlayerState *) aqData;
if (pAqData->mIsRunning == false) return;
UInt32 numBytesReadFromFile;
UInt32 numPackets = pAqData->mNumPacketsToRead;
AudioFileReadPackets (pAqData->mAudioFile,
false,
&numBytesReadFromFile,
pAqData->mPacketDescs,
pAqData->mCurrentPacket,
&numPackets,
inBuffer->mAudioData);
if (numPackets > 0) {
inBuffer->mAudioDataByteSize = numBytesReadFromFile;
AudioQueueEnqueueBuffer (pAqData->mQueue,
inBuffer,
(pAqData->mPacketDescs ? numPackets : 0),
pAqData->mPacketDescs);
pAqData->mCurrentPacket += numPackets;
} else {
// AudioQueueStop(pAqData->mQueue, false);
// AudioQueueDispose(pAqData->mQueue, true);
// AudioFileClose (pAqData->mAudioFile);
// free(pAqData->mPacketDescs);
// free(pAqData->mFloatBuffer);
pAqData->mIsRunning = false;
}
}
And here's my method:
- (void)playFile
{
AQPlayerState aqData;
// get the source file
NSString *p = [[NSBundle mainBundle] pathForResource:#"1_Female" ofType:#"mp3"];
NSURL *url2 = [NSURL fileURLWithPath:p];
CFURLRef srcFile = (__bridge CFURLRef)url2;
OSStatus result = AudioFileOpenURL(srcFile, 0x1/*fsRdPerm*/, 0/*inFileTypeHint*/, &aqData.mAudioFile);
CFRelease (srcFile);
CheckError(result, "Error opinning sound file");
UInt32 size = sizeof(aqData.mDataFormat);
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyDataFormat, &size, &aqData.mDataFormat),
"Error getting file's data format");
CheckError(AudioQueueNewOutput(&aqData.mDataFormat, HandleOutputBuffer, &aqData, CFRunLoopGetCurrent(), kCFRunLoopCommonModes, 0, &aqData.mQueue),
"Error AudioQueueNewOutPut");
// we need to calculate how many packets we read at a time and how big a buffer we need
// we base this on the size of the packets in the file and an approximate duration for each buffer
{
bool isFormatVBR = (aqData.mDataFormat.mBytesPerPacket == 0 || aqData.mDataFormat.mFramesPerPacket == 0);
// first check to see what the max size of a packet is - if it is bigger
// than our allocation default size, that needs to become larger
UInt32 maxPacketSize;
size = sizeof(maxPacketSize);
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyPacketSizeUpperBound, &size, &maxPacketSize),
"Error getting max packet size");
// adjust buffer size to represent about a second of audio based on this format
CalculateBytesForTime(aqData.mDataFormat, maxPacketSize, 1.0/*seconds*/, &aqData.bufferByteSize, &aqData.mNumPacketsToRead);
if (isFormatVBR) {
aqData.mPacketDescs = new AudioStreamPacketDescription [aqData.mNumPacketsToRead];
} else {
aqData.mPacketDescs = NULL; // we don't provide packet descriptions for constant bit rate formats (like linear PCM)
}
printf ("Buffer Byte Size: %d, Num Packets to Read: %d\n", (int)aqData.bufferByteSize, (int)aqData.mNumPacketsToRead);
}
// if the file has a magic cookie, we should get it and set it on the AQ
size = sizeof(UInt32);
result = AudioFileGetPropertyInfo(aqData.mAudioFile, kAudioFilePropertyMagicCookieData, &size, NULL);
if (!result && size) {
char* cookie = new char [size];
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyMagicCookieData, &size, cookie),
"Error getting cookie from file");
CheckError(AudioQueueSetProperty(aqData.mQueue, kAudioQueueProperty_MagicCookie, cookie, size),
"Error setting cookie to file");
delete[] cookie;
}
aqData.mCurrentPacket = 0;
for (int i = 0; i < kBufferNum; ++i) {
CheckError(AudioQueueAllocateBuffer (aqData.mQueue,
aqData.bufferByteSize,
&aqData.mBuffers[i]),
"Error AudioQueueAllocateBuffer");
HandleOutputBuffer (&aqData,
aqData.mQueue,
aqData.mBuffers[i]);
}
// set queue's gain
Float32 gain = 1.0;
CheckError(AudioQueueSetParameter (aqData.mQueue,
kAudioQueueParam_Volume,
gain),
"Error AudioQueueSetParameter");
aqData.mIsRunning = true;
CheckError(AudioQueueStart(aqData.mQueue,
NULL),
"Error AudioQueueStart");
}
And the output when I press play:
Buffer Byte Size: 40310, Num Packets to Read: 38
HandleOutput start
HandleOutput start
HandleOutput start
I tryed replacing CFRunLoopGetCurrent() with CFRunLoopGetMain() and CFRunLoopCommonModes with CFRunLoopDefaultMode, but nothing.
Shouldn't the primed buffers start playing right away I start the queue?
When I start the queue, no callbacks are bang fired.
What am I doing wrong? Thanks for any ideas
What you are basically trying to do here is a basic example of audio playback using Audio Queues. Without looking at your code in detail to see what's missing (that could take a while) i'd rather recommend to you to follow the steps in this basic sample code that does exactly what you're doing (without the extras that aren't really relevant.. for example why are you trying to add audio gain?)
Somewhere else you were trying to play audio using audio units. Audio units are more complex than basic audio queue playback, and I wouldn't attempt them before being very comfortable with audio queues. But you can look at this example project for a basic example of audio queues.
In general when it comes to Core Audio programming in iOS, it's best you take your time with the basic examples and build your way up.. the problem with a lot of tutorials online is that they add extra stuff and often mix it with obj-c code.. when Core Audio is purely C code (ie the extra stuff won't add anything to the learning process). I strongly recommend you go over the book Learning Core Audio if you haven't already. All the sample code is available online, but you can also clone it from this repo for convenience. That's how I learned core audio. It takes time :)
I've successfully recorded audio from the microphone into an audio file using Audio Units with the help of openframeworks and this website http://atastypixel.com/blog/using-remoteio-audio-unit.
I want to be able to stream the file back to audio units and play the audio. According to Play an audio file using RemoteIO and Audio Unit I can use ExtAudioFileOpenURL and ExtAudioFileRead. However, how do I play audio data in my buffer?
This is what I currently have:
static OSStatus setupAudioFileRead() {
//construct the file destination URL
CFURLRef destinationURL = audioSystemFileURL();
OSStatus status = ExtAudioFileOpenURL(destinationURL, &audioFileRef);
CFRelease(destinationURL);
if (checkStatus(status)) { ofLog(OF_LOG_ERROR, "ofxiPhoneSoundStream: Couldn't open file to read"); return status; }
while( TRUE ) {
// Try to fill the buffer to capacity.
UInt32 framesRead = 8000;
status = ExtAudioFileRead( audioFileRef, &framesRead, &inputBufferList );
// error check
if( checkStatus(status) ) { break; }
// 0 frames read means EOF.
if( framesRead == 0 ) { break; }
//play audio???
}
return noErr;
}
From this author: http://atastypixel.com/blog/using-remoteio-audio-unit/, if you scroll down to the PLAYBACK section, try something like this:
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
// Notes: ioData contains buffers (may be more than one!)
// Fill them up as much as you can. Remember to set the size value in each buffer to match how
// much data is in the buffer.
for (int i=0; i < ioData->mNumberBuffers; i++)
{
AudioBuffer buffer = ioData->mBuffers[i];
// copy from your whatever buffer data to output buffer
UInt32 size = min(buffer.mDataByteSize, your buffer.size);
memcpy(buffer.mData, your buffer, size);
buffer.mDataByteSize = size; // indicate how much data we wrote in the buffer
// To test if your Audio Unit setup is working - comment out the three
// lines above and uncomment the for loop below to hear random noise
/*
UInt16 *frameBuffer = buffer.mData;
for (int j = 0; j < inNumberFrames; j++) {
frameBuffer[j] = rand();
}
*/
}
return noErr;
}
If you are only looking for recording from MIC to a file and play it back, the Apple's Speakhere sample is probably much more ready to use.
Basically,
1. Create a RemoteIO unit (See references about how to create RemoteIO);
Create a FilePlayer audio unit which is a dedicated audio unit to read an audio file and provide audio data in the file to output units, for example, the RemoteIO unit created in step 1. To actually use the FilePlayer, a lot of settings (specify which file to play, which part of the file to play, etc.) are needed to be done on the it;
Set kAudioUnitProperty_SetRenderCallback and kAudioUnitProperty_StreamFormat properties of the RemoteIO unit. The first property is essentially a callback function from which the RemoteIO unit pulls audio data and play it. The second property must be set in accordance to StreamFormat that supported by the FilePlayer. It can be derived from a get-property function invoked on the FilePlayer.
Define the callback set in step 3 where the most important thing to do is asking the FilePlayer to render into the buffer provided by the callback for which you will need to invoke AudioUnitRender() on the FilePlayer.
Finally start the RemoteIO unit to play the file.
Above is just a preliminary outline of basic things to do to play files using audio units on iOS. You can refer to Chris Adamson and Kevin Avila's Learning Core Audio for details.
It's a relatively simple approach that utilizes the audio unit mentioned in the Tasty Pixel blog. In the recording callback, instead of filling the buffer with data from the microphone, you could fill it with data from the file using ExtAudioFileRead. I'll try and paste an example below. Mind you this will just work for .caf files.
In the start method call an readAudio or initAudioFile function, something that just gets all the info about the file.
- (void) start {
readAudio();
OSStatus status = AudioOutputUnitStart(audioUnit);
checkStatus(status);
}
Now in the readAudio method you initialize the audio file reference as such.
ExtAudioFileRef fileRef;
void readAudio() {
NSString * name = #"AudioFile";
NSString * source = [[NSBundle mainBundle] pathForResource:name ofType:#"caf"];
const char * cString = [source cStringUsingEncoding:NSASCIIStringEncoding];
CFStringRef str = CFStringCreateWithCString(NULL, cString, kCFStringEncodingMacRoman);
CFURLRef inputFileURL = CFURLCreateWithFileSystemPath(kCFAllocatorDefault, str, kCFURLPOSIXPathStyle, false);
AudioFileID fileID;
OSStatus err = AudioFileOpenURL(inputFileURL, kAudioFileReadPermission, 0, &fileID);
CheckError(err, "AudioFileOpenURL");
err = ExtAudioFileOpenURL(inputFileURL, &fileRef);
CheckError(err, "ExtAudioFileOpenURL");
err = ExtAudioFileSetProperty(fileRef, kExtAudioFileProperty_ClientDataFormat, sizeof(AudioStreamBasicDescription), &audioFormat);
CheckError(err, "ExtAudioFileSetProperty");
}
Now that you have the Audio Data at hand, next step is pretty easy. In the recordingCallback read the data from the file instead of the mic.
static OSStatus recordingCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
// Because of the way our audio format (setup below) is chosen:
// we only need 1 buffer, since it is mono
// Samples are 16 bits = 2 bytes.
// 1 frame includes only 1 sample
AudioBuffer buffer;
buffer.mNumberChannels = 1;
buffer.mDataByteSize = inNumberFrames * 2;
buffer.mData = malloc( inNumberFrames * 2 );
// Put buffer in a AudioBufferList
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = buffer;
// Then:
// Obtain recorded samples
OSStatus err = ExtAudioFileRead(fileRef, &inNumberFrames, &bufferList);
// Now, we have the samples we just read sitting in buffers in bufferList
// Process the new data
[iosAudio processAudio:&bufferList];
// release the malloc'ed data in the buffer we created earlier
free(bufferList.mBuffers[0].mData);
return noErr;
}
This worked for me.