My goal is to be able to play multiple sounds (in order to do a Drum kit), using the Superpowered API since I need very low latency.
I can't find the overall architecture in the .mm file.
For now, I have in my .mm file :
1 SuperpoweredIOSAudioIO* output
10 SuperpoweredAdvancedAudioPlayer* (players[10]), since I have 10 drums
But I can't find how to implement the rest of the .mm file, in particular the audioProcessing loop.
I have this currently:
static bool audioProcessing(Superpowered *clientdata, float **buffers, unsigned int inputChannels, unsigned int outputChannels, unsigned int numberOfSamples, unsigned int samplerate, uint64_t hostTime) {
Superpowered *self = clientdata;
bool overallSilence = true;
for(int i = 0; i < 10; i++) {
SuperpoweredAdvancedAudioPlayer *player = self->players[i];
bool silence = !player->process(self->stereoBuffer, false, numberOfSamples);
if (silence == false) {
overallSilence = false;
}
}
if (!overallSilence) SuperpoweredDeInterleave(self->stereoBuffer, buffers[0], buffers[1], numberOfSamples); // The stereoBuffer is ready now, let's put the finished audio into the requested buffers.
return !overallSilence;
}
and when I want to play a sound:
- (void)playWithPlayerAtIndex:(int) index {
SuperpoweredAdvancedAudioPlayer* player = players[index];
player->seek(0);
player->play(false);
}
But with this, I can only play hear one sound at a time.
Does anyone know the Superpowered API?
All your players are "playing" into "self->stereoBuffer". Check the bufferAdd parameter of the player's process method. You need to "add" the output to the contents of the buffer, if at least one player did output itself into it.
Related
I have a project where I need to decode h264 video from a live network stream and eventually end up with a texture I can display in another framework (Unity3D) on iOS devices. I can successfully decode the video using VTDecompressionSession and then grab the texture with CVMetalTextureCacheCreateTextureFromImage (or the OpenGL variant). It works great when I use a low-latency encoder and the image buffers come out in display order, however, when I use the regular encoder the image buffers do not come out in display order and reordering the image buffers is apparently far more difficult that I expected.
The first attempt was to set the VTDecodeFrameFlags with kVTDecodeFrame_EnableAsynchronousDecompression and kVTDecodeFrame_EnableTemporalProcessing... However, it turns out that VTDecompressionSession can choose to ignore the flag and do whatever it wants... and in my case, it chooses to ignore the flag and still outputs the buffer in encoder order (not display order). Essentially useless.
The next attempt was to associate the image buffers with the presentation time stamp and then throw them into a vector which would allow me to grab the image buffer I needed when I create the texture. The problem seems to be that the image buffer that goes into the VTDecompressionSession, which is associated with a time stamp, is no longer the same buffer that comes out, essentially making the time stamp useless.
For example, going into the decoder...
VTDecodeFrameFlags flags = kVTDecodeFrame_EnableAsynchronousDecompression;
VTDecodeInfoFlags flagOut;
// Presentation time stamp to be passed with the buffer
NSNumber *nsPts = [NSNumber numberWithDouble:pts];
VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flags,
(void*)CFBridgingRetain(nsPts), &flagOut);
On the callback side...
void decompressionSessionDecodeFrameCallback(void *decompressionOutputRefCon, void *sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CVImageBufferRef imageBuffer, CMTime presentationTimeStamp, CMTime presentationDuration)
{
// The presentation time stamp...
// No longer seems to be associated with the buffer that it went in with!
NSNumber* pts = CFBridgingRelease(sourceFrameRefCon);
}
When ordered, the time stamps on the callback side increase monotonically at the expected rate, but the buffers are not in the right order. Does anyone see where I am making an error here? Or know how to determine the order of the buffers on the callback side? At this point I have tried just about everything I can think of.
In my case, the problem wasn't with VTDecompressionSession, it was a problem with the demuxer getting the wrong PTS. While I couldn't get VTDecompressionSession to put out the frames in temporal (display) order with the kVTDecodeFrame_EnableAsynchronousDecompression and kVTDecodeFrame_EnableTemporalProcessing flags, I could sort the frames myself based on PTS with a small vector.
First, make sure you associate all of your timing information with your CMSampleBuffer along with the block buffer so you receive it in the VTDecompressionSession callback.
// Wrap our CMBlockBuffer in a CMSampleBuffer...
CMSampleBufferRef sampleBuffer;
CMTime duration = ...;
CMTime presentationTimeStamp = ...;
CMTime decompressTimeStamp = ...;
CMSampleTimingInfo timingInfo{duration, presentationTimeStamp, decompressTimeStamp};
_sampleTimingArray[0] = timingInfo;
_sampleSizeArray[0] = nalLength;
// Wrap the CMBlockBuffer...
status = CMSampleBufferCreate(kCFAllocatorDefault, blockBuffer, true, NULL, NULL, _formatDescription, 1, 1, _sampleTimingArray, 1, _sampleSizeArray, &sampleBuffer);
Then, decode the frame... It is worth trying to get the frames out in display order with the flags.
VTDecodeFrameFlags flags = kVTDecodeFrame_EnableAsynchronousDecompression | kVTDecodeFrame_EnableTemporalProcessing;
VTDecodeInfoFlags flagOut;
VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flags,
(void*)CFBridgingRetain(NULL), &flagOut);
On the callback side of things, we need a way of sorting the CVImageBufferRefs we receive. I use a struct that contains the CVImageBufferRef and the PTS. Then a vector with a size of two that will do the actual sorting.
struct Buffer
{
CVImageBufferRef imageBuffer = NULL;
double pts = 0;
};
std::vector <Buffer> _buffer;
We also need a way to sort the Buffers. Always writing to and reading from the index with the lowest PTS works well.
-(int) getMinIndex
{
if(_buffer[0].pts > _buffer[1].pts)
{
return 1;
}
return 0;
}
In the callback, we need to fill the vector with Buffers...
void decompressionSessionDecodeFrameCallback(void *decompressionOutputRefCon, void *sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CVImageBufferRef imageBuffer, CMTime presentationTimeStamp, CMTime presentationDuration)
{
StreamManager *streamManager = (__bridge StreamManager *)decompressionOutputRefCon;
#synchronized(streamManager)
{
if (status != noErr)
{
NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:status userInfo:nil];
NSLog(#"Decompressed error: %#", error);
}
else
{
// Get the PTS
double pts = CMTimeGetSeconds(presentationTimeStamp);
// Fill our buffer initially
if(!streamManager->_bufferReady)
{
Buffer buffer;
buffer.pts = pts;
buffer.imageBuffer = imageBuffer;
CVBufferRetain(buffer.imageBuffer);
streamManager->_buffer[streamManager->_bufferIndex++] = buffer;
}
else
{
// Push new buffers to the index with the lowest PTS
int index = [streamManager getMinIndex];
// Release the old CVImageBufferRef
CVBufferRelease(streamManager->_buffer[index].imageBuffer);
Buffer buffer;
buffer.pts = pts;
buffer.imageBuffer = imageBuffer;
// Retain the new CVImageBufferRef
CVBufferRetain(buffer.imageBuffer);
streamManager->_buffer[index] = buffer;
}
// Wrap around the buffer when initialized
// _bufferWindow = 2
if(streamManager->_bufferIndex == streamManager->_bufferWindow)
{
streamManager->_bufferReady = YES;
streamManager->_bufferIndex = 0;
}
}
}
}
Finally we need to drain the Buffers in temporal (display) order...
- (void)drainBuffer
{
#synchronized(self)
{
if(_bufferReady)
{
// Drain buffers from the index with the lowest PTS
int index = [self getMinIndex];
Buffer buffer = _buffer[index];
// Do something useful with the buffer now in display order
}
}
}
I would like to improve upon that answer a bit. While the outlined solution works, it requires knowledge of the number of frames needed to produce an output frame. The example uses a buffer size of 2, but in my case I needed a buffer size of 3.
To avoid having to specify this in advance one can make use of the fact, that frames (in display order) align exactly in terms of pts/duration. I.e. the end of one frame is exactly the beginning of the next. Thus one can simply accumulate frames until there is no "gap" at the beginning, then pop the first frame, and so on. Also one can take the pts of the first frame (which is always an I-frame) as the initial "head" (as it does not have to be zero...).
Here is some code that does this:
#include <CoreVideo/CVImageBuffer.h>
#include <boost/container/flat_set.hpp>
inline bool operator<(const CMTime& left, const CMTime& right)
{
return CMTimeCompare(left, right) < 0;
}
inline bool operator==(const CMTime& left, const CMTime& right)
{
return CMTimeCompare(left, right) == 0;
}
inline CMTime operator+(const CMTime& left, const CMTime& right)
{
return CMTimeAdd(left, right);
}
class reorder_buffer_t
{
public:
struct entry_t
{
CFGuard<CVImageBufferRef> image;
CMTime pts;
CMTime duration;
bool operator<(const entry_t& other) const
{
return pts < other.pts;
}
};
private:
typedef boost::container::flat_set<entry_t> buffer_t;
public:
reorder_buffer_t()
{
}
void push(entry_t entry)
{
if (!_head)
_head = entry.pts;
_buffer.insert(std::move(entry));
}
bool empty() const
{
return _buffer.empty();
}
bool ready() const
{
return !empty() && _buffer.begin()->pts == _head;
}
entry_t pop()
{
assert(ready());
auto entry = *_buffer.begin();
_buffer.erase(_buffer.begin());
_head = entry.pts + entry.duration;
return entry;
}
void clear()
{
_buffer.clear();
_head = boost::none;
}
private:
boost::optional<CMTime> _head;
buffer_t _buffer;
};
Here's a solution that works with any required buffer size, and also does not need any 3rd party libraries. My C++ code might not be the best, but it works.
We create a Buffer struct to identify the buffers by pts:
struct Buffer
{
CVImageBufferRef imageBuffer = NULL;
uint64_t pts = 0;
};
In our decoder, we need to keep track of the buffers, and what pts we want to release next:
#property (nonatomic) std::vector <Buffer> buffers;
#property (nonatomic, assign) uint64_t nextExpectedPts;
Now we are ready to handle the buffers coming in. In my case the buffers were provided asynchronously. Make sure you provide the correct duration and presentation timestamp values to the decompressionsession to be able to sort them properly:
-(void)handleImageBuffer:(CVImageBufferRef)imageBuffer pts:(CMTime)presentationTimeStamp duration:(uint64_t)duration {
//Situation 1, we can directly pass over this buffer
if (self.nextExpectedPts == presentationTimeStamp.value || duration == 0) {
[self sendImageBuffer:imageBuffer duration:duration];
return;
}
//Situation 2, we got this buffer too fast. We will store it, but first we check if we have already stored the expected buffer
Buffer futureBuffer = [self bufferWithImageBuffer:imageBuffer pts:presentationTimeStamp.value];
int smallestPtsInBufferIndex = [self getSmallestPtsBufferIndex];
if (smallestPtsInBufferIndex >= 0 && self.nextExpectedPts == self.buffers[smallestPtsInBufferIndex].pts) {
//We found the next buffer, lets store the current buffer and return this one
Buffer bufferWithSmallestPts = self.buffers[smallestPtsInBufferIndex];
[self sendImageBuffer:bufferWithSmallestPts.imageBuffer duration:duration];
CVBufferRelease(bufferWithSmallestPts.imageBuffer);
[self setBuffer:futureBuffer atIndex:smallestPtsInBufferIndex];
} else {
//We dont have the next buffer yet, lets store this one to a new slot
[self setBuffer:futureBuffer atIndex:self.buffers.size()];
}
}
-(Buffer)bufferWithImageBuffer:(CVImageBufferRef)imageBuffer pts:(uint64_t)pts {
Buffer futureBuffer = Buffer();
futureBuffer.pts = pts;
futureBuffer.imageBuffer = imageBuffer;
CVBufferRetain(futureBuffer.imageBuffer);
return futureBuffer;
}
- (void)sendImageBuffer:(CVImageBufferRef)imageBuffer duration:(uint64_t)duration {
//Send your buffer to wherever you need it here
self.nextExpectedPts += duration;
}
-(int) getSmallestPtsBufferIndex
{
int minIndex = -1;
uint64_t minPts = 0;
for(int i=0;i<_buffers.size();i++) {
if (_buffers[i].pts < minPts || minPts == 0) {
minPts = _buffers[i].pts;
minIndex = i;
}
}
return minIndex;
}
- (void)setBuffer:(Buffer)buffer atIndex:(int)index {
if (_buffers.size() <= index) {
_buffers.push_back(buffer);
} else {
_buffers[index] = buffer;
}
}
Do not forget to release all the buffers in the vector when deallocating your decoder, and if you're working with a looping file for example, keep track of when the file has fully looped to reset the nextExpectedPts and such.
I was trying to implement a function to stretch the sound speed, without changing it's pitch and time scale.
I try the method to set the frequency of channel to slow of fast the speed.
Then use FMOD_DSP_PITCHSHIFT to correct the pitch sounds as default.
I was using wav format sound file for test and build function.
I'm trying to intergrate product resource which sound file was encoded as MP3.
PITCHSHIFT DSP doesn't work at MP3 sound channel. console log looks fine with no exception & error.
Same project and setting everything works fine in iOS Simulator.
After some research and experiments, results indicates even m4a works fine at iOS.
I wonder is this some kind of bug? or I missed something at configuration.
sample code was based on FMOD Sample project Play stream.
`/*==============================================================================
Play Stream Example
Copyright (c), Firelight Technologies Pty, Ltd 2004-2015.
This example shows how to simply play a stream such as an MP3 or WAV. The stream
behaviour is achieved by specifying FMOD_CREATESTREAM in the call to
System::createSound. This makes FMOD decode the file in realtime as it plays,
instead of loading it all at once which uses far less memory in exchange for a
small runtime CPU hit.
==============================================================================*/
#include "fmod.hpp"
#include "common.h"
int FMOD_Main()
{
FMOD::System *system;
FMOD::Sound *sound, *sound_to_play;
FMOD::Channel *channel = 0;
FMOD_RESULT result;
FMOD::DSP * pitch_shift;
unsigned int version;
void *extradriverdata = 0;
int numsubsounds;
Common_Init(&extradriverdata);
/*
Create a System object and initialize.
*/
result = FMOD::System_Create(&system);
ERRCHECK(result);
result = system->getVersion(&version);
ERRCHECK(result);
if (version < FMOD_VERSION)
{
Common_Fatal("FMOD lib version %08x doesn't match header version %08x", version, FMOD_VERSION);
}
result = system->init(32, FMOD_INIT_NORMAL, extradriverdata);
ERRCHECK(result);
result = system->createDSPByType(FMOD_DSP_TYPE_PITCHSHIFT, &pitch_shift);
ERRCHECK(result);
/*
This example uses an FSB file, which is a preferred pack format for fmod containing multiple sounds.
This could just as easily be exchanged with a wav/mp3/ogg file for example, but in this case you wouldnt need to call getSubSound.
Because getNumSubSounds is called here the example would work with both types of sound file (packed vs single).
*/
result = system->createSound(Common_MediaPath("aaa.m4a"), FMOD_LOOP_NORMAL | FMOD_2D, 0, &sound);
ERRCHECK(result);
result = sound->getNumSubSounds(&numsubsounds);
ERRCHECK(result);
if (numsubsounds)
{
sound->getSubSound(0, &sound_to_play);
ERRCHECK(result);
}
else
{
sound_to_play = sound;
}
/*
Play the sound.
*/
result = system->playSound(sound_to_play, 0, false, &channel);
ERRCHECK(result);
result = channel->addDSP(0, pitch_shift);
ERRCHECK(result);
float pitch = 1.f;
result = pitch_shift->setParameterFloat(FMOD_DSP_PITCHSHIFT_PITCH, pitch);
ERRCHECK(result);
pitch_shift->setActive(true);
ERRCHECK(result);
float defaultFrequency;
result = channel->getFrequency(&defaultFrequency);
ERRCHECK(result);
/*
Main loop.
*/
do
{
Common_Update();
if (Common_BtnPress(BTN_ACTION1))
{
bool paused;
result = channel->getPaused(&paused);
ERRCHECK(result);
result = channel->setPaused(!paused);
ERRCHECK(result);
}
if (Common_BtnPress(BTN_DOWN)) {
char valuestr;
int valuestrlen;
pitch_shift->getParameterFloat(FMOD_DSP_PITCHSHIFT_PITCH, &pitch, &valuestr, valuestrlen);
pitch+=0.1f;
pitch = pitch>2.0f?2.0f:pitch;
pitch_shift->setParameterFloat(FMOD_DSP_PITCHSHIFT_PITCH, pitch);
channel->setFrequency(defaultFrequency/pitch);
}
if (Common_BtnPress(BTN_UP)) {
char valuestr;
int valuestrlen;
pitch_shift->getParameterFloat(FMOD_DSP_PITCHSHIFT_PITCH, &pitch, &valuestr, valuestrlen);
pitch-=0.1f;
pitch = pitch<0.5f?0.5f:pitch;
pitch_shift->setParameterFloat(FMOD_DSP_PITCHSHIFT_PITCH, pitch);
channel->setFrequency(defaultFrequency/pitch);
}
result = system->update();
ERRCHECK(result);
{
unsigned int ms = 0;
unsigned int lenms = 0;
bool playing = false;
bool paused = false;
if (channel)
{
result = channel->isPlaying(&playing);
if ((result != FMOD_OK) && (result != FMOD_ERR_INVALID_HANDLE))
{
ERRCHECK(result);
}
result = channel->getPaused(&paused);
if ((result != FMOD_OK) && (result != FMOD_ERR_INVALID_HANDLE))
{
ERRCHECK(result);
}
result = channel->getPosition(&ms, FMOD_TIMEUNIT_MS);
if ((result != FMOD_OK) && (result != FMOD_ERR_INVALID_HANDLE))
{
ERRCHECK(result);
}
result = sound_to_play->getLength(&lenms, FMOD_TIMEUNIT_MS);
if ((result != FMOD_OK) && (result != FMOD_ERR_INVALID_HANDLE))
{
ERRCHECK(result);
}
}
Common_Draw("==================================================");
Common_Draw("Play Stream Example.");
Common_Draw("Copyright (c) Firelight Technologies 2004-2015.");
Common_Draw("==================================================");
Common_Draw("");
Common_Draw("Press %s to toggle pause", Common_BtnStr(BTN_ACTION1));
Common_Draw("Press %s to quit", Common_BtnStr(BTN_QUIT));
Common_Draw("");
Common_Draw("Time %02d:%02d:%02d/%02d:%02d:%02d : %s", ms / 1000 / 60, ms / 1000 % 60, ms / 10 % 100, lenms / 1000 / 60, lenms / 1000 % 60, lenms / 10 % 100, paused ? "Paused " : playing ? "Playing" : "Stopped");
Common_Draw("Pitch %02f",pitch);
}
Common_Sleep(50);
} while (!Common_BtnPress(BTN_QUIT));
/*
Shut down
*/
result = sound->release(); /* Release the parent, not the sound that was retrieved with getSubSound. */
ERRCHECK(result);
result = system->close();
ERRCHECK(result);
result = system->release();
ERRCHECK(result);
Common_Close();
return 0;
}
`
After some more experiments , i can approach the destination through switch sound file format to m4a on iOS Device.
MP3 still not working .
So, basically I want to play some audio files (mp3 and caf mostly). But the callback never gets called. Only when I call them to prime the queue.
Here's my data struct:
struct AQPlayerState
{
CAStreamBasicDescription mDataFormat;
AudioQueueRef mQueue;
AudioQueueBufferRef mBuffers[kBufferNum];
AudioFileID mAudioFile;
UInt32 bufferByteSize;
SInt64 mCurrentPacket;
UInt32 mNumPacketsToRead;
AudioStreamPacketDescription *mPacketDescs;
bool mIsRunning;
};
Here's my callback function:
static void HandleOutputBuffer (void *aqData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer)
{
NSLog(#"HandleOutput");
AQPlayerState *pAqData = (AQPlayerState *) aqData;
if (pAqData->mIsRunning == false) return;
UInt32 numBytesReadFromFile;
UInt32 numPackets = pAqData->mNumPacketsToRead;
AudioFileReadPackets (pAqData->mAudioFile,
false,
&numBytesReadFromFile,
pAqData->mPacketDescs,
pAqData->mCurrentPacket,
&numPackets,
inBuffer->mAudioData);
if (numPackets > 0) {
inBuffer->mAudioDataByteSize = numBytesReadFromFile;
AudioQueueEnqueueBuffer (pAqData->mQueue,
inBuffer,
(pAqData->mPacketDescs ? numPackets : 0),
pAqData->mPacketDescs);
pAqData->mCurrentPacket += numPackets;
} else {
// AudioQueueStop(pAqData->mQueue, false);
// AudioQueueDispose(pAqData->mQueue, true);
// AudioFileClose (pAqData->mAudioFile);
// free(pAqData->mPacketDescs);
// free(pAqData->mFloatBuffer);
pAqData->mIsRunning = false;
}
}
And here's my method:
- (void)playFile
{
AQPlayerState aqData;
// get the source file
NSString *p = [[NSBundle mainBundle] pathForResource:#"1_Female" ofType:#"mp3"];
NSURL *url2 = [NSURL fileURLWithPath:p];
CFURLRef srcFile = (__bridge CFURLRef)url2;
OSStatus result = AudioFileOpenURL(srcFile, 0x1/*fsRdPerm*/, 0/*inFileTypeHint*/, &aqData.mAudioFile);
CFRelease (srcFile);
CheckError(result, "Error opinning sound file");
UInt32 size = sizeof(aqData.mDataFormat);
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyDataFormat, &size, &aqData.mDataFormat),
"Error getting file's data format");
CheckError(AudioQueueNewOutput(&aqData.mDataFormat, HandleOutputBuffer, &aqData, CFRunLoopGetCurrent(), kCFRunLoopCommonModes, 0, &aqData.mQueue),
"Error AudioQueueNewOutPut");
// we need to calculate how many packets we read at a time and how big a buffer we need
// we base this on the size of the packets in the file and an approximate duration for each buffer
{
bool isFormatVBR = (aqData.mDataFormat.mBytesPerPacket == 0 || aqData.mDataFormat.mFramesPerPacket == 0);
// first check to see what the max size of a packet is - if it is bigger
// than our allocation default size, that needs to become larger
UInt32 maxPacketSize;
size = sizeof(maxPacketSize);
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyPacketSizeUpperBound, &size, &maxPacketSize),
"Error getting max packet size");
// adjust buffer size to represent about a second of audio based on this format
CalculateBytesForTime(aqData.mDataFormat, maxPacketSize, 1.0/*seconds*/, &aqData.bufferByteSize, &aqData.mNumPacketsToRead);
if (isFormatVBR) {
aqData.mPacketDescs = new AudioStreamPacketDescription [aqData.mNumPacketsToRead];
} else {
aqData.mPacketDescs = NULL; // we don't provide packet descriptions for constant bit rate formats (like linear PCM)
}
printf ("Buffer Byte Size: %d, Num Packets to Read: %d\n", (int)aqData.bufferByteSize, (int)aqData.mNumPacketsToRead);
}
// if the file has a magic cookie, we should get it and set it on the AQ
size = sizeof(UInt32);
result = AudioFileGetPropertyInfo(aqData.mAudioFile, kAudioFilePropertyMagicCookieData, &size, NULL);
if (!result && size) {
char* cookie = new char [size];
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyMagicCookieData, &size, cookie),
"Error getting cookie from file");
CheckError(AudioQueueSetProperty(aqData.mQueue, kAudioQueueProperty_MagicCookie, cookie, size),
"Error setting cookie to file");
delete[] cookie;
}
aqData.mCurrentPacket = 0;
for (int i = 0; i < kBufferNum; ++i) {
CheckError(AudioQueueAllocateBuffer (aqData.mQueue,
aqData.bufferByteSize,
&aqData.mBuffers[i]),
"Error AudioQueueAllocateBuffer");
HandleOutputBuffer (&aqData,
aqData.mQueue,
aqData.mBuffers[i]);
}
// set queue's gain
Float32 gain = 1.0;
CheckError(AudioQueueSetParameter (aqData.mQueue,
kAudioQueueParam_Volume,
gain),
"Error AudioQueueSetParameter");
aqData.mIsRunning = true;
CheckError(AudioQueueStart(aqData.mQueue,
NULL),
"Error AudioQueueStart");
}
And the output when I press play:
Buffer Byte Size: 40310, Num Packets to Read: 38
HandleOutput start
HandleOutput start
HandleOutput start
I tryed replacing CFRunLoopGetCurrent() with CFRunLoopGetMain() and CFRunLoopCommonModes with CFRunLoopDefaultMode, but nothing.
Shouldn't the primed buffers start playing right away I start the queue?
When I start the queue, no callbacks are bang fired.
What am I doing wrong? Thanks for any ideas
What you are basically trying to do here is a basic example of audio playback using Audio Queues. Without looking at your code in detail to see what's missing (that could take a while) i'd rather recommend to you to follow the steps in this basic sample code that does exactly what you're doing (without the extras that aren't really relevant.. for example why are you trying to add audio gain?)
Somewhere else you were trying to play audio using audio units. Audio units are more complex than basic audio queue playback, and I wouldn't attempt them before being very comfortable with audio queues. But you can look at this example project for a basic example of audio queues.
In general when it comes to Core Audio programming in iOS, it's best you take your time with the basic examples and build your way up.. the problem with a lot of tutorials online is that they add extra stuff and often mix it with obj-c code.. when Core Audio is purely C code (ie the extra stuff won't add anything to the learning process). I strongly recommend you go over the book Learning Core Audio if you haven't already. All the sample code is available online, but you can also clone it from this repo for convenience. That's how I learned core audio. It takes time :)
I'm having a hard time, searching how to play a video file from a TMemoryStream (or a similar buffer in memory) using FFMpeg. I've seen many things, including UltraStarDX, expensive FFMpeg components for Delphi and so on.
One component called FFMpeg Vcl Player claims to play video formats from a memory stream. I downloaded the trial version and I guess it uses CircularBuffer.pas for that matter (maybe).
Does any one know how to do this?
Edit:
Now the better question is how to play an encrypted video file, using FFMpeg or similar libraries.
To play video from memory stream, you can use custom AVIOContext.
static const int kBufferSize = 4 * 1024;
class my_iocontext_private
{
private:
my_iocontext_private(my_iocontext_private const &);
my_iocontext_private& operator = (my_iocontext_private const &);
public:
my_iocontext_private(IInputStreamPtr inputStream)
: inputStream_(inputStream)
, buffer_size_(kBufferSize)
, buffer_(static_cast<unsigned char*>(::av_malloc(buffer_size_))) {
ctx_ = ::avio_alloc_context(buffer_, buffer_size_, 0, this,
&my_iocontext_private::read, NULL, &my_iocontext_private::seek);
}
~my_iocontext_private() {
::av_free(ctx_);
::av_free(buffer_);
}
void reset_inner_context() { ctx_ = NULL; buffer_ = NULL; }
static int read(void *opaque, unsigned char *buf, int buf_size) {
my_iocontext_private* h = static_cast<my_iocontext_private*>(opaque);
return h->inputStream_->Read(buf, buf_size);
}
static int64_t seek(void *opaque, int64_t offset, int whence) {
my_iocontext_private* h = static_cast<my_iocontext_private*>(opaque);
if (0x10000 == whence)
return h->inputStream_->Size();
return h->inputStream_->Seek(offset, whence);
}
::AVIOContext *get_avio() { return ctx_; }
private:
IInputStreamPtr inputStream_; // abstract stream interface, You can adapt it to TMemoryStream
int buffer_size_;
unsigned char * buffer_;
::AVIOContext * ctx_;
};
//// ..........
/// prepare input stream:
IInputStreamPtr inputStream = MyCustomCreateInputStreamFromMemory();
my_iocontext_private priv_ctx(inputStream);
AVFormatContext * ctx = ::avformat_alloc_context();
ctx->pb = priv_ctx.get_avio();
int err = avformat_open_input(&ctx, "arbitrarytext", NULL, NULL);
if (err < 0)
return -1;
//// normal usage of ctx
//// avformat_find_stream_info(ctx, NULL);
//// av_read_frame(ctx, &pkt);
//// etc..
You can waste your time rewriting FFMPEG from C++ to Delphi, or mess with wrapper libraries.
Or if you're just interested in playing a video in Delphi, then check out Mitov's VideoLab components.
http://www.mitov.com/products/videolab#components
If you want play Stream from memory you can make a virtual memory. I suggest BoxedAppSdk.
This will help you to make a virtual drive with virtual files that you can write on it and then give the virtual path to the player component that you have.
BoxedApp is not free but it is really awesome and very simple in use!
I'm writing an iOS app that streams video and audio over the network.
I am using AVCaptureSession to grab raw video frames using AVCaptureVideoDataOutput and encode them in software using x264. This works great.
I wanted to do the same for audio, only that I don't need that much control on the audio side so I wanted to use the built in hardware encoder to produce an AAC stream. This meant using Audio Converter from the Audio Toolbox layer. In order to do so I put in a handler for AVCaptudeAudioDataOutput's audio frames:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
// get the audio samples into a common buffer _pcmBuffer
CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
CMBlockBufferGetDataPointer(blockBuffer, 0, NULL, &_pcmBufferSize, &_pcmBuffer);
// use AudioConverter to
UInt32 ouputPacketsCount = 1;
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0].mNumberChannels = 1;
bufferList.mBuffers[0].mDataByteSize = sizeof(_aacBuffer);
bufferList.mBuffers[0].mData = _aacBuffer;
OSStatus st = AudioConverterFillComplexBuffer(_converter, converter_callback, (__bridge void *) self, &ouputPacketsCount, &bufferList, NULL);
if (0 == st) {
// ... send bufferList.mBuffers[0].mDataByteSize bytes from _aacBuffer...
}
}
In this case the callback function for the audio converter is pretty simple (assuming packet sizes and counts are setup properly):
- (void) putPcmSamplesInBufferList:(AudioBufferList *)bufferList withCount:(UInt32 *)count
{
bufferList->mBuffers[0].mData = _pcmBuffer;
bufferList->mBuffers[0].mDataByteSize = _pcmBufferSize;
}
And the setup for the audio converter looks like this:
{
// ...
AudioStreamBasicDescription pcmASBD = {0};
pcmASBD.mSampleRate = ((AVAudioSession *) [AVAudioSession sharedInstance]).currentHardwareSampleRate;
pcmASBD.mFormatID = kAudioFormatLinearPCM;
pcmASBD.mFormatFlags = kAudioFormatFlagsCanonical;
pcmASBD.mChannelsPerFrame = 1;
pcmASBD.mBytesPerFrame = sizeof(AudioSampleType);
pcmASBD.mFramesPerPacket = 1;
pcmASBD.mBytesPerPacket = pcmASBD.mBytesPerFrame * pcmASBD.mFramesPerPacket;
pcmASBD.mBitsPerChannel = 8 * pcmASBD.mBytesPerFrame;
AudioStreamBasicDescription aacASBD = {0};
aacASBD.mFormatID = kAudioFormatMPEG4AAC;
aacASBD.mSampleRate = pcmASBD.mSampleRate;
aacASBD.mChannelsPerFrame = pcmASBD.mChannelsPerFrame;
size = sizeof(aacASBD);
AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &size, &aacASBD);
AudioConverterNew(&pcmASBD, &aacASBD, &_converter);
// ...
}
This seems pretty straight forward only the IT DOES NOT WORK. Once the AVCaptureSession is running, the audio converter (specifically AudioConverterFillComplexBuffer) returns an 'hwiu' (hardware in use) error. Conversion works fine if the session is stopped but then I can't capture anything...
I was wondering if there was a way to get an AAC stream out of AVCaptureSession. The options I'm considering are:
Somehow using AVAssetWriterInput to encode audio samples into AAC and then get the encoded packets somehow (not through AVAssetWriter, which would only write to a file).
Reorganizing my app so that it uses AVCaptureSession only on the video side and uses Audio Queues on the audio side. This will make flow control (starting and stopping recording, responding to interruptions) more complicated and I'm afraid that it might cause synching problems between the audio and video. Also, it just doesn't seem like a good design.
Does anyone know if getting the AAC out of AVCaptureSession is possible? Do I have to use Audio Queues here? Could this get me into synching or control problems?
I ended up asking Apple for advice (it turns out you can do that if you have a paid developer account).
It seems that AVCaptureSession grabs a hold of the AAC hardware encoder but only lets you use it to write directly to file.
You can use the software encoder but you have to ask for it specifically instead of using AudioConverterNew:
AudioClassDescription *description = [self
getAudioClassDescriptionWithType:kAudioFormatMPEG4AAC
fromManufacturer:kAppleSoftwareAudioCodecManufacturer];
if (!description) {
return false;
}
// see the question as for setting up pcmASBD and arc ASBD
OSStatus st = AudioConverterNewSpecific(&pcmASBD, &aacASBD, 1, description, &_converter);
if (st) {
NSLog(#"error creating audio converter: %s", OSSTATUS(st));
return false;
}
with
- (AudioClassDescription *)getAudioClassDescriptionWithType:(UInt32)type
fromManufacturer:(UInt32)manufacturer
{
static AudioClassDescription desc;
UInt32 encoderSpecifier = type;
OSStatus st;
UInt32 size;
st = AudioFormatGetPropertyInfo(kAudioFormatProperty_Encoders,
sizeof(encoderSpecifier),
&encoderSpecifier,
&size);
if (st) {
NSLog(#"error getting audio format propery info: %s", OSSTATUS(st));
return nil;
}
unsigned int count = size / sizeof(AudioClassDescription);
AudioClassDescription descriptions[count];
st = AudioFormatGetProperty(kAudioFormatProperty_Encoders,
sizeof(encoderSpecifier),
&encoderSpecifier,
&size,
descriptions);
if (st) {
NSLog(#"error getting audio format propery: %s", OSSTATUS(st));
return nil;
}
for (unsigned int i = 0; i < count; i++) {
if ((type == descriptions[i].mSubType) &&
(manufacturer == descriptions[i].mManufacturer)) {
memcpy(&desc, &(descriptions[i]), sizeof(desc));
return &desc;
}
}
return nil;
}
The software encoder will take up CPU resources, of course, but will get the job done.