Play a Video from MemoryStream, Using FFMpeg - delphi

I'm having a hard time, searching how to play a video file from a TMemoryStream (or a similar buffer in memory) using FFMpeg. I've seen many things, including UltraStarDX, expensive FFMpeg components for Delphi and so on.
One component called FFMpeg Vcl Player claims to play video formats from a memory stream. I downloaded the trial version and I guess it uses CircularBuffer.pas for that matter (maybe).
Does any one know how to do this?
Edit:
Now the better question is how to play an encrypted video file, using FFMpeg or similar libraries.

To play video from memory stream, you can use custom AVIOContext.
static const int kBufferSize = 4 * 1024;
class my_iocontext_private
{
private:
my_iocontext_private(my_iocontext_private const &);
my_iocontext_private& operator = (my_iocontext_private const &);
public:
my_iocontext_private(IInputStreamPtr inputStream)
: inputStream_(inputStream)
, buffer_size_(kBufferSize)
, buffer_(static_cast<unsigned char*>(::av_malloc(buffer_size_))) {
ctx_ = ::avio_alloc_context(buffer_, buffer_size_, 0, this,
&my_iocontext_private::read, NULL, &my_iocontext_private::seek);
}
~my_iocontext_private() {
::av_free(ctx_);
::av_free(buffer_);
}
void reset_inner_context() { ctx_ = NULL; buffer_ = NULL; }
static int read(void *opaque, unsigned char *buf, int buf_size) {
my_iocontext_private* h = static_cast<my_iocontext_private*>(opaque);
return h->inputStream_->Read(buf, buf_size);
}
static int64_t seek(void *opaque, int64_t offset, int whence) {
my_iocontext_private* h = static_cast<my_iocontext_private*>(opaque);
if (0x10000 == whence)
return h->inputStream_->Size();
return h->inputStream_->Seek(offset, whence);
}
::AVIOContext *get_avio() { return ctx_; }
private:
IInputStreamPtr inputStream_; // abstract stream interface, You can adapt it to TMemoryStream
int buffer_size_;
unsigned char * buffer_;
::AVIOContext * ctx_;
};
//// ..........
/// prepare input stream:
IInputStreamPtr inputStream = MyCustomCreateInputStreamFromMemory();
my_iocontext_private priv_ctx(inputStream);
AVFormatContext * ctx = ::avformat_alloc_context();
ctx->pb = priv_ctx.get_avio();
int err = avformat_open_input(&ctx, "arbitrarytext", NULL, NULL);
if (err < 0)
return -1;
//// normal usage of ctx
//// avformat_find_stream_info(ctx, NULL);
//// av_read_frame(ctx, &pkt);
//// etc..

You can waste your time rewriting FFMPEG from C++ to Delphi, or mess with wrapper libraries.
Or if you're just interested in playing a video in Delphi, then check out Mitov's VideoLab components.
http://www.mitov.com/products/videolab#components

If you want play Stream from memory you can make a virtual memory. I suggest BoxedAppSdk.
This will help you to make a virtual drive with virtual files that you can write on it and then give the virtual path to the player component that you have.
BoxedApp is not free but it is really awesome and very simple in use!

Related

How to play multiple sound with the Superpowered library

My goal is to be able to play multiple sounds (in order to do a Drum kit), using the Superpowered API since I need very low latency.
I can't find the overall architecture in the .mm file.
For now, I have in my .mm file :
1 SuperpoweredIOSAudioIO* output
10 SuperpoweredAdvancedAudioPlayer* (players[10]), since I have 10 drums
But I can't find how to implement the rest of the .mm file, in particular the audioProcessing loop.
I have this currently:
static bool audioProcessing(Superpowered *clientdata, float **buffers, unsigned int inputChannels, unsigned int outputChannels, unsigned int numberOfSamples, unsigned int samplerate, uint64_t hostTime) {
Superpowered *self = clientdata;
bool overallSilence = true;
for(int i = 0; i < 10; i++) {
SuperpoweredAdvancedAudioPlayer *player = self->players[i];
bool silence = !player->process(self->stereoBuffer, false, numberOfSamples);
if (silence == false) {
overallSilence = false;
}
}
if (!overallSilence) SuperpoweredDeInterleave(self->stereoBuffer, buffers[0], buffers[1], numberOfSamples); // The stereoBuffer is ready now, let's put the finished audio into the requested buffers.
return !overallSilence;
}
and when I want to play a sound:
- (void)playWithPlayerAtIndex:(int) index {
SuperpoweredAdvancedAudioPlayer* player = players[index];
player->seek(0);
player->play(false);
}
But with this, I can only play hear one sound at a time.
Does anyone know the Superpowered API?
All your players are "playing" into "self->stereoBuffer". Check the bufferAdd parameter of the player's process method. You need to "add" the output to the contents of the buffer, if at least one player did output itself into it.

Audioqueue callback not being called

So, basically I want to play some audio files (mp3 and caf mostly). But the callback never gets called. Only when I call them to prime the queue.
Here's my data struct:
struct AQPlayerState
{
CAStreamBasicDescription mDataFormat;
AudioQueueRef mQueue;
AudioQueueBufferRef mBuffers[kBufferNum];
AudioFileID mAudioFile;
UInt32 bufferByteSize;
SInt64 mCurrentPacket;
UInt32 mNumPacketsToRead;
AudioStreamPacketDescription *mPacketDescs;
bool mIsRunning;
};
Here's my callback function:
static void HandleOutputBuffer (void *aqData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer)
{
NSLog(#"HandleOutput");
AQPlayerState *pAqData = (AQPlayerState *) aqData;
if (pAqData->mIsRunning == false) return;
UInt32 numBytesReadFromFile;
UInt32 numPackets = pAqData->mNumPacketsToRead;
AudioFileReadPackets (pAqData->mAudioFile,
false,
&numBytesReadFromFile,
pAqData->mPacketDescs,
pAqData->mCurrentPacket,
&numPackets,
inBuffer->mAudioData);
if (numPackets > 0) {
inBuffer->mAudioDataByteSize = numBytesReadFromFile;
AudioQueueEnqueueBuffer (pAqData->mQueue,
inBuffer,
(pAqData->mPacketDescs ? numPackets : 0),
pAqData->mPacketDescs);
pAqData->mCurrentPacket += numPackets;
} else {
// AudioQueueStop(pAqData->mQueue, false);
// AudioQueueDispose(pAqData->mQueue, true);
// AudioFileClose (pAqData->mAudioFile);
// free(pAqData->mPacketDescs);
// free(pAqData->mFloatBuffer);
pAqData->mIsRunning = false;
}
}
And here's my method:
- (void)playFile
{
AQPlayerState aqData;
// get the source file
NSString *p = [[NSBundle mainBundle] pathForResource:#"1_Female" ofType:#"mp3"];
NSURL *url2 = [NSURL fileURLWithPath:p];
CFURLRef srcFile = (__bridge CFURLRef)url2;
OSStatus result = AudioFileOpenURL(srcFile, 0x1/*fsRdPerm*/, 0/*inFileTypeHint*/, &aqData.mAudioFile);
CFRelease (srcFile);
CheckError(result, "Error opinning sound file");
UInt32 size = sizeof(aqData.mDataFormat);
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyDataFormat, &size, &aqData.mDataFormat),
"Error getting file's data format");
CheckError(AudioQueueNewOutput(&aqData.mDataFormat, HandleOutputBuffer, &aqData, CFRunLoopGetCurrent(), kCFRunLoopCommonModes, 0, &aqData.mQueue),
"Error AudioQueueNewOutPut");
// we need to calculate how many packets we read at a time and how big a buffer we need
// we base this on the size of the packets in the file and an approximate duration for each buffer
{
bool isFormatVBR = (aqData.mDataFormat.mBytesPerPacket == 0 || aqData.mDataFormat.mFramesPerPacket == 0);
// first check to see what the max size of a packet is - if it is bigger
// than our allocation default size, that needs to become larger
UInt32 maxPacketSize;
size = sizeof(maxPacketSize);
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyPacketSizeUpperBound, &size, &maxPacketSize),
"Error getting max packet size");
// adjust buffer size to represent about a second of audio based on this format
CalculateBytesForTime(aqData.mDataFormat, maxPacketSize, 1.0/*seconds*/, &aqData.bufferByteSize, &aqData.mNumPacketsToRead);
if (isFormatVBR) {
aqData.mPacketDescs = new AudioStreamPacketDescription [aqData.mNumPacketsToRead];
} else {
aqData.mPacketDescs = NULL; // we don't provide packet descriptions for constant bit rate formats (like linear PCM)
}
printf ("Buffer Byte Size: %d, Num Packets to Read: %d\n", (int)aqData.bufferByteSize, (int)aqData.mNumPacketsToRead);
}
// if the file has a magic cookie, we should get it and set it on the AQ
size = sizeof(UInt32);
result = AudioFileGetPropertyInfo(aqData.mAudioFile, kAudioFilePropertyMagicCookieData, &size, NULL);
if (!result && size) {
char* cookie = new char [size];
CheckError(AudioFileGetProperty(aqData.mAudioFile, kAudioFilePropertyMagicCookieData, &size, cookie),
"Error getting cookie from file");
CheckError(AudioQueueSetProperty(aqData.mQueue, kAudioQueueProperty_MagicCookie, cookie, size),
"Error setting cookie to file");
delete[] cookie;
}
aqData.mCurrentPacket = 0;
for (int i = 0; i < kBufferNum; ++i) {
CheckError(AudioQueueAllocateBuffer (aqData.mQueue,
aqData.bufferByteSize,
&aqData.mBuffers[i]),
"Error AudioQueueAllocateBuffer");
HandleOutputBuffer (&aqData,
aqData.mQueue,
aqData.mBuffers[i]);
}
// set queue's gain
Float32 gain = 1.0;
CheckError(AudioQueueSetParameter (aqData.mQueue,
kAudioQueueParam_Volume,
gain),
"Error AudioQueueSetParameter");
aqData.mIsRunning = true;
CheckError(AudioQueueStart(aqData.mQueue,
NULL),
"Error AudioQueueStart");
}
And the output when I press play:
Buffer Byte Size: 40310, Num Packets to Read: 38
HandleOutput start
HandleOutput start
HandleOutput start
I tryed replacing CFRunLoopGetCurrent() with CFRunLoopGetMain() and CFRunLoopCommonModes with CFRunLoopDefaultMode, but nothing.
Shouldn't the primed buffers start playing right away I start the queue?
When I start the queue, no callbacks are bang fired.
What am I doing wrong? Thanks for any ideas
What you are basically trying to do here is a basic example of audio playback using Audio Queues. Without looking at your code in detail to see what's missing (that could take a while) i'd rather recommend to you to follow the steps in this basic sample code that does exactly what you're doing (without the extras that aren't really relevant.. for example why are you trying to add audio gain?)
Somewhere else you were trying to play audio using audio units. Audio units are more complex than basic audio queue playback, and I wouldn't attempt them before being very comfortable with audio queues. But you can look at this example project for a basic example of audio queues.
In general when it comes to Core Audio programming in iOS, it's best you take your time with the basic examples and build your way up.. the problem with a lot of tutorials online is that they add extra stuff and often mix it with obj-c code.. when Core Audio is purely C code (ie the extra stuff won't add anything to the learning process). I strongly recommend you go over the book Learning Core Audio if you haven't already. All the sample code is available online, but you can also clone it from this repo for convenience. That's how I learned core audio. It takes time :)

Changing DCT coefficients

I decided to use libjpeg as the main library working with jpeg files.
I've read libjpg.txt file. And I was pleased that library allows DCT coefficients reading/writing in a convenient way. Since writing an own decoder will take a long time.
My work is related to the lossless embedding. Currently I need to read DCT coefficients from a file then modify some of them and write changed coefficients in the same file.
Well, I found jpeg_write_coefficients() function. And I naively thought that I could apply it to a decompression object (struct jpeg_decompress_struct). But it does not work and requires a compression object.
I can't believe that such the powerful library is not able to do this.
I think that most likely I'm missing something. Although I tried to be attentive.
Perhaps the writing coefficients can be done more sophisticated way.
But I don't know how to.
I will be very glad if you propose your ideas.
You can ue jpeg_write_coefficients to write your changed DCT.
The following information is avaliable in libjpeg.txt
To write the contents of a JPEG file as DCT coefficients, you must provide
the DCT coefficients stored in virtual block arrays. You can either pass
block arrays read from an input JPEG file by jpeg_read_coefficients(), or
allocate virtual arrays from the JPEG compression object and fill them
yourself. In either case, jpeg_write_coefficients() is substituted for
jpeg_start_compress() and jpeg_write_scanlines(). Thus the sequence is
* Create compression object
* Set all compression parameters as necessary
* Request virtual arrays if needed
* jpeg_write_coefficients()
* jpeg_finish_compress()
* Destroy or re-use compression object
jpeg_write_coefficients() is passed a pointer to an array of virtual block
array descriptors; the number of arrays is equal to cinfo.num_components.
The virtual arrays need only have been requested, not realized, before
jpeg_write_coefficients() is called. A side-effect of
jpeg_write_coefficients() is to realize any virtual arrays that have been
requested from the compression object's memory manager. Thus, when obtaining
the virtual arrays from the compression object, you should fill the arrays
after calling jpeg_write_coefficients(). The data is actually written out
when you call jpeg_finish_compress(); jpeg_write_coefficients() only writes
the file header.
When writing raw DCT coefficients, it is crucial that the JPEG quantization
tables and sampling factors match the way the data was encoded, or the
resulting file will be invalid. For transcoding from an existing JPEG file,
we recommend using jpeg_copy_critical_parameters(). This routine initializes
all the compression parameters to default values (like jpeg_set_defaults()),
then copies the critical information from a source decompression object.
The decompression object should have just been used to read the entire
JPEG input file --- that is, it should be awaiting jpeg_finish_decompress().
jpeg_write_coefficients() marks all tables stored in the compression object
as needing to be written to the output file (thus, it acts like
jpeg_start_compress(cinfo, TRUE)). This is for safety's sake, to avoid
emitting abbreviated JPEG files by accident. If you really want to emit an
abbreviated JPEG file, call jpeg_suppress_tables(), or set the tables'
individual sent_table flags, between calling jpeg_write_coefficients() and
jpeg_finish_compress().
So to change a single dct, you can use the following simple code:
To access any dct coeff, you need to change four index, cx, bx, by, bi.
In my code, I used blockptr_one[bi]++; to increase one dct Coeff
#include <stdio.h>
#include <jpeglib.h>
#include <stdlib.h>
#include <iostream>
#include <string>
int write_jpeg_file(std::string outname,jpeg_decompress_struct in_cinfo, jvirt_barray_ptr *coeffs_array ){
struct jpeg_compress_struct cinfo;
struct jpeg_error_mgr jerr;
FILE * infile;
if ((infile = fopen(outname.c_str(), "wb")) == NULL) {
fprintf(stderr, "can't open %s\n", outname.c_str());
return 0;
}
cinfo.err = jpeg_std_error(&jerr);
jpeg_create_compress(&cinfo);
jpeg_stdio_dest(&cinfo, infile);
j_compress_ptr cinfo_ptr = &cinfo;
jpeg_copy_critical_parameters((j_decompress_ptr)&in_cinfo,cinfo_ptr);
jpeg_write_coefficients(cinfo_ptr, coeffs_array);
jpeg_finish_compress( &cinfo );
jpeg_destroy_compress( &cinfo );
fclose( infile );
return 1;
}
int read_jpeg_file( std::string filename, std::string outname )
{
struct jpeg_decompress_struct cinfo;
struct jpeg_error_mgr jerr;
FILE * infile;
if ((infile = fopen(filename.c_str(), "rb")) == NULL) {
fprintf(stderr, "can't open %s\n", filename.c_str());
return 0;
}
cinfo.err = jpeg_std_error(&jerr);
jpeg_create_decompress(&cinfo);
jpeg_stdio_src(&cinfo, infile);
(void) jpeg_read_header(&cinfo, TRUE);
jvirt_barray_ptr *coeffs_array = jpeg_read_coefficients(&cinfo);
//change one dct:
int ci = 0; // between 0 and number of image component
int by = 0; // between 0 and compptr_one->height_in_blocks
int bx = 0; // between 0 and compptr_one->width_in_blocks
int bi = 0; // between 0 and 64 (8x8)
JBLOCKARRAY buffer_one;
JCOEFPTR blockptr_one;
jpeg_component_info* compptr_one;
compptr_one = cinfo.comp_info + ci;
buffer_one = (cinfo.mem->access_virt_barray)((j_common_ptr)&cinfo, coeffs_array[ci], by, (JDIMENSION)1, FALSE);
blockptr_one = buffer_one[0][bx];
blockptr_one[bi]++;
write_jpeg_file(outname, cinfo, coeffs_array);
jpeg_finish_decompress( &cinfo );
jpeg_destroy_decompress( &cinfo );
fclose( infile );
return 1;
}
int main()
{
std::string infilename = "you_image.jpg", outfilename = "out_image.jpg";
/* Try opening a jpeg*/
if( read_jpeg_file( infilename, outfilename ) > 0 )
{
std::cout << "It's Okay..." << std::endl;
}
else return -1;
return 0;
}
You should really take a look at transupp.h and sources for jpegtran that comes with the library.
Anyway, here is my dirty code with comments, assembled partially from jpegtran. It lets you manipulate DCT coefficients one by one.
#include "jpeglib.h" /* Common decls for cjpeg/djpeg applications */
#include "transupp.h" /* Support routines for jpegtran */
struct jpeg_decompress_struct srcinfo;
struct jpeg_compress_struct dstinfo;
struct jpeg_error_mgr jsrcerr, jdsterr;
static jpeg_transform_info transformoption; /* image transformation options */
transformoption.transform = JXFORM_NONE;
transformoption.trim = FALSE;
transformoption.force_grayscale = FALSE;
jvirt_barray_ptr * src_coef_arrays;
jvirt_barray_ptr * dst_coef_arrays;
/* Initialize the JPEG decompression object with default error handling. */
srcinfo.err = jpeg_std_error(&jsrcerr);
jpeg_create_decompress(&srcinfo);
/* Initialize the JPEG compression object with default error handling. */
dstinfo.err = jpeg_std_error(&jdsterr);
jpeg_create_compress(&dstinfo);
FILE *fp;
if((fp = fopen(filePath], "rb")) == NULL) {
//Throw an error
} else {
//Continue
}
/* Specify data source for decompression */
jpeg_stdio_src(&srcinfo, fp);
/* Enable saving of extra markers that we want to copy */
jcopy_markers_setup(&srcinfo, JCOPYOPT_ALL);
/* Read file header */
(void) jpeg_read_header(&srcinfo, TRUE);
jtransform_request_workspace(&srcinfo, &transformoption);
src_coef_arrays = jpeg_read_coefficients(&srcinfo);
jpeg_copy_critical_parameters(&srcinfo, &dstinfo);
/* Do your DCT shenanigans here on src_coef_arrays like this (I've moved it into a separate function): */
moveDCTAround(&srcinfo, &dstinfo, 0, src_coef_arrays);
/* ..when done with DCT, do this: */
dst_coef_arrays = jtransform_adjust_parameters(&srcinfo, &dstinfo, src_coef_arrays, &transformoption);
fclose(fp);
//And write everything back
fp = fopen(filePath, "wb");
/* Specify data destination for compression */
jpeg_stdio_dest(&dstinfo, fp);
/* Start compressor (note no image data is actually written here) */
jpeg_write_coefficients(&dstinfo, dst_coef_arrays);
/* Copy to the output file any extra markers that we want to preserve */
jcopy_markers_execute(&srcinfo, &dstinfo, JCOPYOPT_ALL);
jpeg_finish_compress(&dstinfo);
jpeg_destroy_compress(&dstinfo);
(void) jpeg_finish_decompress(&srcinfo);
jpeg_destroy_decompress(&srcinfo);
fclose(fp);
And the function itself:
void moveDCTAround (j_decompress_ptr srcinfo, j_compress_ptr dstinfo, JDIMENSION x_crop_offset, jvirt_barray_ptr *src_coef_arrays)
{
size_t block_row_size;
JBLOCKARRAY coef_buffers[MAX_COMPONENTS];
JBLOCKARRAY row_ptrs[MAX_COMPONENTS];
//Allocate DCT array buffers
for (JDIMENSION compnum=0; compnum<srcinfo->num_components; compnum++)
{
coef_buffers[compnum] = (dstinfo->mem->alloc_barray)((j_common_ptr) dstinfo, JPOOL_IMAGE, srcinfo->comp_info[compnum].width_in_blocks,
srcinfo->comp_info[compnum].height_in_blocks);
}
//For each component,
for (JDIMENSION compnum=0; compnum<srcinfo->num_components; compnum++)
{
block_row_size = (size_t) sizeof(JCOEF)*DCTSIZE2*srcinfo->comp_info[compnum].width_in_blocks;
//...iterate over rows,
for (JDIMENSION rownum=0; rownum<srcinfo->comp_info[compnum].height_in_blocks; rownum++)
{
row_ptrs[compnum] = ((dstinfo)->mem->access_virt_barray)((j_common_ptr) &dstinfo, src_coef_arrays[compnum], rownum, (JDIMENSION) 1, FALSE);
//...and for each block in a row,
for (JDIMENSION blocknum=0; blocknum<srcinfo->comp_info[compnum].width_in_blocks; blocknum++)
//...iterate over DCT coefficients
for (JDIMENSION i=0; i<DCTSIZE2; i++)
{
//Manipulate your DCT coefficients here. For instance, the code here inverts the image.
coef_buffers[compnum][rownum][blocknum][i] = -row_ptrs[compnum][0][blocknum][i];
}
}
}
//Save the changes
//For each component,
for (JDIMENSION compnum=0; compnum<srcinfo->num_components; compnum++)
{
block_row_size = (size_t) sizeof(JCOEF)*DCTSIZE2 * srcinfo->comp_info[compnum].width_in_blocks;
//...iterate over rows
for (JDIMENSION rownum=0; rownum < srcinfo->comp_info[compnum].height_in_blocks; rownum++)
{
//Copy the whole rows
row_ptrs[compnum] = (dstinfo->mem->access_virt_barray)((j_common_ptr) dstinfo, src_coef_arrays[compnum], rownum, (JDIMENSION) 1, TRUE);
memcpy(row_ptrs[compnum][0][0], coef_buffers[compnum][rownum][0], block_row_size);
}
}

Open CV Image editing library for windows 8 and windows phone 8

Is there any support of OpenCV graphics library is available for Windows Phone 8 and Windows 8. I made a search on Google but didn't find any resource related with OpenCV to connect with Windows Phone 8 / Windows 8. If any of you know more about this please help me, and provide some link to reach the library.
This is the latest information what I get from OpenCV team.
OpenCV development team is working on port for Windows RT. Here is current development branch for WinRT(https://github.com/asmorkalov/opencv/tree/winrt). You can build it for ARM using Visual Studio Express for Windows 8 and Platform SDK.
Open Visual Studio development console.
Setup environment for cross compilation by command "C:\Program Files(x86)\Microsoft
Visual Studio 11.0\VC\bin\x86_arm\vcvarsx86_arm.bat"
cd <opencv_source_dir>/platforms/winrt/
run scripts/cmake_winrt.cmd
run ninja
Alternatively you can use nmake instead ninja. You need to edit cmake_winrt.cmd and change project generator fro -GNinja to -G "NMake Makefiles". Only algorithmic part of the library is supported now, no tbb, no UI, no video IO.
Please check the below given URL from more details.
http://answers.opencv.org/question/9847/opencv-for-windows-8-tablet/?answer=9851#post-id-9851
By windows-8, I guess you mean winRT ? AFAIK, there is no official port to winRT. You need to compile it by yourself as a Win8 Store DLL for instance, so that you can reference it from a Win8 Store Application.
Just start by opencv-core, then add the lib you need, one by one, because all the components will not be able to compile (for instance, opencv-highgui is highly dependant on Windows API which is not fully compatible with Win8 Store Apps).
You'll also need to code by yourself some Win32 methods used by OpenCV and not accessible from Win8 App like GetSystemInfo(), GetTempPathA(), GetTempFileNameA() and all methods related to thread local storage (TLS).
I've been able to use a small subset of OpenCV in WinRT by compiling opencv_core, opencv_imgproc and zlib, as 3 seperate static libs. I've added one another, called opencv_winrt, that contains only the two following files:
opencv_winrt.h
#pragma once
#include "combaseapi.h"
void WINAPI GetSystemInfo(
_Out_ LPSYSTEM_INFO lpSystemInfo
);
DWORD WINAPI GetTempPathA(
_In_ DWORD nBufferLength,
_Out_ char* lpBuffer
);
UINT WINAPI GetTempFileNameA(
_In_ const char* lpPathName,
_In_ const char* lpPrefixString,
_In_ UINT uUnique,
_Out_ char* lpTempFileName
);
DWORD WINAPI TlsAlloc();
BOOL WINAPI TlsFree(
_In_ DWORD dwTlsIndex
);
LPVOID WINAPI TlsGetValue(
_In_ DWORD dwTlsIndex
);
BOOL WINAPI TlsSetValue(
_In_ DWORD dwTlsIndex,
_In_opt_ LPVOID lpTlsValue
);
void WINAPI TlsShutdown();
# define TLS_OUT_OF_INDEXES ((DWORD)0xFFFFFFFF)
opencv_winrt.cpp
#include "opencv_winrt.h"
#include <vector>
#include <set>
#include <mutex>
#include "assert.h"
void WINAPI GetSystemInfo(LPSYSTEM_INFO lpSystemInfo)
{
GetNativeSystemInfo(lpSystemInfo);
}
DWORD WINAPI GetTempPathA(DWORD nBufferLength, char* lpBuffer)
{
return 0;
}
UINT WINAPI GetTempFileNameA(const char* lpPathName, const char* lpPrefixString, UINT uUnique, char* lpTempFileName)
{
return 0;
}
// Thread local storage.
typedef std::vector<void*> ThreadLocalData;
static __declspec(thread) ThreadLocalData* currentThreadData = nullptr;
static std::set<ThreadLocalData*> allThreadData;
static DWORD nextTlsIndex = 0;
static std::vector<DWORD> freeTlsIndices;
static std::mutex tlsAllocationLock;
DWORD WINAPI TlsAlloc()
{
std::lock_guard<std::mutex> lock(tlsAllocationLock);
// Can we reuse a previously freed TLS slot?
if (!freeTlsIndices.empty())
{
DWORD result = freeTlsIndices.back();
freeTlsIndices.pop_back();
return result;
}
// Allocate a new TLS slot.
return nextTlsIndex++;
}
_Use_decl_annotations_ BOOL WINAPI TlsFree(DWORD dwTlsIndex)
{
std::lock_guard<std::mutex> lock(tlsAllocationLock);
assert(dwTlsIndex < nextTlsIndex);
assert(find(freeTlsIndices.begin(), freeTlsIndices.end(), dwTlsIndex) == freeTlsIndices.end());
// Store this slot for reuse by TlsAlloc.
try
{
freeTlsIndices.push_back(dwTlsIndex);
}
catch (...)
{
return false;
}
// Zero the value for all threads that might be using this now freed slot.
for each (auto threadData in allThreadData)
{
if (threadData->size() > dwTlsIndex)
{
threadData->at(dwTlsIndex) = nullptr;
}
}
return true;
}
_Use_decl_annotations_ LPVOID WINAPI TlsGetValue(DWORD dwTlsIndex)
{
ThreadLocalData* threadData = currentThreadData;
if (threadData && threadData->size() > dwTlsIndex)
{
// Return the value of an allocated TLS slot.
return threadData->at(dwTlsIndex);
}
else
{
// Default value for unallocated slots.
return nullptr;
}
}
_Use_decl_annotations_ BOOL WINAPI TlsSetValue(DWORD dwTlsIndex, LPVOID lpTlsValue)
{
ThreadLocalData* threadData = currentThreadData;
if (!threadData)
{
// First time allocation of TLS data for this thread.
try
{
threadData = new ThreadLocalData(dwTlsIndex + 1, nullptr);
std::lock_guard<std::mutex> lock(tlsAllocationLock);
allThreadData.insert(threadData);
currentThreadData = threadData;
}
catch (...)
{
if (threadData)
delete threadData;
return false;
}
}
else if (threadData->size() <= dwTlsIndex)
{
// This thread already has a TLS data block, but it must be expanded to fit the specified slot.
try
{
std::lock_guard<std::mutex> lock(tlsAllocationLock);
threadData->resize(dwTlsIndex + 1, nullptr);
}
catch (...)
{
return false;
}
}
// Store the new value for this slot.
threadData->at(dwTlsIndex) = lpTlsValue;
return true;
}
// Called at thread exit to clean up TLS allocations.
void WINAPI TlsShutdown()
{
ThreadLocalData* threadData = currentThreadData;
if (threadData)
{
{
std::lock_guard<std::mutex> lock(tlsAllocationLock);
allThreadData.erase(threadData);
}
currentThreadData = nullptr;
delete threadData;
}
}
And I modify the file cvconfig.h: I've commented out every #define, except PACKAGE* and VERSION, and I added #include "opencv_winrt.h" at the end.
Just a hint - there is a C# wrapper for OpenCV called EmguCV (http://www.emgu.com/wiki/index.php/Main_Page), by looking at the forum posts I see that there is some activity towards using it on Windows 8 but it's hard to tell if it's now working since the posts claiming issues are quite old. I'd suggest you just give it a try and see if this C# wrapper is able to run on Windows Phone 8, I think it should definitely run on Windows 8.

Can I use AVCaptureSession to encode an AAC stream to memory?

I'm writing an iOS app that streams video and audio over the network.
I am using AVCaptureSession to grab raw video frames using AVCaptureVideoDataOutput and encode them in software using x264. This works great.
I wanted to do the same for audio, only that I don't need that much control on the audio side so I wanted to use the built in hardware encoder to produce an AAC stream. This meant using Audio Converter from the Audio Toolbox layer. In order to do so I put in a handler for AVCaptudeAudioDataOutput's audio frames:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
// get the audio samples into a common buffer _pcmBuffer
CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
CMBlockBufferGetDataPointer(blockBuffer, 0, NULL, &_pcmBufferSize, &_pcmBuffer);
// use AudioConverter to
UInt32 ouputPacketsCount = 1;
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0].mNumberChannels = 1;
bufferList.mBuffers[0].mDataByteSize = sizeof(_aacBuffer);
bufferList.mBuffers[0].mData = _aacBuffer;
OSStatus st = AudioConverterFillComplexBuffer(_converter, converter_callback, (__bridge void *) self, &ouputPacketsCount, &bufferList, NULL);
if (0 == st) {
// ... send bufferList.mBuffers[0].mDataByteSize bytes from _aacBuffer...
}
}
In this case the callback function for the audio converter is pretty simple (assuming packet sizes and counts are setup properly):
- (void) putPcmSamplesInBufferList:(AudioBufferList *)bufferList withCount:(UInt32 *)count
{
bufferList->mBuffers[0].mData = _pcmBuffer;
bufferList->mBuffers[0].mDataByteSize = _pcmBufferSize;
}
And the setup for the audio converter looks like this:
{
// ...
AudioStreamBasicDescription pcmASBD = {0};
pcmASBD.mSampleRate = ((AVAudioSession *) [AVAudioSession sharedInstance]).currentHardwareSampleRate;
pcmASBD.mFormatID = kAudioFormatLinearPCM;
pcmASBD.mFormatFlags = kAudioFormatFlagsCanonical;
pcmASBD.mChannelsPerFrame = 1;
pcmASBD.mBytesPerFrame = sizeof(AudioSampleType);
pcmASBD.mFramesPerPacket = 1;
pcmASBD.mBytesPerPacket = pcmASBD.mBytesPerFrame * pcmASBD.mFramesPerPacket;
pcmASBD.mBitsPerChannel = 8 * pcmASBD.mBytesPerFrame;
AudioStreamBasicDescription aacASBD = {0};
aacASBD.mFormatID = kAudioFormatMPEG4AAC;
aacASBD.mSampleRate = pcmASBD.mSampleRate;
aacASBD.mChannelsPerFrame = pcmASBD.mChannelsPerFrame;
size = sizeof(aacASBD);
AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &size, &aacASBD);
AudioConverterNew(&pcmASBD, &aacASBD, &_converter);
// ...
}
This seems pretty straight forward only the IT DOES NOT WORK. Once the AVCaptureSession is running, the audio converter (specifically AudioConverterFillComplexBuffer) returns an 'hwiu' (hardware in use) error. Conversion works fine if the session is stopped but then I can't capture anything...
I was wondering if there was a way to get an AAC stream out of AVCaptureSession. The options I'm considering are:
Somehow using AVAssetWriterInput to encode audio samples into AAC and then get the encoded packets somehow (not through AVAssetWriter, which would only write to a file).
Reorganizing my app so that it uses AVCaptureSession only on the video side and uses Audio Queues on the audio side. This will make flow control (starting and stopping recording, responding to interruptions) more complicated and I'm afraid that it might cause synching problems between the audio and video. Also, it just doesn't seem like a good design.
Does anyone know if getting the AAC out of AVCaptureSession is possible? Do I have to use Audio Queues here? Could this get me into synching or control problems?
I ended up asking Apple for advice (it turns out you can do that if you have a paid developer account).
It seems that AVCaptureSession grabs a hold of the AAC hardware encoder but only lets you use it to write directly to file.
You can use the software encoder but you have to ask for it specifically instead of using AudioConverterNew:
AudioClassDescription *description = [self
getAudioClassDescriptionWithType:kAudioFormatMPEG4AAC
fromManufacturer:kAppleSoftwareAudioCodecManufacturer];
if (!description) {
return false;
}
// see the question as for setting up pcmASBD and arc ASBD
OSStatus st = AudioConverterNewSpecific(&pcmASBD, &aacASBD, 1, description, &_converter);
if (st) {
NSLog(#"error creating audio converter: %s", OSSTATUS(st));
return false;
}
with
- (AudioClassDescription *)getAudioClassDescriptionWithType:(UInt32)type
fromManufacturer:(UInt32)manufacturer
{
static AudioClassDescription desc;
UInt32 encoderSpecifier = type;
OSStatus st;
UInt32 size;
st = AudioFormatGetPropertyInfo(kAudioFormatProperty_Encoders,
sizeof(encoderSpecifier),
&encoderSpecifier,
&size);
if (st) {
NSLog(#"error getting audio format propery info: %s", OSSTATUS(st));
return nil;
}
unsigned int count = size / sizeof(AudioClassDescription);
AudioClassDescription descriptions[count];
st = AudioFormatGetProperty(kAudioFormatProperty_Encoders,
sizeof(encoderSpecifier),
&encoderSpecifier,
&size,
descriptions);
if (st) {
NSLog(#"error getting audio format propery: %s", OSSTATUS(st));
return nil;
}
for (unsigned int i = 0; i < count; i++) {
if ((type == descriptions[i].mSubType) &&
(manufacturer == descriptions[i].mManufacturer)) {
memcpy(&desc, &(descriptions[i]), sizeof(desc));
return &desc;
}
}
return nil;
}
The software encoder will take up CPU resources, of course, but will get the job done.

Resources