I have an array of Int16[19200]
I want to turn it into an Image[160,120,1]
What is the fastest way of doing this?
I need to do it at 120fps, so it needs to be really efficient.
Thanks
SW
Found it:
GCHandle handle = GCHandle.Alloc(dataArray, GCHandleType.Pinned);
IntPtr imageHeaderForBytes = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(MIplImage)));
CvInvoke.cvInitImageHeader(
imageHeaderForBytes,
new Size(160, 120),
Emgu.CV.CvEnum.IPL_DEPTH.IPL_DEPTH_16S, 1, 0, 4);
Marshal.WriteIntPtr(
imageHeaderForBytes,
(int)Marshal.OffsetOf(typeof(MIplImage), "imageData"),
handle.AddrOfPinnedObject());
CvInvoke.cvCopy(imageHeaderForBytes, EMGUImage.Ptr, IntPtr.Zero);
Marshal.FreeHGlobal(imageHeaderForBytes);
handle.Free();
Related
Suppose we have the following color:
const Scalar TRANSPARENT2 = Scalar(255, 0, 255,0);
which is magenta but fully transparent: alpha = 0 (to be fully opaque is 255).
Now I made the following test based on:
http://blogs.msdn.com/b/lucian/archive/2015/12/04/opencv-first-version-up-on-nuget.aspx
WriteableBitmap^ Grabcut::TestTransparent()
{
Mat res(400,400, CV_8UC4);
res.setTo(TRANSPARENT2);
WriteableBitmap^ wbmp = ref new WriteableBitmap(res.cols, res.rows);
IBuffer^ buffer = wbmp->PixelBuffer;
unsigned char* dstPixels;
ComPtr<IBufferByteAccess> pBufferByteAccess;
ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
pBuffer.As(&pBufferByteAccess);
pBufferByteAccess->Buffer(&dstPixels);
memcpy(dstPixels, res.data, res.step.buf[1] * res.cols * res.rows);
return wbmp;
}
The issue I have is that the image created is not fully transparent, it has a bit of alpha:
I understand there is a fila in the memcpy data, but I am not really sure about how to solve this. any idea to get it to alpha 0?
more details
To see I saving the image could then read and test if it works, I saw that the imwrite contains an snippet about transparency like in the image, but well imwrite is not implemented yet. But the transparency method is not working neither.
Any light with this snippet?
Thanks.
Finally I did the conversion in the C# code, first avoid calling CreateAlphaMat.
Then what I did is use a BitmapEncoder to convert data:
WriteableBitmap wb = new WriteableBitmap(bitmap.PixelWidth, bitmap.PixelHeight);
using (IRandomAccessStream stream = new InMemoryRandomAccessStream())
{
BitmapEncoder encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, stream);
Stream pixelStream = bitmap.PixelBuffer.AsStream();
byte[] pixels = new byte[pixelStream.Length];
await pixelStream.ReadAsync(pixels, 0, pixels.Length);
encoder.SetPixelData(BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied,
(uint)bitmap.PixelWidth, (uint)bitmap.PixelHeight, 96.0, 96.0, pixels);
await encoder.FlushAsync();
wb.SetSource(stream);
}
this.MainImage.Source = wb;
where bitmap is the WriteableBitmap from the OpenCV result. And now the image is fully transparent.
NOTE: Do not use MemoryStream and then .AsRandomAccessStream because it won't FlushAsync
I'm trying to fill a 1D texture with values manually and pass that texture to a compute shader (these are 2 pixels that I want to set via code, they don't represent any image).
Due to the current small amount of Metal examples, all examples I could find deal with 2D textures that load the texture by converting a loaded UIImage to raw bytes data, but creating a dummy UIImage felt like a hack for me.
This is the "naive" way I started with -
...
var manualTextureData: [Float] = [ 1.0, 0.0, 0.0, 1.0,
0.0, 0.0, 1.0, 1.0 ];
let region: MTLRegion = MTLRegionMake1D(0, textureDescriptor.width);
myTexture.replaceRegion(region, mipmapLevel: 0, withBytes: &manualTextureData, bytesPerRow: 0);
but Metal doesn't recognize those values in the shader (it gets an empty texture, except for the first value).
I quickly realized that the Float array probably has to be converted into a bytes array (e.g UInt8), but couldn't find a way to convert from [Float] to [UInt8] either.
Another possible option I consider is using a CVPixelBuffer object, but that also felt like a workaround to the problem.
So whats the right way to tackle that?
Thanks in advance.
Please note I'm not familiar with Objective-C, hence I'm not sure whether using CVPixelBuffer / UIImage is exaggerated for something which should be straight-forward.
Please forgive the terse reply, but you may find it useful to take a look at my experiments with Swift and Metal. I've created a particle system in Swift which is passed to a Metal compute shader as a one dimensional array of Particle structs. By using posix_memalign, I'm able to eliminate the bottleneck caused by passing the array between Metal and Swift.
I've blogged extensively about this: http://flexmonkey.blogspot.co.uk/search/label/Metal
I hope this helps.
Simon
I don't see any reason for you to pass data using 1D texture. Instead I would go with just passing a buffer. Like this:
var dataBuffer:MTLBuffer? = device.newBufferWithBytes(&manualTextureData, length: sizeOf(manualTextureData), options: MTLResourceOptions.OptionCPUCacheModeDefault)
Then you hook it to your renderCommandEncoder like this:
renderCommandEncoder.setFragmentBuffer(dataBuffer, offset: 0, atIndex: 1)//Note that if you want this buffer to be passed to you vertex shader you should use setVertexBuffer
Then in your shader, you should add parameter like this const device float* bufferPassed [[ buffer(1) ]]
And then use it like this, inside your shader implementation:
float firstFloat = bufferPassed[0];
This will get the job done.
Not really answering your question, but you could just define an array in your metal shader instead of passing the values as a texture.
Something like:
constant float manualData[8] = { 1.0, 0.0, 0.0, 1.0,
0.0, 0.0, 1.0, 1.0 };
vertex float4 world_vertex(unsigned int vid[[vertex_id]], ...) {
int manualIndex = vid % 8;
float manualValue = manualData[manualIndex];
// something deep and meaningful here...
return float4(manualValue);
}
If you want a float texture bytesPerRow should be 4 for times the width, because a float has a size of 4 bytes. Metal copies the memory and dont care about the values. That is your task ;-)
Something like:
myTexture.replaceRegion(region, mipmapLevel: 0, withBytes: &manualTextureData, bytesPerRow: manualTextureData.count * sizeof(Float));
is it possible to get back original image from image ROI? for example say we have
cv::Mat image = imread("image.jpg", 0);
cv::Mat imageROI = image(0, 0, 100, 100);
myFunction(imageROI);
and in myFunction I want to work with original image. is there any way to convert imageROI to original image when we don't access the original image?
I don't know if I understood the question exactly like you think, but if you ask if let's say we have header
void myFunc(cv::Mat &m);
// .... later on
cv::Mat image = imread("image.jpg", 0);
cv::Mat imageROI = image(0, 0, 100, 100);
myFunction(imageROI);
// .... later on myFuncDefinition
void myFunc(cv::Mat &m) {
// some code
// here you would like to have an original image, right?
}
So the answer for that is no and the proof is by simplicity: why want you to design opencv api in such way to make it possible store unnecessary data? If you do
cv::Mat imageROI = image(0, 0, 100, 100);
by purpose you would like to forgot about entire image and you are particulary interested in some ROI. Mat container is designed in such way to copy only matrix 'headers' and not matrix content. So if you do cv::Mat imageROI = image(0, 0, 100, 100) perhaps the matrix content (ie image data) might be stored somewhere in memory (because roi is the part of it, so by optimalization purposes it might no be deleted even is original image variable went out of scope), but your matrix header changed. Namely, from pointing to (0, 0, imageWisth, imageHeight) to (0, 0, 100, 100) and there's no way to bring it back just using variable m.
Why don't pass additional parameter as a reference?
Just incase anybody looks at this question you can actually do this
cv::Mat mat = ...
cv::Size size;
cv::Point offset;
// find original image size, and get offset of roi
mat.locateROI(size, offset);
// put image back to original size;
mat.adjustROI(offset.y, size.height - mat.rows, offset.x, size.width- mat.cols);
I am trying to display a spectrum analyser for iOS and am stuck after two weeks. I have read pretty much every post about FFT and the Accelerate Frameworks on here and have downloaded the aurioTouch2 example from Apple.
I think I understand the mechanism of FFT (did it in Uni 20 years ago) and am a fairly experienced iOS programmer but I have hit a wall.
I am using AudioUnit to play mp3, m4a, and wav files and have that working beautifully. I have attached a Render Callback to the AUGraph and I can plot Waveforms to the music. The waveform goes with the music nicely.
When I take the data from the Render Callback which is in Float form in the range 0 .. 1 and attempt to pass that through the FFT code (either my own or aurioTouch2's FFTBufferManager.mm) I get something thats not completely wrong, but is not correct either. or instance this is a 440Hz sine wave:
That peak value is -6.1306, followed by -24. -31., -35. and those values towards the end are around -63.
Animated gif for "Black Betty":
Animated gif for "Black Betty
The format I receive from the Render callback:
AudioStreamBasicDescription outputFileFormat;
outputFileFormat.mSampleRate = 44100;
outputFileFormat.mFormatID = kAudioFormatLinearPCM;
outputFileFormat.mFormatFlags = kAudioFormatFlagsNativeFloatPacked | kAudioFormatFlagIsNonInterleaved;
outputFileFormat.mBitsPerChannel = 32;
outputFileFormat.mChannelsPerFrame = 2;
outputFileFormat.mFramesPerPacket = 1;
outputFileFormat.mBytesPerFrame = outputFileFormat.mBitsPerChannel / 8;
outputFileFormat.mBytesPerPacket = outputFileFormat.mBytesPerFrame;
In looking at the aurioTouch2 example it looks like they are receiving their data in a signed int format but then running an AudioConverter to convert it to Float. Their format is hard to decipher but is using a macro:
drawFormat.SetAUCanonical(2, false);
drawFormat.mSampleRate = 44100;
XThrowIfError(AudioConverterNew(&thruFormat, &drawFormat, &audioConverter), "couldn't setup AudioConverter");
In their render callback they are copying the data out of the AudioBufferList into mAudioBuffer (Float32*) and passing it to the CalculateFFT method which calls vDSP_ctoz
//Generate a split complex vector from the real data
vDSP_ctoz((COMPLEX *)mAudioBuffer, 2, &mDspSplitComplex, 1, mFFTLength);
I think this is where my problem is. What format does vDSP_ctoz expect? It is cast as a (COMPLEX*) but I cannot find anywhere in the aurioTouch2 code which puts the mAudioBuffer data into the (COMPLEX*) format. So is must be coming from the Render Callback in this format?
typedef struct DSPComplex {
float real;
float imag;
} DSPComplex;
typedef DSPComplex COMPLEX;
If I don't have the format correct at this point (or understand the format) then there is no point in debugging the rest of it.
Any help would be greatly appreciated.
Code from AurioTouch2 that I am using:
Boolean FFTBufferManager::ComputeFFTFloat(Float32 *outFFTData)
{
if (HasNewAudioData())
{
// Added after Hotpaw2 comment.
UInt32 windowSize = mFFTLength;
Float32 *window = (float *) malloc(windowSize * sizeof(float));
memset(window, 0, windowSize * sizeof(float));
vDSP_hann_window(window, windowSize, 0);
vDSP_vmul( mAudioBuffer, 1, window, 1, mAudioBuffer, 1, mFFTLength);
// Added after Hotpaw2 comment.
DSPComplex *audioBufferComplex = new DSPComplex[mFFTLength];
for (int i=0; i < mFFTLength; i++)
{
audioBufferComplex[i].real = mAudioBuffer[i];
audioBufferComplex[i].imag = 0.0f;
}
//Generate a split complex vector from the real data
vDSP_ctoz((COMPLEX *)audioBufferComplex, 2, &mDspSplitComplex, 1, mFFTLength);
//Take the fft and scale appropriately
vDSP_fft_zrip(mSpectrumAnalysis, &mDspSplitComplex, 1, mLog2N, kFFTDirection_Forward);
vDSP_vsmul(mDspSplitComplex.realp, 1, &mFFTNormFactor, mDspSplitComplex.realp, 1, mFFTLength);
vDSP_vsmul(mDspSplitComplex.imagp, 1, &mFFTNormFactor, mDspSplitComplex.imagp, 1, mFFTLength);
//Zero out the nyquist value
mDspSplitComplex.imagp[0] = 0.0;
//Convert the fft data to dB
vDSP_zvmags(&mDspSplitComplex, 1, outFFTData, 1, mFFTLength);
//In order to avoid taking log10 of zero, an adjusting factor is added in to make the minimum value equal -128dB
vDSP_vsadd( outFFTData, 1, &mAdjust0DB, outFFTData, 1, mFFTLength);
Float32 one = 1;
vDSP_vdbcon(outFFTData, 1, &one, outFFTData, 1, mFFTLength, 0);
free( audioBufferComplex);
free( window);
OSAtomicDecrement32Barrier(&mHasAudioData);
OSAtomicIncrement32Barrier(&mNeedsAudioData);
mAudioBufferCurrentIndex = 0;
return true;
}
else if (mNeedsAudioData == 0)
OSAtomicIncrement32Barrier(&mNeedsAudioData);
return false;
}
After reading the answer below I tried adding this to the top of the method:
DSPComplex *audioBufferComplex = new DSPComplex[mFFTLength];
for (int i=0; i < mFFTLength; i++)
{
audioBufferComplex[i].real = mAudioBuffer[i];
audioBufferComplex[i].imag = 0.0f;
}
//Generate a split complex vector from the real data
vDSP_ctoz((COMPLEX *)audioBufferComplex, 2, &mDspSplitComplex, 1, mFFTLength);
And the result I got was this:
I am now rendering the 5 last results, they are the faded ones behind.
After adding hann window:
Now looks a lot better after applying the hann window (Thanks hotpaw2). Not worried about the mirror image.
My main problem now is using a real song it doesn't look like other Spectrum Analysers. Everything is always pushed high on the left no matter what music i push through it. After applying the window it seems to go to the beat a lot better though.
The AU render callback only returns the real part of the complex input required. To use a complex FFT, you need to fill an equal number of imaginary components with zeros yourself, and copy over the elements of the real part, if needed.
I have an FFMPEG AVFrame in YUVJ420P and I want to convert it to a CVPixelBufferRef with CVPixelBufferCreateWithBytes. The reason I want to do this is to use AVFoundation to show/encode the frames.
I selected kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange and tried converting it since the AVFrame has the data in three planes
Y480 Cb240 Cr240. And according to what I've researched this matches the selected kCVPixelFormatType. By being biplanar I need to convert it into a buffer that contains Y480 and CbCr480 Interleaved.
I tried to create a buffer with 2 planes:
frame->data[0] on the first plane,
frame->data[1] and frame->data[2] interleaved on the second plane.
However, I'm getting return error -6661 (invalid a) from CVPixelBufferCreateWithBytes:
"Invalid function parameter. For example, out of range or the wrong type."
I don't have expertise on image processing at all, so any pointers to documentation that can get me started in the right approach to this problem are appreciated. My C skills aren't top of the line either so maybe I'm making a basic mistake here.
uint8_t **buffer = malloc(2*sizeof(int *));
buffer[0] = frame->data[0];
buffer[1] = malloc(frame->linesize[0]*sizeof(int));
for(int i = 0; i<frame->linesize[0]; i++){
if(i%2){
buffer[1][i]=frame->data[1][i/2];
}else{
buffer[1][i]=frame->data[2][i/2];
}
}
int ret = CVPixelBufferCreateWithBytes(NULL, frame->width, frame->height, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, buffer, frame->linesize[0], NULL, 0, NULL, cvPixelBufferSample)
The frame is the AVFrame with the rawData from FFMPEG Decoding.
My C skills aren't top of the line either so maybe im making a basic mistake here.
You're making several:
You should be using CVPixelBufferCreateWithPlanarBytes(). I do not know if CVPixelBufferCreateWithBytes() can be used to create a planar video frame; if so, it will require a pointer to a "plane descriptor block" (I can't seem to find the struct in the docs).
frame->linesize[0] is the bytes per row, not the size of the whole image. The docs are unclear, but the usage is fairly unambiguous.
frame->linesize[0] refers to the Y plane; you care about the UV planes.
Where is sizeof(int) from?
You're passing in cvPixelBufferSample; you might mean &cvPixelBufferSample.
You're not passing in a release callback. The documentation does not say that you can pass NULL.
Try something like this:
size_t srcPlaneSize = frame->linesize[1]*frame->height;
size_t dstPlaneSize = srcPlaneSize *2;
uint8_t *dstPlane = malloc(dstPlaneSize);
void *planeBaseAddress[2] = { frame->data[0], dstPlane };
// This loop is very naive and assumes that the line sizes are the same.
// It also copies padding bytes.
assert(frame->linesize[1] == frame->linesize[2]);
for(size_t i = 0; i<srcPlaneSize; i++){
// These might be the wrong way round.
dstPlane[2*i ]=frame->data[2][i];
dstPlane[2*i+1]=frame->data[1][i];
}
// This assumes the width and height are even (it's 420 after all).
assert(!frame->width%2 && !frame->height%2);
size_t planeWidth[2] = {frame->width, frame->width/2};
size_t planeHeight[2] = {frame->height, frame->height/2};
// I'm not sure where you'd get this.
size_t planeBytesPerRow[2] = {frame->linesize[0], frame->linesize[1]*2};
int ret = CVPixelBufferCreateWithPlanarBytes(
NULL,
frame->width,
frame->height,
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
NULL,
0,
2,
planeBaseAddress,
planeWidth,
planeHeight,
planeBytesPerRow,
YOUR_RELEASE_CALLBACK,
YOUR_RELEASE_CALLBACK_CONTEXT,
NULL,
&cvPixelBufferSample);
Memory management is left as an exercise to the reader, but for test code you might get away with passing in NULL instead of a release callback.