I have a decoder which produces NV12 yuv frames. What I would like to do is to copy that frame to ID3D11Texture2D texture with single memcpy. Because D3D11_MAPPED_SUBRESOURCE has a different RowPitch than my NV12 frame has I have to copy whole yuv row by row which is pretty expensive. Is there a better way to do this?
I've tried to find any information on this topic but so far without any luck.
// Copy from CPU access texture to bitmap buffer
D3D11_MAPPED_SUBRESOURCE resource;
UINT subresource = D3D11CalcSubresource(0, 0, 0);
OutMgr.m_DeviceContext->Map(OutMgr.m_texture, subresource, D3D11_MAP_WRITE_DISCARD, 0, &resource);
BYTE* dptr = reinterpret_cast<BYTE*>(resource.pData);
for (int i = 0; i < nv12Frame->height; i++)
{
memcpy(dptr + resource.RowPitch * i, nv12Frame->Y + nv12Frame->pitch * i, nv12Frame->pitch);
}
for (int i = 0; i < nv12Frame->height / 2; i++)
{
memcpy(dptr + resource.RowPitch *(nv12Frame->height + i), nv12Frame->UV + nv12Frame->pitch * i, nv12Frame->pitch);
}
OutMgr.m_DeviceContext->Unmap(OutMgr.m_texture, subresource);
Related
How can I copy cv::Mat data back into the sampleBuffer?
My scenario as follow :
I create a cv::Mat from pixelBuffer for landmark detection and add the landmarks to cv::Mat image data. I'd like to copy this cv::Mat into the sample buffer to be shown with the landmark.
Is this possible ?
I achieved this with dlib but need to know how to do it with cv::mat:
char *baseBuffer = (char *)CVPixelBufferGetBaseAddress(imageBuffer);
img.reset();
long position = 0;
while (img.move_next()) {
dlib::bgr_pixel& pixel = img.element();
long bufferLocation = position * 4; //(row * width + column) * 4;
char b = baseBuffer[bufferLocation];
char g = baseBuffer[bufferLocation + 1];
char r = baseBuffer[bufferLocation + 2];
dlib::bgr_pixel newpixel(b, g, r);
pixel = newpixel;
position++;
}
I am answering my own question.
First thing, you need to access the pixel data of cv::mat Image, I followed this great solution
Then you need to copy pixel into the buffer starting from the basebuffer. Following code should help you achieve this :
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
char *baseBuffer = (char *)CVPixelBufferGetBaseAddress(imageBuffer);
long position = 0;
uint8_t* pixelPtr = (uint8_t*)targetImage.data;
int cn = targetImage.channels();
cv::Scalar_<uint8_t> rgbPixel;
for(int i = 0; i < targetImage.rows; i++)
{
for(int j = 0; j < targetImage.cols; j++)
{
long bufferLocation = position * 4;
rgbPixel.val[0] = pixelPtr[i*targetImage.cols*cn + j*cn + 0]; // B
rgbPixel.val[1] = pixelPtr[i*targetImage.cols*cn + j*cn + 1]; // G
rgbPixel.val[2] = pixelPtr[i*targetImage.cols*cn + j*cn + 2]; // R
baseBuffer[bufferLocation] = rgbPixel.val[2];
baseBuffer[bufferLocation + 1] = rgbPixel.val[1];
baseBuffer[bufferLocation + 2] = rgbPixel.val[0];
position++;
}
}
Some things to take note of
make sure you CVPixelBufferLockBaseAddress and
CVPixelBufferUnlockBaseAddress before and after the operation. I
am doing this on CV_8UC3, you might want to check your cv::mat
type.
I haven't done the performance analysis but I am getting smooth output with this.
An intermediate step of my current project requires conversion of opencv's cv::Mat to MTLTexture, the texture container of Metal. I need to store the Floats in the Mat as Floats in the texture; my project cannot quite afford the loss of precision.
This is my attempt at such a conversion.
- (id<MTLTexture>)texForMat:(cv::Mat)image context:(MBEContext *)context
{
id<MTLTexture> texture;
int width = image.cols;
int height = image.rows;
Float32 *rawData = (Float32 *)calloc(height * width * 4,sizeof(float));
int bytesPerPixel = 4;
int bytesPerRow = bytesPerPixel * width;
float r, g, b,a;
for(int i = 0; i < height; i++)
{
Float32* imageData = (Float32*)(image.data + image.step * i);
for(int j = 0; j < width; j++)
{
r = (Float32)(imageData[4 * j]);
g = (Float32)(imageData[4 * j + 1]);
b = (Float32)(imageData[4 * j + 2]);
a = (Float32)(imageData[4 * j + 3]);
rawData[image.step * (i) + (4 * j)] = r;
rawData[image.step * (i) + (4 * j + 1)] = g;
rawData[image.step * (i) + (4 * j + 2)] = b;
rawData[image.step * (i) + (4 * j + 3)] = a;
}
}
MTLTextureDescriptor *textureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA16Float
width:width
height:height
mipmapped:NO];
texture = [context.device newTextureWithDescriptor:textureDescriptor];
MTLRegion region = MTLRegionMake2D(0, 0, width, height);
[texture replaceRegion:region mipmapLevel:0 withBytes:rawData bytesPerRow:bytesPerRow];
free(rawData);
return texture;
}
But it doesn't seem to be working. It reads zeroes every time from the Mat, and throws up EXC_BAD_ACCESS. I need the MTLTexture in MTLPixelFormatRGBA16Float to keep the precision.
Thanks for considering this issue.
One problem here is you’re loading up rawData with Float32s but your texture is RGBA16Float, so the data will be corrupted (16Float is half the size of Float32). This shouldn’t cause your crash, but it’s an issue you’ll have to deal with.
Also as “chappjc” noted you’re using ‘image.step’ when writing your data out, but that buffer should be contiguous and not ever have a step that’s not just (width * bytesPerPixel).
I want to make an effect on an image, such that specific color remains same, and the rest becomes grayscale. However, the output of my program is quite bad, could anyone please help me to improve my code? It breaks when the coordinates exceed 500, and the color range shown is too small. Thanks in advance
Code:
int i1=10, i2=50, a = 10;// coordinates of pixel value, whose color must remain; hue range
IplImage* GetNewImage(IplImage* frame, IplImage* imgHSV){
IplImage* imgNew = cvCreateImage(cvGetSize(frame), IPL_DEPTH_8U, 3);
IplImage* imgBW = cvCreateImage(cvGetSize(frame), IPL_DEPTH_8U, 1);
CvScalar s1;
CvScalar s2;
CvScalar s3;
s3 = cvGet2D(imgHSV, i1, i2);
cvInRangeS(imgHSV, cvScalar(s3.val[0] - 10, s3.val[1] - 50, s3.val[2] - 50), cvScalar(s3.val[0] + 10, s3.val[1] + 50, s3.val[2] + 50), imgBW);
imgNew = cvCloneImage(frame);
for (int i = 0; i < frame->height; i++) for (int j = 0; j < frame->width; j++) {
s1 = cvGet2D(imgBW, i, j);
s2 = cvGet2D(imgNew, i, j);
if (s1.val[0] == 0){
for (int k = 0; k < 3; k++){
s3.val[k] = (s2.val[0] * 0.114 + s2.val[1] * 0.587 + s2.val[2] * 0.299);//converting to grayscale
cvSet2D(imgNew, i, j, s3);
}
}
}
cvReleaseImage(&imgBW);
return imgNew;
}
drop that outdated c-api, it's dead & gone.
you already found out about inRange(), use the outcome as a mask for further processing
there's cvtColor(src,dst,CV_BGR2GRAY) already, prefer the builtin stuff to rolling your own
opencv is a computer-vision / ml library, you're abusing it as a 2nd class photoshop. - don't !
Like the subject says. i am trying to implement openCVSharp surf in unity3d and kinda stuck in the converting part from iplimage to texture2d. Also considering that this converting proces should run at least at 25 fps. So any tips or suggestions are very helpfull!
Might be a bit late, I am working on the same thing now and here is my solution:
void IplImageToTexture2D (IplImage displayImg)
{
for (int i = 0; i < height; i++)
{
for (int j = 0; j < width; j++)
{
float b = (float)displayImg[i, j].Val0;
float g = (float)displayImg[i, j].Val1;
float r = (float)displayImg[i, j].Val2;
Color color = new Color(r / 255.0f, g / 255.0f, b / 255.0f);
videoTexture.SetPixel(j, height - i - 1, color);
}
}
videoTexture.Apply();
}
But it is a bit slow.
Still trying to improve the performance.
Texture2D tex = new Texture2D(640, 480);
CvMat img = new CvMat(640, 480, MatrixType.U8C3);
byte[] data = new byte[640 * 480 * 3];
Marshal.Copy(img.Data, data, 0, 640 * 480 * 3);
tex.LoadImage(data);
To improve performance use Unity3d's undocumented function LoadRawTextureData :
Texture2D IplImageToTexture2D(IplImage img)
{
Texture2D videoTexture = new Texture2D(imWidth, imHeight, TextureFormat.RGB24, false);
byte[] data = new byte[imWidth * imHeight * 3];
Marshal.Copy(img.ImageData, data, 0, imWidth * imHeight * 3);
videoTexture.LoadRawTextureData(data);
videoTexture.Apply();
return videoTexture;
}
I'm trying to create a program that will draw a 2d greyscale spectrum of a given image. I'm using OpenCV and FFTW libraries. By using tips and codes from the internet and modifying them I've managed to load an image, calculate fft of this image and recreate the image from the fft (it's the same). What I'm unable to do is to draw the fourier spectrum itself. Could you please help me?
Here's the code (less important lines removed):
/* Copy input image */
/* Create output image */
/* Allocate input data for FFTW */
in = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * N);
dft = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * N);
/* Create plans */
plan_f = fftw_plan_dft_2d(w, h, in, dft, FFTW_FORWARD, FFTW_ESTIMATE);
/* Populate input data in row-major order */
for (i = 0, k = 0; i < h; i++)
{
for (j = 0; j < w; j++, k++)
{
in[k][0] = ((uchar*)(img1->imageData + i * img1->widthStep))[j];
in[k][1] = 0.;
}
}
/* forward DFT */
fftw_execute(plan_f);
/* spectrum */
for (i = 0, k = 0; i < h; i++)
{
for (j = 0; j < w; j++, k++)
((uchar*)(img2->imageData + i * img2->widthStep))[j] = sqrt(pow(dft[k][0],2) + pow(dft[k][1],2));
}
cvShowImage("iplimage_dft(): original", img1);
cvShowImage("iplimage_dft(): result", img2);
cvWaitKey(0);
/* Free memory */
}
The problem is in the "Spectrum" section. Instead of a spectrum I get some noise. What am I doing wrong? I would be grateful for your help.
You need to draw magnitude of spectrum. here is the code.
void ForwardFFT(Mat &Src, Mat *FImg)
{
int M = getOptimalDFTSize( Src.rows );
int N = getOptimalDFTSize( Src.cols );
Mat padded;
copyMakeBorder(Src, padded, 0, M - Src.rows, 0, N - Src.cols, BORDER_CONSTANT, Scalar::all(0));
// Создаем комплексное представление изображения
// planes[0] содержит само изображение, planes[1] его мнимую часть (заполнено нулями)
Mat planes[] = {Mat_<float>(padded), Mat::zeros(padded.size(), CV_32F)};
Mat complexImg;
merge(planes, 2, complexImg);
dft(complexImg, complexImg);
// После преобразования результат так-же состоит из действительной и мнимой части
split(complexImg, planes);
// обрежем спектр, если у него нечетное количество строк или столбцов
planes[0] = planes[0](Rect(0, 0, planes[0].cols & -2, planes[0].rows & -2));
planes[1] = planes[1](Rect(0, 0, planes[1].cols & -2, planes[1].rows & -2));
Recomb(planes[0],planes[0]);
Recomb(planes[1],planes[1]);
// Нормализуем спектр
planes[0]/=float(M*N);
planes[1]/=float(M*N);
FImg[0]=planes[0].clone();
FImg[1]=planes[1].clone();
}
void ForwardFFT_Mag_Phase(Mat &src, Mat &Mag,Mat &Phase)
{
Mat planes[2];
ForwardFFT(src,planes);
Mag.zeros(planes[0].rows,planes[0].cols,CV_32F);
Phase.zeros(planes[0].rows,planes[0].cols,CV_32F);
cv::cartToPolar(planes[0],planes[1],Mag,Phase);
}
Mat LogMag;
LogMag.zeros(Mag.rows,Mag.cols,CV_32F);
LogMag=(Mag+1);
cv::log(LogMag,LogMag);
//---------------------------------------------------
imshow("Логарифм амплитуды", LogMag);
imshow("Фаза", Phase);
imshow("Результат фильтрации", img);
Can you try to do the IFFT step and see if you recover the original image ? then , you can check step by step where is your problem. Another solution to find the problem is to do this process with a small matrix predefined by you ,and calculate it FFT in MATLAB, and check step by step, it worked for me!