OpenCV Mat class: Accessing elements of a multi-channel matrix - opencv

I currently want to read in some values into a 3-channel, 480 row by 640 column matrix of 8 bit unsigned integer values. I am initializing the matrix like this:
Declaration:
rgbMatrix = Mat::zeros(480,640,CV_8UC3);
When I try to iterate through the entire matrix I am unable to assign/grab values using the following method. The values simply stay 0. My code looks like this:
for (int i = 0; i < rgbMatrix.rows; i++)
{
for (int j = 0; j < rgbMatrix.cols; j++)
{
(rgbMatrix.data + rgbMatrix.step * i)[j * rgbMatrix.channels() + 0] = *value0*;
(rgbMatrix.data + rgbMatrix.step * i)[j * rgbMatrix.channels() + 1] = *value1*;
(rgbMatrix.data + rgbMatrix.step * i)[j * rgbMatrix.channels() + 2] = *value2*;
}
}
However, when I declare three separate 1-channel matrices (also 480 row by 640 column of 8 bit unsigned integer values) and attempt to access elements of those matrices the following code works:
Declaration:
rgbMatrix0 = Mat::zeros(480,640,CV_8UC1);
rgbMatrix1 = Mat::zeros(480,640,CV_8UC1);
rgbMatrix2 = Mat::zeros(480,640,CV_8UC1);
for (int i = 0; i < rgbMatrix0.rows; i++)
{
for (int j = 0; j < rgbMatrix0.cols; j++)
{
(rgbMatrix0.data + rgbMatrix0.step * i)[j] = *value0*;
(rgbMatrix1.data + rgbMatrix1.step * i)[j] = *value1*;
(rgbMatrix2.data + rgbMatrix2.step * i)[j] = *value2*;
}
}
Now, I want to use just one matrix for these operations, as having to keep track of three separate variables will get tiresome after a while. I have a feeling that I am not accessing the right point in memory for the three-channel matrix. Does anyone know how I can accomplish what I did in the second portion of code but using one three-channel matrix instead of three separate one-channel matrices?
Thanks.

There are plenty of ways to do it, for example:
cv::Mat rgbMatrix(480,640,CV_8UC3);
for (int i = 0; i < rgbMatrix.rows; i++)
for (int j = 0; j < rgbMatrix.cols; j++)
for (int k = 0; k < 3; k++)
rgbMatrix.at<cv::Vec3b>(i,j)[k] = value;
[k] here is the channel value.
To set the all the matrix elements to a specific value like 5 for example you can do this:
cv::Mat rgbMatrix2(cv::Size(480,640), CV_8UC3, cv::Scalar(5,5,5));
std::cout << rgbMatrix2 << std::endl;

Sorry I can't see your code since I am writing from iPhone. When you use 3 channel matrix you can get the pixel using:
Vec3b pix = rgbMatrix.at(row,col);
Now you can access channel using:
pix[0] = 255; pix[1] += pix[2];
P.s. Generally rgbMatrix pixel is of type vec3b or vec3d. Always cast image.at<> with relevant type

Very Simple using Vec3b - for uchar, Vec3i - for int, Vec3f - for float, Vec3d - for double
Mat rgbMatrix = Mat::zeros(480,640,CV_8UC1);
for (int i = 0; i < rgbMatrix.rows; i++)
{
for (int j = 0; j < rgbMatrix.cols; j++)
{
rgbMatrix.at<Vec3b>(i,j)[0] = *value0;
rgbMatrix.at<Vec3b>(i,j)[1] = *value1;
rgbMatrix.at<Vec3b>(i,j)[2] = *value2;
}
}

vector<cv::Point3f> xyzBuffer;
cv::Mat xyzBuffMat = cv::Mat(307200, 1, CV_32FC3);
for (int i = 0; i < xyzBuffer.size(); i++) {
xyzBuffMat.at<cv::Vec3f>(i, 1, 0) = xyzBuffer[i].x;
xyzBuffMat.at<cv::Vec3f>(i, 1, 1) = xyzBuffer[i].y;
xyzBuffMat.at<cv::Vec3f>(i, 1, 2) = xyzBuffer[i].z;
}
Here, 0, 1, and 2 are respectively the channels that store x, y and z values.

Related

Opencv Mat efficiency linearized by right triangle

How to efficiency linearized Mat (symmetric matrix) to one row by right triangle.
For example, when I have:
0aabbb
b0aaaa
ba0bba
bac0aa
aaaa0c
abcab0
and then from that I get:
aabbbaaaabbaaac
Something like this:
...
template<class T>
Mat SSMJ::triangleLinearized(Mat mat){
int c = mat.cols;
Mat row = Mat(1, ((c*c)-c)/2, mat.type());
int i = 0;
for(int y = 1; y < mat.rows; y++)
for(int x = y; x < mat.cols; x++) {
row.at<T>(i)=mat.at<T>(y, x);
i++;
}
return row;
}
...
Since data in your mat is just a 1d array stored in row.data you can do whatever you want with it. I don't think you will find anything more special (w/o using vectorized methods) than just copying from this array.
int rows = 6;
char data[] = { 0,1,2,3,4,5,
0,1,2,3,4,5,
0,1,2,3,4,5,
0,1,2,3,4,5,
0,1,2,3,4,5};
char result[100];
int offset = 0;
for (int i = 0; i < 5; offset += 5-i, i++) {
memcpy(&result[offset] , &data[rows * i + i + 1], 5 - i);
}
Or with opencv Mat it would be
int rows = mat.cols;
char result[100]; // you can calculate how much data u need
int offset = 0;
for (int i = 0; i < 5; offset += 5-i, i++) {
memcpy(&result[offset] , &mat.data[rows * i + i + 1], 5 - i);
}
Mat resultMat(1, offset, result);

OpenCV C++ Convert Byte array to Mat

How can I convert byte array to Mat which is received from socket ?.
My client application will send color image data like this
Mat frame; //colour image
int imgSize = frame.total()*frame.elemSize();
int bytes = send(clientSock, frame.data, imgSize, 0));//write to the socket
And the server will receives the data like
char sockData[imgSize];
Mat img;
for (int i = 0; i < imgSize; i += bytes) {
bytes = recv(connectSock, sockData +i, imgSize - i, 0));
}
// Write to mat
for (int i = 0; i < img.rows; i++) {
for (int j = 0; j < img.cols; j++) {
(img.row(i)).col(j) = (uchar)sockData[((img.cols)*i)+j];
}
}
I am getting distorted image at the receiver. Is there any problem in my code ?
Thanks in advance.......
If you have colour image you may read it in a math with 3 channels of uchar so change this piece of code:
for (int i = 0; i < img.rows; i++) {
for (int j = 0; j < img.cols; j++) {
(img.row(i)).col(j) = (uchar)sockData[((img.cols)*i)+j];
}
}
with this:
int baseIndex = 0;
for (int i = 0; i < img.rows; i++) {
for (int j = 0; j < img.cols; j++) {
img.at<cv::Vec3b>(i,j) = cv::Vec3b(sockData[baseIndex + 0],
sockData[baseIndex + 1],
sockData[baseIndex + 2]);
baseIndex = baseIndex + 3;
}
}
Maybe this should work.
Doesn't this work?
cv::Mat frame(img.rows, img.cols, CV_8UC3, sockData);
Just replace CV_8UC3 with the correct image format:
CV_<bit-depth>{U|S|F}C(<number_of_channels>)
see https://docs.opencv.org/2.4/modules/core/doc/basic_structures.html
Edit: There is a 5th additional field which can be useful. The number of bytes per row (in case there are a few padding bytes). In working with V4L2 today, I successfully used this cv::Mat constructor:
v4l2_format camera_format = ...; // see https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/vidioc-g-fmt.html#description
cv::Mat mat(camera_format.fmt.pix.height,
camera_format.fmt.pix.width,
CV_8UC3,
raw_data_ptr,
camera_format.fmt.pix.bytesperline);
I solved the problem using below code.
int ptr=0;
for (int i = 0; i < img.rows; i++) {
for (int j = 0; j < img.cols; j++) {
img.at<cv::Vec3b>(i,j) = cv::Vec3b(sockData[ptr+0],sockData[ptr+1],sockData[ptr+2]);
ptr=ptr+3;
}
}
Adding to Michele answer, one can also use the MatIterator to solve this.
cv::Mat m;
m.create(10, 10, CV_32FC3);
// This is the socket data.
float *array = (float *)malloc( 3*sizeof(float)*10*10 );
cv::MatIterator_<cv::Vec3f> it = m.begin<cv::Vec3f>();
for (unsigned i = 0; it != m.end<cv::Vec3f>(); it++ ) {
for ( unsigned j = 0; j < 3; j++ ) {
(*it)[j] = *(array + i );
i++;
}
}
Now you have a float cv::Mat. In case of 8 bit, simply change float to uchar and Vec3f to Vec3b and CV_32FC3 to CV_8UC3

Opencv - trouble creating matrix from other matrix (malloc.c:2451: sYSMALLOc: Assertion )

I'm creating a new Mat by jumping pixels of the original image, but I get this error:
PRM algorithm: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
My code is:
int width = round(img.cols / M);
int height = round(img.rows / M);
cv::Mat res(height, width, CV_8U);
for (int i = 0; i < width; i++) {
for (int j = 0; j < height; j++) {
res.at<cv::Vec3b>(i, j)[0] = img.at<cv::Vec3b>(i * M, j * M)[0];
res.at<cv::Vec3b>(i, j)[1] = img.at<cv::Vec3b>(i * M, j * M)[1];
res.at<cv::Vec3b>(i, j)[2] = img.at<cv::Vec3b>(i * M, j * M)[2];
}
}
return res;
I also tried using uchar* ptr = img.ptr<uchar>(i) and ptr[j] so it's possible to access the data directly, but I receive the same error.
I was searching, and tried some "solution" such as sYSMALLOc: Assertion Failed error in opencv, but the trouble keeps appearing.
In general, this error occurs when you try to access memory at an invalid location. Your code has two issues that contribute to the problem.
First, you access elements of res with cv::Vec3b, which implies a three-channel image, but you initialize it as single channel. Change the initialization to:
cv::Mat res(height, width, CV_8UC3); // Needs to be three channels!
Second, .at<>(i, j) accesses elements by their row index i and column index j. However, your i and j indices refer to column and row, respectively. Your iteration limits should be swapped:
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
// ...

Fast Gaussian Blur image filter with ARM NEON

I'm trying to make a mobile fast version of Gaussian Blur image filter.
I've read other questions, like: Fast Gaussian blur on unsigned char image- ARM Neon Intrinsics- iOS Dev
For my purpose i need only a fixed size (7x7) fixed sigma (2) Gaussian filter.
So, before optimizing for ARM NEON, I'm implementing 1D Gaussian Kernel in C++, and comparing performance with OpenCV GaussianBlur() method directly in mobile environment (Android with NDK). This way it will result in a much simpler code to optimize.
However the result is that my implementation is 10 times slower then OpenCV4Android version. I've read that OpenCV4 Tegra have optimized GaussianBlur implementation, but I don't think that standard OpenCV4Android have those kind of optimizations, so why is my code so slow?
Here is my implementation (note: reflect101 is used for pixel reflection when applying filter near borders):
Mat myGaussianBlur(Mat src){
Mat dst(src.rows, src.cols, CV_8UC1);
Mat temp(src.rows, src.cols, CV_8UC1);
float sum, x1, y1;
// coefficients of 1D gaussian kernel with sigma = 2
double coeffs[] = {0.06475879783, 0.1209853623, 0.1760326634, 0.1994711402, 0.1760326634, 0.1209853623, 0.06475879783};
//Normalize coeffs
float coeffs_sum = 0.9230247873f;
for (int i = 0; i < 7; i++){
coeffs[i] /= coeffs_sum;
}
// filter vertically
for(int y = 0; y < src.rows; y++){
for(int x = 0; x < src.cols; x++){
sum = 0.0;
for(int i = -3; i <= 3; i++){
y1 = reflect101(src.rows, y - i);
sum += coeffs[i + 3]*src.at<uchar>(y1, x);
}
temp.at<uchar>(y,x) = sum;
}
}
// filter horizontally
for(int y = 0; y < src.rows; y++){
for(int x = 0; x < src.cols; x++){
sum = 0.0;
for(int i = -3; i <= 3; i++){
x1 = reflect101(src.rows, x - i);
sum += coeffs[i + 3]*temp.at<uchar>(y, x1);
}
dst.at<uchar>(y,x) = sum;
}
}
return dst;
}
A big part of the problem, here, is that the algorithm is overly precise, as #PaulR pointed out. It's usually best to keep your coefficient table no more precise than your data. In this case, since you appear to be processing uchar data, you would use roughly an 8-bit coefficient table.
Keeping these weights small will particularly matter in your NEON implementation because the narrower you have the arithmetic, the more lanes you can process at once.
Beyond that, the first major slowdown that stands out is that having the image edge reflection code within the main loop. That's going to make the bulk of the work less efficient because it will generally not need to do anything special in that case.
It might work out better if you use a special version of the loop near the edges, and then when you're safe from that you use a simplified inner loop that doesn't call that reflect101() function.
Second (more relevant to prototype code) is that it's possible to add the wings of the window together before applying the weighting function, because the table contains the same coefficients on both sides.
sum = src.at<uchar>(y1, x) * coeffs[3];
for(int i = -3; i < 0; i++) {
int tmp = src.at<uchar>(y + i, x) + src.at<uchar>(y - i, x);
sum += coeffs[i + 3] * tmp;
}
This saves you six multiplies per pixel, and it's a step towards some other optimisations around controlling overflow conditions.
Then there are a couple of other problems related to the memory system.
The two-pass approach is good in principle, because it saves you from performing a lot of recomputation. Unfortunately it can push the useful data out of L1 cache, which can make everything quite a lot slower. It also means that when you write the result out to memory, you're quantising the intermediate sum, which can reduce precision.
When you convert this code to NEON, one of the things you will want to focus on is trying to keep your working set inside the register file, but without discarding calculations before they've been fully utilised.
When people do use two passes, it's usual for the intermediate data to be transposed -- that is, a column of input becomes a row of output.
This is because the CPU will really not like fetching small amounts of data across multiple lines of the input image. It works out much more efficient (because of the way the cache works) if you collect together a bunch of horizontal pixels, and filter those. If the temporary buffer is transposed, then the second pass also collects together a bunch of horizontal points (which would vertical in the original orientation) and it transposes its output again so it comes out the right way.
If you optimise to keep your working set localised, then you might not need this transposition trick, but it's worth knowing about so that you can set yourself a healthy baseline performance. Unfortunately, localisation like this does force you to go back to the non-optimal memory fetches, but with the wider data types that penalty can be mitigated.
If this is specifically for 8 bit images then you really don't want floating point coefficients, especially not double precision. Also you don't want to use floats for x1, y1. You should just use integers for coordinates and you can use fixed point (i.e. integer) for the coefficients to keep all the filter arithmetic in the integer domain, e.g.
Mat myGaussianBlur(Mat src){
Mat dst(src.rows, src.cols, CV_8UC1);
Mat temp(src.rows, src.cols, CV_16UC1); // <<<
int sum, x1, y1; // <<<
// coefficients of 1D gaussian kernel with sigma = 2
double coeffs[] = {0.06475879783, 0.1209853623, 0.1760326634, 0.1994711402, 0.1760326634, 0.1209853623, 0.06475879783};
int coeffs_i[7] = { 0 }; // <<<
//Normalize coeffs
float coeffs_sum = 0.9230247873f;
for (int i = 0; i < 7; i++){
coeffs_i[i] = (int)(coeffs[i] / coeffs_sum * 256); // <<<
}
// filter vertically
for(int y = 0; y < src.rows; y++){
for(int x = 0; x < src.cols; x++){
sum = 0; // <<<
for(int i = -3; i <= 3; i++){
y1 = reflect101(src.rows, y - i);
sum += coeffs_i[i + 3]*src.at<uchar>(y1, x); // <<<
}
temp.at<uchar>(y,x) = sum;
}
}
// filter horizontally
for(int y = 0; y < src.rows; y++){
for(int x = 0; x < src.cols; x++){
sum = 0; // <<<
for(int i = -3; i <= 3; i++){
x1 = reflect101(src.rows, x - i);
sum += coeffs_i[i + 3]*temp.at<uchar>(y, x1); // <<<
}
dst.at<uchar>(y,x) = sum / (256 * 256); // <<<
}
}
return dst;
}
This is the code after implementing all the suggestions of #Paul R and #sh1, summarized as follows:
1) use only integer arithmetic (with precision to taste)
2) add the values ​​of the pixels at the same distance from the mask center before applying the multiplications, to reduce the number of multiplications.
3) apply only horizontal filters to take advantage of the storage by rows of the matrices
4) separate cycles around the edges from those inside the image not to make unnecessary calls to reflection functions. I totally removed the functions of reflection, including them inside the loops along the edges.
5) In addition, as a personal observation, to improve rounding without calling a (slow) function "round" or "cvRound", I've added to both temporary and final pixel results 0.5f (= 32768 in integers precision) to reduce the error / difference compared to OpenCV.
Now the performance is much better from about 15 to about 6 times slower than OpenCV.
However, the resulting matrix is not perfectly identical to that obtained with the Gaussian Blur of OpenCV. This is not due to arithmetic length (sufficient) as well as removing the error remains. Note that this is a minimum difference, between 0 and 2 (in absolute value) of pixel intensity, between the matrices resulting from the two versions. Coefficient are the same used by OpenCV, obtained with getGaussianKernel with same size and sigma.
Mat myGaussianBlur(Mat src){
Mat dst(src.rows, src.cols, CV_8UC1);
Mat temp(src.rows, src.cols, CV_8UC1);
int sum;
int x1;
double coeffs[] = {0.070159, 0.131075, 0.190713, 0.216106, 0.190713, 0.131075, 0.070159};
int coeffs_i[7] = { 0 };
for (int i = 0; i < 7; i++){
coeffs_i[i] = (int)(coeffs[i] * 65536); //65536
}
// filter horizontally - inside the image
for(int y = 0; y < src.rows; y++){
uchar *ptr = src.ptr<uchar>(y);
for(int x = 3; x < (src.cols - 3); x++){
sum = ptr[x] * coeffs_i[3];
for(int i = -3; i < 0; i++){
int tmp = ptr[x+i] + ptr[x-i];
sum += coeffs_i[i + 3]*tmp;
}
temp.at<uchar>(y,x) = (sum + 32768) / 65536;
}
}
// filter horizontally - edges - needs reflect
for(int y = 0; y < src.rows; y++){
uchar *ptr = src.ptr<uchar>(y);
for(int x = 0; x <= 2; x++){
sum = 0;
for(int i = -3; i <= 3; i++){
x1 = x + i;
if(x1 < 0){
x1 = -x1;
}
sum += coeffs_i[i + 3]*ptr[x1];
}
temp.at<uchar>(y,x) = (sum + 32768) / 65536;
}
}
for(int y = 0; y < src.rows; y++){
uchar *ptr = src.ptr<uchar>(y);
for(int x = (src.cols - 3); x < src.cols; x++){
sum = 0;
for(int i = -3; i <= 3; i++){
x1 = x + i;
if(x1 >= src.cols){
x1 = 2*src.cols - x1 - 2;
}
sum += coeffs_i[i + 3]*ptr[x1];
}
temp.at<uchar>(y,x) = (sum + 32768) / 65536;
}
}
// transpose to apply again horizontal filter - better cache data locality
transpose(temp, temp);
// filter horizontally - inside the image
for(int y = 0; y < src.rows; y++){
uchar *ptr = temp.ptr<uchar>(y);
for(int x = 3; x < (src.cols - 3); x++){
sum = ptr[x] * coeffs_i[3];
for(int i = -3; i < 0; i++){
int tmp = ptr[x+i] + ptr[x-i];
sum += coeffs_i[i + 3]*tmp;
}
dst.at<uchar>(y,x) = (sum + 32768) / 65536;
}
}
// filter horizontally - edges - needs reflect
for(int y = 0; y < src.rows; y++){
uchar *ptr = temp.ptr<uchar>(y);
for(int x = 0; x <= 2; x++){
sum = 0;
for(int i = -3; i <= 3; i++){
x1 = x + i;
if(x1 < 0){
x1 = -x1;
}
sum += coeffs_i[i + 3]*ptr[x1];
}
dst.at<uchar>(y,x) = (sum + 32768) / 65536;
}
}
for(int y = 0; y < src.rows; y++){
uchar *ptr = temp.ptr<uchar>(y);
for(int x = (src.cols - 3); x < src.cols; x++){
sum = 0;
for(int i = -3; i <= 3; i++){
x1 = x + i;
if(x1 >= src.cols){
x1 = 2*src.cols - x1 - 2;
}
sum += coeffs_i[i + 3]*ptr[x1];
}
dst.at<uchar>(y,x) = (sum + 32768) / 65536;
}
}
transpose(dst, dst);
return dst;
}
According to Google document, on Android device, using float/double is twice slower than using int/uchar.
You may find some solutions to speed up your C++ code on this Android documents.
https://developer.android.com/training/articles/perf-tips

Obtain array from IplImage in JavaCV

I need to convert the code below from C++ to Java. In C++ I use openCV and I need to convert it in Java using JavaCV.
IplImage* img = cvLoadImage(argv[0]);
int rows = img->height;
int cols = img->width;
Mat matimg(img);
vector<vector<double> > img_vec(rows, vector<double>(cols));
for (int i=0; i < rows; i++) {
for (int j =0; j < cols; j++){
unsigned char temp;
temp = ((uchar*) matimg.data + i * matimg.step)[j * matimg.elemSize() + 1 ];
img_vec[i][j] = (double) temp;
}
}
I've tried the following conversion to java, but it doesn't work properly. I printed the values of temp and it is 0 all the times and for the same imgage the values of matimg.step and matimg.elemSize() are different in the C++ code and the Java code.
In c++ I get matimg.step = 2400 and matimg.elemSize() = 3 while in Java i get 3000 and 1.
Here is the code in java:
IplImage img = cvLoadImage(argv[0]);
int rows = img.height();
int cols = img.width();
CvMat matimg = img.asCvMat();
double img_vec[][] = new double[rows][cols];
for (int i=0; i < rows; i++) {
for (int j =0; j < cols; j++){
short temp;
temp = matimg.data_s().get(i * matimg.step() + j * matimg.elemSize() + 1);
img_vec[i][j] = (double) temp;
}
}
I don't understand where am I doing wrong?
Any help is appreciated,
Thanks.
I've solved my problem using this:
ByteBuffer buffer = img.getByteBuffer();
double img_vec[][] = new double[rows][cols];
for (int i=0; i < rows; i++) {
for (int j =0; j < cols; j++){
int ind = i * img.widthStep() + j * img.nChannels() + 1;
img_vec[i][j] = (buffer.get(ind) & 0xFF);
}
}

Resources