I have a PointGrey Ladybug3 Camera. It's a panoramic (multi)camera (5 camera to do a 360º and 1 camera looking up).
I've done all the calibration and rectification so what I end up is from all pixels of the 6 images I know it's 3d position wrt a global frame.
What I would do now is convert this 3d points to a panoramic image. The most common is a radial (Equirectangular) projection like the following one:
For all the 3D points (X,Y,Z) it's possible to find theta and phi coordinate like:
My question is, Is it possible to do this automatically with opencv? Or if I do this manually what is the best way to convert that bunch of pixels in theta,phi coordinates to an image?
The official ladybug SDK uses OpenGL for all this operations, but I was wondering if it's possible to do this in opencv.
Thanks,
Josep
The approach I used to solve this problem was the following:
Create an empty image with the desired output size.
For every pixel in the output image find the theta and phi coordinates. (Linearly) Theta goes from -Pi to Pi and phi from 0 to Pi
Set a projection radius R and find 3D coordinate from theta, phi and R.
Find for how many cameras is the 3D point visible and the correspondent pixel position.
Copy the pixel of the image where the pixel is closer to the principal point. Or any other valid criteria...
My code looks like:
cv::Mat panoramic;
panoramic=cv::Mat::zeros(PANO_HEIGHT,PANO_WIDTH,CV_8UC3);
double theta, phi;
double R=calibration.getSphereRadius();
int result;
double dRow=0;
double dCol=0;
for(int y = 0; y!= PANO_HEIGHT; y++){
for(int x = 0; x !=PANO_WIDTH ; x++) {
//Rescale to [-pi, pi]
theta=-(2*PI*x/(PANO_WIDTH-1)-PI); //Sign change needed.
phi=PI*y/(PANO_HEIGHT-1);
//From theta and phi find the 3D coordinates.
double globalZ=R*cos(phi);
double globalX=R*sin(phi)*cos(theta);
double globalY=R*sin(phi)*sin(theta);
float minDistanceCenter=5000; // Doesn't depend on the image.
float distanceCenter;
//From the 3D coordinates, find in how many camera falls the point!
for(int cam = 0; cam!= 6; cam++){
result=calibration.ladybugXYZtoRC(globalX, globalY, globalZ, cam, dRow, dCol);
if (result==0){ //The 3d point is visible from this camera
cv::Vec3b intensity = image[cam].at<cv::Vec3b>(dRow,dCol);
distanceCenter=sqrt(pow(dRow-imageHeight/2,2)+pow(dCol-imageWidth/2,2));
if (distanceCenter<minDistanceCenter) {
panoramic.ptr<unsigned char>(y,x)[0]=intensity.val[0];
panoramic.ptr<unsigned char>(y,x)[1]=intensity.val[1];
panoramic.ptr<unsigned char>(y,x)[2]=intensity.val[2];
minDistanceCenter=distanceCenter;
}
}
}
}
}
Related
I am using opencv::solvePnP to return a camera pose. I run PnP, and it returns the rvec and tvec values.(rotation vector and position).
I then run this function to convert the values to the camera pose:
void GetCameraPoseEigen(cv::Vec3d tvecV, cv::Vec3d rvecV, Eigen::Vector3d &Translate, Eigen::Quaterniond &quats)
{
Mat R;
Mat tvec, rvec;
tvec = DoubleMatFromVec3b(tvecV);
rvec = DoubleMatFromVec3b(rvecV);
cv::Rodrigues(rvec, R); // R is 3x3
R = R.t(); // rotation of inverse
tvec = -R*tvec; // translation of inverse
Eigen::Matrix3d mat;
cv2eigen(R, mat);
Eigen::Quaterniond EigenQuat(mat);
quats = EigenQuat;
double x_t = tvec.at<double>(0, 0);
double y_t = tvec.at<double>(1, 0);
double z_t = tvec.at<double>(2, 0);
Translate.x() = x_t * 10;
Translate.y() = y_t * 10;
Translate.z() = z_t * 10;
}
This works, yet at some rotation angles, the converted rotation values flip randomly between positive and negative values. Yet, the source rvecV value does not. I assume this means I am going wrong with my conversion. How can i get a stable Quaternion from the PnP returned cv::Vec3d?
EDIT: This seems to be Quaternion flipping, as mentioned here:
Quaternion is flipping sign for very similar rotations?
Based on that, i have tried adding:
if(quat.w() < 0)
{
quat = quat.Inverse();
}
But I see the same flipping.
Both quat and -quat represent the same rotation. You can check that by taking a unit quaternion, converting it to a rotation matrix, then doing
quat.coeffs() = -quat.coeffs();
and converting that to a rotation matrix as well.
If for some reason you always want a positive w value, negate all coefficients if w is negative.
The sign should not matter...
... rotation-wise, as long as all four fields of the 4D quaternion are getting flipped. There's more to it explained here:
Quaternion to EulerXYZ, how to differentiate the negative and positive quaternion
Think of it this way:
Angle/axis both flipped mean the same thing
and mind the clockwise to counterclockwise transition much like in a mirror image.
There may be convention to keep the quat.w() or quat[0] component positive and change other components to opposite accordingly. Assume w = cos(angle/2) then setting w > 0 just means: I want angle to be within the (-pi, pi) range. So that the -270 degrees rotation becomes +90 degrees rotation.
Doing the quat.Inverse() is probably not what you want, because this creates a rotation in the opposite direction. That is -quat != quat.Inverse().
Also: check that both systems have the same handedness (chirality)! Test if your rotation matrix determinant is +1 or -1.
(sry for the image link, I don't have enough reputation to embed them).
I have a set of 3-D points in the world. Using OpenCV I have calibrated the camera for extrinsic parameters.
So now I am able to map 3-D points to the corresponding pixel in the 2-D image.
i.e for [X Y Z] I have the corresponding [u v] in the image.
In OpenGL I have normalized the world 3-D points and defined a surface , where I want my texture mapping to be done.
The 3-D surface obtained from the code looks like in the figure below.
3-D SURFACE in OPENGL
It is a bowl shaped surface.
Now I want to map the texture in the image to the 3-D points in OpenGL.
Information I have:
1. 3-D co-ordinates of a parabolic surface in openGL obtained from world points.
2. Corresponding 2-D Image coordinates and R-G-B colour info of pixels.
How would I go about doing this.
Here is my code snippet for getting the 3-D points lying on the model surface shown in the link, using real world coordinates
Also it stores the corresponding (u v) pixel coordinate's RGB colour info in image for rendering:
for (int z_3D = 0; z_3D < 30; z_3D+=1)
{
for (int x_3D = 0; x_3D < 102; x_3D+=1)
{
for (int y_3D = 0; y_3D < 135.5; y_3D+=1)
{
//3-D point in real world(in cms)
x = x_3D;
y = y_3D;
z = z_3D;
object_point[0].x = x;
object_point[1].y = y;
object_point[2].z = z;
//Project 3-D point to 2-D image and get the corresponding (u,v)
//rvec and tvec Obtained using SolvPnP in openCV
projectPoints(object_point, rvec_front, tvec_front, cameraMatrix_Front, distCoeffs_Front, check_front_image_pts);
//Store colour information in the corresponding 2-D point
//Points not lying on the surface is black
rgb.r = 0;
rgb.g = 0;
rgb.b = 0;
//Convert real world coordinates into openGl coordinates(-1 to +1)
x = (x - CHART_WIDTH / 2) / (CHART_WIDTH / 2);
y = -(y - CHART_LENGTH / 2) / (CHART_LENGTH / 2);
z = z / CHART_HEIGHT;
//Parabolic surface model
float x_4 = x*x*x*x;
float y_4 = y*y*y*y;
if (x_4 + y_4 < r_4)
{
//Store 3-D point
vertex_obj.vertex_x.push_back(x);
vertex_obj.vertex_y.push_back(y);
vertex_obj.vertex_z.push_back((x_4 + y_4) / (a_4));
/**/
//Store colour information in the corresponding 2-D point
rgb.r = front.at<Vec3b>(check_front_image_pts[0].y, check_front_image_pts[0].x)[2];
rgb.g = front.at<Vec3b>(check_front_image_pts[0].y, check_front_image_pts[0].x)[1];
rgb.b = front.at<Vec3b>(check_front_image_pts[0].y, check_front_image_pts[0].x)[0];
//printf("%f %f %f\n", rgb.r, rgb.g, rgb.b);
vertex_obj.vertex_colour.push_back(rgb);
}
else if (sqrt((x_4 + y_4 - r_4)*(x_4 + y_4 - r_4)) < 0.0001)
{
//Store 3-D point
vertex_obj.vertex_x.push_back(x);
vertex_obj.vertex_y.push_back(y);
//vertex_obj.vertex_z.push_back(1.0);
vertex_obj.vertex_z.push_back((x_4 + y_4) / (a_4)+0.0001);
/*
//Store colour information in the corresponding 2-D point
rgb.r = front.at<Vec3b>(check_front_image_pts[0].y, check_front_image_pts[0].x)[2];
rgb.g = front.at<Vec3b>(check_front_image_pts[0].y, check_front_image_pts[0].x)[1];
rgb.b = front.at<Vec3b>(check_front_image_pts[0].y, check_front_image_pts[0].x)[0];*/
vertex_obj.vertex_colour.push_back(rgb);
}
}
}
}
This is my rendering code snippet
void render()
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glLoadIdentity(); // Reset the model-view matrix
glRotated(ph, 1, 0, 0);
glRotated(th, 0, 1, 0);
glBegin(GL_POINTS);
for (int i = 0; i < vertex_obj.vertex_x.size(); i++)
{
//Give the colour info from the pixel in the image
glColor3f(vertex_obj.vertex_colour[i].r/255.0, vertex_obj.vertex_colour[i].g/255.0, vertex_obj.vertex_colour[i].b/255.0);
//Give the vertex of points lying on the surface defined
glVertex3f(vertex_obj.vertex_x[i], vertex_obj.vertex_y[i], vertex_obj.vertex_z[i]);
}
//glColor3f(0, 0, 1);
//glVertex2f(1.0, -1.0);
glEnd();
glutSwapBuffers();
}
QUESTIIONS
How do I fill the surface with the image.
I am aware that only a part of the surface is going to be filled based on the 3-D and the corresponding 2-D image coordinates.
Also If I give colour to a vertex, how do I interpolate it in the gaps. OpenGL interpolates for known shapes like triangles and quads.
But this is almost a random point cloud and I want to interpolate between nearest pixels.
How do I do this.
I'm assuming that the texture image is shaped something like the surface you want to paint it onto, for example a human face being applied to a 3D hemisphere?
First, define your surface as a surface, not a point cloud. You're already generating vertex coordinates for points on the surface, so just link them up into a triangle mesh. (GL_TRIANGLE_STRIP)
Second, don't apply color to the vertices of your surface. Set the texture coordinates instead. (In OpenGL they're often called s,t not u,v but the meaning is the same.) The vertex color should be just plain white. Pass the image to OpenGL with glTexture2D.
This way you get the GPU to look up the texture coords, interpolate, and fill in the surface rather than writing your own code.
Oh, and you are using very old fashioned glBegin..glEnd blocks. Before someone else does, you really need to learn how to use vertex arrays or preferably OpenGL 4 VBOs.
Hope this helps.
There are different techniques to do this if you want to do this in real time search for voxel based dense mapping. If performance is not major concern and you want to do it in opengl application you can triangulate point clouds using algorithms like delnauy triangulation and then use uv to do texture mapping. If you want it to be offline process you can Use meshlab like software export to 3d formats triangulated and load it in opengl application.
I want to find the corner position of an blurred image with a corner inside it. like the following example:
I can make sure that only one corner is inside the image, and I assume that
the corner is part of a black and white chessboard.
How can I detect the cross position with openCV?
Thanks!
Usually you can determine the corner using the gradient:
Gx = im[i][j+1] - im[i][j-1]; Gy = im[i+1][j] – im[i-1][j];
G^2 = Gx^2 + Gy^2;
teta = atan2 (Gy, Gx);
As your image is blurred, you should compute the gradient at a larger scale:
Gx = im[i][j+delta] - im[i][j-delta]; Gy = im[i+ delta][j] – im[i- delta][j];
Here is the result that I obtained for delta = 50:
The gradient norm (multiplied by 20)
gradient norm http://imageshack.us/scaled/thumb/822/xdpp.jpg
The gradient direction:
gradient direction http://imageshack.us/scaled/thumb/844/h6zp.jpg
another solution
#include <opencv2/opencv.hpp>
using namespace cv;
int main()
{
Mat img=imread("c:/data/corner.jpg");
Mat gray;
cvtColor(img,gray,CV_BGR2GRAY);
threshold(gray,gray,100,255,CV_THRESH_BINARY);
int step=15;
std::vector<Point> points;
for(int i=0;i<gray.rows;i+=step)
for(int j=0;j<gray.cols;j+=step)
if(gray.at<uchar>(i,j)==255)
points.push_back(Point(j,i));
//fit a rotated rectangle
RotatedRect box = minAreaRect(Mat(points));
//circle(img,box.center,2,Scalar(255,0,0),-1);
//invert it,fit again and get average of centers(may not be needed if a 'good' threshold is found)
Point p1=Point(box.center.x,box.center.y);
points.clear();
gray=255-gray;
for(int i=0;i<gray.rows;i+=step)
for(int j=0;j<gray.cols;j+=step)
if(gray.at<uchar>(i,j)==255)
points.push_back(Point(j,i));
box = minAreaRect(Mat(points));
Point p2=Point(box.center.x,box.center.y);
//circle(img,p2,2,Scalar(0,255,0),-1);
circle(img,Point((p1.x+p2.x)/2,(p1.y+p2.y)/2),3,Scalar(0,0,255),-1);
imshow("img",img);
waitKey();
return 0;
}
Rather than work right away at a ridiculously large scale, as suggested by others, I recommend downsizing first (which has the effect of deblurring), do one pass of Harris to find the corner, then upscale its position and do a pass of findCornerSubpix at full resolution with a large window (large enough to encompass the obvious saddle point of the intensity).
In this way you get the best of both worlds: fast detection to initialize the refinement, and accurate refinement given the original imagery.
See also this other relevant answer
I'm working on a 3D Pose estimation system. I used OpenCVs function cvPosit to calculate the rotation matrix and the translation vector.
I also need the angles of the rotation matrix, but no algorithms seem to be working.
The function cv::RQDecomp3x3(), which was the answer of topic "in opencv : how to get yaw, roll, pitch from POSIT rotation matrix" cannot work, because the function needs the 3x3 matrix of the projection matrix.
Furthermore I tried to use algorithms from the links below, but nothing worked.
visionopen.com/cv/vosm/doc/html/recognitionalgs_8cpp_source.html
stackoverflow.com/questions/16266740/in-opencv-how-to-get-yaw-roll-pitch-from-posit-rotation-matrix
quad08pyro.groups.et.byu.net/vision.htm
stackoverflow.com/questions/13565625/opencv-c-posit-why-are-my-values-always-nan-with-small-focal-lenght
www.c-plusplus.de/forum/308773-full
I used the most common Posit Tutorial and an own example with Blender, so I could render an image to retreive the image points and to know the exact angles. The object's Z-Axis in Blender was rotated by 10 degrees - And I checked all the degrees of all 3 Axis due to changes in Axis between Blender and OpenCV.
double focalLength = 700.0;
CvPOSITObject* positObject;
std::vector<CvPoint3D32f> modelPoints;
modelPoints.push_back(cvPoint3D32f(0.0f, 0.0f, 0.0f));
modelPoints.push_back(cvPoint3D32f(CUBE_SIZE, 0.0f, 0.0f));
modelPoints.push_back(cvPoint3D32f(0.0f, CUBE_SIZE, 0.0f));
modelPoints.push_back(cvPoint3D32f(0.0f, 0.0f, CUBE_SIZE));
std::vector<CvPoint2D32f> imagePoints;
imagePoints.push_back( cvPoint2D32f( 157,372) );
imagePoints.push_back( cvPoint2D32f(423,386 ));
imagePoints.push_back( cvPoint2D32f( 157,108 ));
imagePoints.push_back( cvPoint2D32f(250,337));
// Moving the points to the image center as described in the tutorial
for (int i = 0; i < imagePoints.size();i++) {
imagePoints[i] = cvPoint2D32f(imagePoints[i].x -320, 240 - imagePoints[i].y);
}
CvVect32f translation_vector = new float[3];
CvTermCriteria criteria = cvTermCriteria(CV_TERMCRIT_EPS | CV_TERMCRIT_ITER,iterations, 0.1f);
positObject = cvCreatePOSITObject( &modelPoints[0], static_cast<int>(modelPoints.size()));
CvMatr32f rotation_matrix = new float[9];
cvPOSIT( positObject, &imagePoints[0], focalLength, criteria, rotation_matrix, translation_vector );
algorithms to get angles...
I already tried to calculate the results from radian to degree and clockwise but I already get bad results using the rotation matrix of cvPosit from OpenCV. I also changed matrix format to check wrong formatting...
I used simple rotation matrices - like only doing a rotation of the x-axis, y and z-axis and some algorithm worked. The rotation matrix of cvPosit didn't work with that algorithm.
I appreciate any support.
When finding a reference image in a scene using SURF, I would like to crop the found object in the scene, and "straighten" it back using warpPerspective and the reversed homography matrix.
Meaning, let's say I have this SURF result:
Now, I would like to crop the found object in the scene:
and "straighten" only the cropped image with warpPerspective using the reversed homography matrix. The result I'm aiming at is that I'll get an image containing, roughly, only the object, and some distorted leftovers from the original scene (as the cropping is not a 100% the object alone).
Cropping the found object, and finding the homography matrix and reversing it are simple enough. Problem is, I can't seem to understand the results from warpPerspective. Seems like the resulting image contains only a small portion of the cropped image, and in a very large size.
While researching warpPerspective I found that the resulting image is very large due to the nature of the process, but I can't seem to wrap my head around how to do this properly. Seems like I just don't understand the process well enough. Would I need to warpPerspective the original (not cropped) image and than crop the "straightened" object?
Any advice?
try this.
given that you have the unconnected contour of your object (e.g. the outer corner points of the box contour) you can transform them with your inverse homography and adjust that homography to place the result of that transformation to the top left region of the image.
compute where those object points will be warped to (use the inverse homography and the contour points as input):
cv::Rect computeWarpedContourRegion(const std::vector<cv::Point> & points, const cv::Mat & homography)
{
std::vector<cv::Point2f> transformed_points(points.size());
for(unsigned int i=0; i<points.size(); ++i)
{
// warp the points
transformed_points[i].x = points[i].x * homography.at<double>(0,0) + points[i].y * homography.at<double>(0,1) + homography.at<double>(0,2) ;
transformed_points[i].y = points[i].x * homography.at<double>(1,0) + points[i].y * homography.at<double>(1,1) + homography.at<double>(1,2) ;
}
// dehomogenization necessary?
if(homography.rows == 3)
{
float homog_comp;
for(unsigned int i=0; i<transformed_points.size(); ++i)
{
homog_comp = points[i].x * homography.at<double>(2,0) + points[i].y * homography.at<double>(2,1) + homography.at<double>(2,2) ;
transformed_points[i].x /= homog_comp;
transformed_points[i].y /= homog_comp;
}
}
// now find the bounding box for these points:
cv::Rect boundingBox = cv::boundingRect(transformed_points);
return boundingBox;
}
modify your inverse homography (result of computeWarpedContourRegion and inverseHomography as input)
cv::Mat adjustHomography(const cv::Rect & transformedRegion, const cv::Mat & homography)
{
if(homography.rows == 2) throw("homography adjustement for affine matrix not implemented yet");
// unit matrix
cv::Mat correctionHomography = cv::Mat::eye(3,3,CV_64F);
// correction translation
correctionHomography.at<double>(0,2) = -transformedRegion.x;
correctionHomography.at<double>(1,2) = -transformedRegion.y;
return correctionHomography * homography;
}
you will call something like
cv::warpPerspective(objectWithBackground, output, adjustedInverseHomography, sizeOfComputeWarpedContourRegionResult);
hope this helps =)