I have filmed an object on aruco board from two positions with the same camera. I've undistorted images and calibrated camera before work. I take one red point on one shot and calculate 3d line, which corresponds to this point 3D space, and then project it to another image:
The problem is that there is some discrepancy (~5px-15px) between line and point on another image. I also observed the same problem with opengl-generated images, so it doesn't seem to be a problem of my camera. I use this piece of code to detect board position:
MarkerMapPoseTracker MSPoseTracker; // tracks the pose of the marker map
MSPoseTracker.setParams(camParam, theMarkerMapConfig);
MSPoseTracker.estimatePose(ret.markers);
Is it possible to increase tolerance somehow? I've also found function which has some sort of tolerance parameter:
bool estimatePose(Marker& m, const CameraParameters& cam_params, float markerSize, float minErrorRatio = 4 /*tau_e in paper*/)
but I don't know how to pass this parameter to MSPoseTracker.estimatePose. How can I improve precision, if I believe its possible to do it at least theory?
I have ended up with brute-force solution, which I have implemented in lack of time. I've taken coordinates of markers borders on the calibration board with getMarker3DInfo, found coordinates of the points being projected with cv::projectPoints, then found a homography between these two groups of four points. This allowed me to calculate L2 norm between original photo and projected calibration board. Then I have connected bobyqa algorithm, which is opensource GNU optimisation libraty, specialised for multidimensional optimisation with no derivatives for heavy-calculating discrepancy function. This gives pretty good result:
Related
I have a multi-camera system where the field of views are mostly non-overlapping. I have been researching on methods to calibrate the camera extrinsics and the first thing I'm going to try is to take a picture of a chessboard at a known location and use solvePnP from OpenCV to find the extrinsic rotation and translation vectors for each camera separately (following the method described in the answer here).
My problem is, this method uses only one measurement and as every measurement it is prone to errors. I assume that by taking multiple measurements, either by changing the position or the orientation of the chessboard, the accuracy can be improved. But what would be the best way to combine the rotation and translation obtained from the different measurements? A simple average?
In theory I would think that an option could be using solvePnP on all the points at the same time. Since I am calculating extrinsics the camera can't be moved so I would have to change to position and/or orientation of the board for each picture and measure the 3D points positions as accurately as possible each time.
I'm also wondering if using two chessboards in the same picture would be a possible solution, even if OpenCV doesn't seem to support multiple chessboard detection.
Is there a better way to measure extrinsics or anything that I'm missing?
I have two images obtained by a calibrated camera from two different poses. I also have correspondences of 2D points between the images. Some of the points have depth information, so I also know their 3D coordinates. I want to calculate the relative pose between the images.
I know I can compute a fundamental matrix or an essential matrix from the 2D points. I also know PnP can find the pose with 2D-to-3D correspondences and that it's also doable getting just correspondences of 3D points. However, I don't know any algorithm that takes advantage of all the available information. Is there any?
There is only one such algorithm: Bundle Adjustment - everything else is a hack. Get your initial estimates separately, use any "reasonable & simple" hacky way of merging them to get an initial estimate, then byte the bullet and bundle. If you are coding in C++, Google's Ceres is my recommended B.A. library.
I'm using open cv in C++ in multi-view scene with two cameras. I have the intrinsic and extrinsic parameters for both cameras.
I would like to map a (X,Y) point in View 1 to the same point in the second View. I'm am slightly unsure how I should use the intrinsic and extrinsic matrices in order to convert the points to a 3D world and finally end up with the new 2D point in view 2.
It is (normally) not possible to take a 2D coordinate in one image and map it into another 2D coordinate without some additional information.
The main problem is that a single point in the left image will map to a line in the right image (an epipolar line). There are an infinite number of possible corresponding locations because depth is a free parameter. Secondly it's entirely possible that the point doesn't exist in the right image i.e. it's occluded. Finally it may be difficult to determine exactly which point is the right correspondence, e.g. if there is no texture in the scene or if it contains lots of repeating features.
Although the fundamental matrix (which you get out of cv::StereoCalibrate anyway) gives you a constraint between points in each camera: x'Fx = 0, for a given x' there will be a whole family of x's which will satisfy the equation.
Some possible solutions are as follows:
You know the 3D location of a 2D point in one image. Provided that 3D point is in a common coordinate system, you just use cv::projectPoints with the calibration parameters of the other camera you want to project into.
You do some sparse feature detection and matching using something like SIFT or ORB. Then you can calculate a homography to map the points from one image to the other. This makes a few assumptions about things being planes. If you Google panorama homography, there are plenty of lecture slides detailing this.
You calibrate your cameras, perform an epipolar rectification (cv::StereoRectify, cv::initUndistortRectifyMap, cv::remap) and then run them through a stereo matcher. The output is a disparity map which gives you exactly what you want: a per-pixel mapping from one camera to the other. That is, left[y,x] = right[y, x+disparity_map[y,x]].
(1) is by far the easiest, but it's unlikely you have that information already. (2) is often doable and might be suitable, and as another commenter pointed out will be poor where the planarity assumption fails. (3) is the general (ideal) solution, but has its own drawbacks and relies on the images being amenable to dense matching.
I try to match two overlapping images captured with a camera. To do this, I'd like to use OpenCV. I already extracted the features with the SurfFeatureDetector. Now I try to to compute the rotation and translation vector between the two images.
As far as I know, I should use cvFindExtrinsicCameraParams2(). Unfortunately, this method require objectPoints as an argument. These objectPoints are the world coordinates of the extracted features. These are not known in the current context.
Can anybody give me a hint how to solve this problem?
The problem of simultaneously computing relative pose between two images and the unknown 3d world coordinates has been treated here:
Berthold K. P. Horn. Relative orientation revisited. Berthold K. P. Horn. Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 545 Technology ...
EDIT: here is a link to the paper:
http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.64.4700
Please see my answer to a related question where I propose a solution to this problem:
OpenCV extrinsic camera from feature points
EDIT: You may want to take a look at bundle adjustments too,
http://en.wikipedia.org/wiki/Bundle_adjustment
That assumes an initial estimate is available.
EDIT: I found some code resources you might want to take a look at:
Resource I:
http://www.maths.lth.se/vision/downloads/
Two View Geometry Estimation with Outliers
C++ code for finding the relative orientation of two calibrated
cameras in presence of outliers. The obtained solution is optimal in
the sense that the number of inliers is maximized.
Resource II:
http://lear.inrialpes.fr/people/triggs/src/ Relative orientation from
5 points: a somewhat more polished C routine implementing the minimal
solution for relative orientation of two calibrated cameras from
unknown 3D points. 5 points are required and there can be as many as
10 feasible solutions (but 2-5 is more common). Also requires a few
CLAPACK routines for linear algebra. There's also a short technical
report on this (included with the source).
Resource III:
http://www9.in.tum.de/praktika/ppbv.WS02/doc/html/reference/cpp/toc_tools_stereo.html
vector_to_rel_pose Compute the relative orientation between two
cameras given image point correspondences and known camera parameters
and reconstruct 3D space points.
There is a theoretical solution, however, the OpenCV implementation of camera pose estimation lacks the needed tools.
The theoretical approach:
Step 1: extract the homography (the matrix describing the geometrical transform between images). use findHomography()
Step 2. Decompose the result matrix into rotations and translations. Use cv::solvePnP();
Problem: findHomography() returns a 3x3 matrix, corresponding to a projection from a plane to another. solvePnP() needs a 3x4 matrix, representing the 3D rotation/translation of the objects. I think that with some approximations, you can modify the solvePnP to give you some results, but it requires a lot of math and a very good understanding of 3D geometry.
Read more about at http://en.wikipedia.org/wiki/Transformation_matrix
i have a stereopair,
photo 1: http://savepic.org/1671682.jpg
photo 2: http://savepic.org/1667586.jpg
there is coordinate system in each image. How can I find coordinates of point A in this system using OpenCV library. It would be nice to see sample code.
I've looked for it at opencv.willowgarage.com/documentation/cpp/camera_calibration_and_3d_reconstruction.html but haven't found (or haven't understood :) )
Your 'stereo' images are fine. What you have already done is solve the correspondence problem: in both images you have indicated points 'A'. This means that you know which pixel corresponds to eachother labeling point 'A'.
What you want to do, is triangulate where your camera is. You can only do this by first calibrating your camera. This is inside of OpenCV already.
http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html
http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html
This gives you the exact vector/ray of light for each vector, and the optical center of your cameras through which the ray passes. Moreover, you need stereo calibration. This establishes the orientation and position of each camera with respect through each other.
From that point on, your triangulation is simple, knowing the pixel location in both images of point 'A'. You have
Location and orientation of camera 1 and camera 2
Otical Ray Vector (pixel location) from the cameras to label 'A'.
So you have 2 locations in space, and 2 rays from these location. The intersection of these rays is your 3D answer.
Note that in practice there rays will never exactly intersect (2 lines in 3D rarely do), so you need to approximate. Use opencv function triangulatePoints(), using the input of the stereo calibration and the pixel index relating to label A.
Firstly of all this is not truly a stereo pair. A nice stereo pair needs to have 60%-80% overlap usually small rotation differences between images. Even if this pair had the necessary BASE to be a good stereo pair due to the extremely kappa rotation the resulting epipolar image would be useless.
Secondly among others you should take a look at the camera calibration and collinearity equations both supported by OpenCV
http://en.wikipedia.org/wiki/Camera_resectioning
http://en.wikipedia.org/wiki/Collinearity_equation
You need to understand the maths.
If the page isn't enough then you should look at the opencv book - it devotes a couple of chapters to this. Then there are a lot of textbooks that cover it in more detail