I have been trying to calibrate my camera recently. To check the accuracy of the calibration, I was told that I should be checking reprojection errors. The problem is I could not exactly find out how to calculate the reprojection errors.
For this reason, I, first, reprojected the points with the camera parameters I found, and printed those side by side with the initial twodpoints that were detected in the calibration pattern. Since, I did not know the formula; I only visually wanted to see the differences in pixels. I saw 1-2 pixel differences for each of the corresponding entries of the two matrices. Then, I saw the reprojection error calculation in the OpenCV documentation, and implemented it in my code. There comes the strange part; reprojection error turned out to be (0.1676704). L2 norm was used to calculate reprojection error in the below code. If my pixels are 1-2 pixel of the ground truth value, how can my reprojection error be around 0.16? I do think the error is divided once more than it was meant to. In the line 3, norm difference between twodpoints[i] and imgpoints2 is calculated and divided by total number of points. At this point, I thought that was like a root-mean-square approach; thus, explanation for division is taking the mean. However, after summing mean errors at line 4, that sum is again divided by number of points. I did not understand why? Does this code correctly calculate reprojection error?
for i in range(len(threedpoints)):
imgpoints2, _ = cv2.projectPoints(threedpoints[i], r_vecs[i], t_vecs[i], mtx, dist) error = cv2.norm(twodpoints[i], imgpoints2, cv2.NORM_L2)/len(imgpoints2)
mean_error += error
print( "total error: {}".format(mean_error/len(threedpoints)) )
If you'd like to check, it is provided here as well: https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html
Related
I've been working on a pose estimation project and one of the steps is finding the pose using the recoverPose function of OpenCV.
int cv::recoverPose(InputArray E,
InputArray points1,
InputArray points2,
InputArray cameraMatrix,
OutputArray R,
OutputArray t,
InputOutputArray mask = noArray()
)
I have all the required info: essential matrix E, key points in image 1 points1, corresponding key points in image 2 points2, and the cameraMatrix. However, the one thing that still confuses me a lot is the int value (i.e. the number of inliers) returned by the function. As per the documentation:
Recover relative camera rotation and translation from an estimated essential matrix and the corresponding points in two images, using cheirality check. Returns the number of inliers which pass the check.
However, I don't completely understand that yet. I'm concerned with this because, at some point, the yaw angle (calculated using the output rotation matrix R) suddenly jumps by more than 150 degrees. For that particular frame, the number of inliers is 0. So, as per the documentation, no points passed the cheirality check. But still, what does it mean exactly? Can that be the reason for the sudden jump in yaw angle? If yes, what are my options to avoid that? As the process is iterative, that one sudden jump affects all the further poses!
This function decomposes the Essential matrix E into R and t. However, you can get up to 4 solutions, i. e. pairs of R and t. Of these 4, only one is physically realizable, meaning that the other 3 project the 3D points behind one or both cameras.
The cheirality check is what you use to find that one physically realizable solution, and this is why you need to pass matching points into the function. It will use the matching 2D points to triangulate the corresponding 3D points using each of the 4 R and t pairs, and choose the one for which it gets the most 3D points in front of both cameras. This accounts for the possibility that some of the point matches can be wrong. The number of points that end up in front of both cameras is the number of inliers that the functions returns.
So, if the number of inliers is 0, then something went very wrong. Either your E is wrong, or the point matches are wrong, or both. In this case you simply cannot estimate the camera motion from those two images.
There are several things you can check.
After you call findEssentialMat you get the inliers from the RANSAC used to find E. Make sure that you are passing only those inlier points into recoverPose. You don't want to pass in all the points that you passed into findEssentialMat.
Before you pass E into recoverPose check if it is of rank 2. If it is not, then you can enforce the rank 2 constraint on E. You can take the SVD of E, set the smallest eigenvalue to 0, and then reconstitute E.
After you get R and t from recoverPose, you can check that R is indeed a rotation matrix with the determinate equal to 1. If the determinant is equal to -1, then R is a reflection, and things have gone wrong.
I am using OpenCV's calibrateCamera and am trying to understand how it calculates the reprojection error as well as what this error represents. It appears to be the RMS of the (Euclidean) distance between the projected points and measured image points--is this right? However, what does it mean for the final reprojection error to be "minimized"? Does calibrateCamera() explicitly use the function projectPoints() to find the projected points?
The reprojection error is the error (Euclidean distance for example) between the 3D points reprojected with the estimated intrinsic and extrinsic matrices and the 2D image points detected by some image processing techniques (corners of the chessboard pattern for example).
The final reprojection error is minimized because the problem to estimate the set of intrinsic and extrinsic parameters is a non-linear problem and thus you have to find the set of parameters that minimizes this reprojection error iteratively.
More information: A Flexible New Technique for Camera Calibration ; Zhengyou Zhang ; 2000.
The reprojection error is not defined 100% mathematically in literature. Formally, a single reprojection error is the 2d vector relating measured to projected pixel coordinates.
OpenCV and most other software and related bundle adjustment algorithms use the sum of squares of Euclidean lengths of these 2d vectors as the objective function during optimization.
As Geoff and Alessandro Jacopson have pointed out, the return value of cv::calibrateCamera() is the RMS of Euclidean errors (contrary to the doc of v3.1). This quantity is directly related to the objective function value, but not exactly the same.
Other definitions of the reprojection error include the mean Euclidean length and the median of Euclidean lengths. Both are legitimate and care must be given when comparing values.
An in-depth article on this topic can be found here: https://calib.io/blogs/knowledge-base/understanding-reprojection-errors
I am referring to OpenCV version 3.1.0, here you find doc for calibrateCamera
http://docs.opencv.org/3.1.0/d9/d0c/group__calib3d.html#ga687a1ab946686f0d85ae0363b5af1d7b and the doc says (bold is mine):
The algorithm performs the following steps:
Compute the initial intrinsic parameters (the option only available
for planar calibration patterns) or read them from the input
parameters. The distortion coefficients are all set to zeros initially
unless some of CV_CALIB_FIX_K? are specified.
Estimate the initial
camera pose as if the intrinsic parameters have been already known.
This is done using solvePnP .
Run the global Levenberg-Marquardt
optimization algorithm to minimize the reprojection error, that is,
the total sum of squared distances between the observed feature points
imagePoints and the projected (using the current estimates for camera
parameters and the poses) object points objectPoints. See
projectPoints for details.
The function returns the final re-projection error.
Anyway, instead of relying on the doc, I will prefer to have a look at the code:
https://github.com/Itseez/opencv/blob/3.1.0/modules/calib3d/src/calibration.cpp#L3298
When you use projectPoints, You need to calculate RMS by hand after the reprojection. This might help.
OPENCV: Calibratecamera 2 reprojection error and custom computed one not agree
Here is the reprojection error calculation from the OpenCV code calibrate.cpp line 1629:
return std::sqrt(reprojErr/total);
Where total is the sum of all points for all images.
This is one of the two Euler Lagrange equations for de-blurring which I need to solve:
http://i.stack.imgur.com/AtCLZ.jpg
u_r is the reference image, which is known. u_0 is the original blurred image. k is the blurring kernel, unknown. lambda is a constant which is also known, or rather empirically determined. I need to solve this equation for k.
I tried solving it from the Fourier domain, but the results were disappointing due to some reason. The output image did not look much different from the input image, but pixel level difference of 2 or 3 gray-scale levels were there.
In all the papers that I found, they say that they solved the equation in code, using lagged diffusivity to make it linear and then using the conjugate gradient or fixed point method. But I can't get my head around this, because the kernel k which is being convolved with image u_r, is an unknown. How do I implement it then, in code? I can't if the unknown k is in a convolution.
I performed the calibration with chessboard (I defined the size of the corner in mm). Now, are the results of the calibration (matrix roto translation) in mm?
Short and long answer: yes.
You basically set the units when you define the chessboard square size, so since you defined it in mm, your results will be in mm.
Edit: Keep in mind though that the reprojection error is in pixels and not in mm.
When using OpenCV's findHomography function to estimate an homography between two sets of points, from different images, you will sometimes get a bad homography due to outliers within your input points, even if you use RANSAC or LMEDS.
// opencv java example:
Mat H = Calib3d.findHomography( src_points, dst_points, Calib3d.RANSAC, 10 );
How can you tell if the resulting 3x3 homography matrix is acceptable or not?
I have looked for an answer to this here in Stackoverflow and in Google and was unable to find it.
I found this article, but it is a bit cryptic to me:
"The geometric error for homographies"
The best way to tell if the homography is acceptable is.
1- Take the points of one image and reproject them using the computed homography.
//for one 3D point, this would be the projection
px' = H * px;
py' = H * py;
pz' = H * pz;
2- Calculate the euclidean distance between the reprojected points and the real points in the image.
Reprojection error for one point. p is the projected point and q is the real point.
3- Establish a treshold that decides if the reprojection error is acceptable.
For example, an error greater than one pixel wouldn't be acceptable for many tracking applications.