I am using OpenCV's calibrateCamera and am trying to understand how it calculates the reprojection error as well as what this error represents. It appears to be the RMS of the (Euclidean) distance between the projected points and measured image points--is this right? However, what does it mean for the final reprojection error to be "minimized"? Does calibrateCamera() explicitly use the function projectPoints() to find the projected points?
The reprojection error is the error (Euclidean distance for example) between the 3D points reprojected with the estimated intrinsic and extrinsic matrices and the 2D image points detected by some image processing techniques (corners of the chessboard pattern for example).
The final reprojection error is minimized because the problem to estimate the set of intrinsic and extrinsic parameters is a non-linear problem and thus you have to find the set of parameters that minimizes this reprojection error iteratively.
More information: A Flexible New Technique for Camera Calibration ; Zhengyou Zhang ; 2000.
The reprojection error is not defined 100% mathematically in literature. Formally, a single reprojection error is the 2d vector relating measured to projected pixel coordinates.
OpenCV and most other software and related bundle adjustment algorithms use the sum of squares of Euclidean lengths of these 2d vectors as the objective function during optimization.
As Geoff and Alessandro Jacopson have pointed out, the return value of cv::calibrateCamera() is the RMS of Euclidean errors (contrary to the doc of v3.1). This quantity is directly related to the objective function value, but not exactly the same.
Other definitions of the reprojection error include the mean Euclidean length and the median of Euclidean lengths. Both are legitimate and care must be given when comparing values.
An in-depth article on this topic can be found here: https://calib.io/blogs/knowledge-base/understanding-reprojection-errors
I am referring to OpenCV version 3.1.0, here you find doc for calibrateCamera
http://docs.opencv.org/3.1.0/d9/d0c/group__calib3d.html#ga687a1ab946686f0d85ae0363b5af1d7b and the doc says (bold is mine):
The algorithm performs the following steps:
Compute the initial intrinsic parameters (the option only available
for planar calibration patterns) or read them from the input
parameters. The distortion coefficients are all set to zeros initially
unless some of CV_CALIB_FIX_K? are specified.
Estimate the initial
camera pose as if the intrinsic parameters have been already known.
This is done using solvePnP .
Run the global Levenberg-Marquardt
optimization algorithm to minimize the reprojection error, that is,
the total sum of squared distances between the observed feature points
imagePoints and the projected (using the current estimates for camera
parameters and the poses) object points objectPoints. See
projectPoints for details.
The function returns the final re-projection error.
Anyway, instead of relying on the doc, I will prefer to have a look at the code:
https://github.com/Itseez/opencv/blob/3.1.0/modules/calib3d/src/calibration.cpp#L3298
When you use projectPoints, You need to calculate RMS by hand after the reprojection. This might help.
OPENCV: Calibratecamera 2 reprojection error and custom computed one not agree
Here is the reprojection error calculation from the OpenCV code calibrate.cpp line 1629:
return std::sqrt(reprojErr/total);
Where total is the sum of all points for all images.
Related
I have been trying to calibrate my camera recently. To check the accuracy of the calibration, I was told that I should be checking reprojection errors. The problem is I could not exactly find out how to calculate the reprojection errors.
For this reason, I, first, reprojected the points with the camera parameters I found, and printed those side by side with the initial twodpoints that were detected in the calibration pattern. Since, I did not know the formula; I only visually wanted to see the differences in pixels. I saw 1-2 pixel differences for each of the corresponding entries of the two matrices. Then, I saw the reprojection error calculation in the OpenCV documentation, and implemented it in my code. There comes the strange part; reprojection error turned out to be (0.1676704). L2 norm was used to calculate reprojection error in the below code. If my pixels are 1-2 pixel of the ground truth value, how can my reprojection error be around 0.16? I do think the error is divided once more than it was meant to. In the line 3, norm difference between twodpoints[i] and imgpoints2 is calculated and divided by total number of points. At this point, I thought that was like a root-mean-square approach; thus, explanation for division is taking the mean. However, after summing mean errors at line 4, that sum is again divided by number of points. I did not understand why? Does this code correctly calculate reprojection error?
for i in range(len(threedpoints)):
imgpoints2, _ = cv2.projectPoints(threedpoints[i], r_vecs[i], t_vecs[i], mtx, dist) error = cv2.norm(twodpoints[i], imgpoints2, cv2.NORM_L2)/len(imgpoints2)
mean_error += error
print( "total error: {}".format(mean_error/len(threedpoints)) )
If you'd like to check, it is provided here as well: https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html
In this Distill article (https://distill.pub/2017/feature-visualization/) in footnote 8 authors write:
The Fourier transforms decorrelates spatially, but a correlation will still exist
between colors. To address this, we explicitly measure the correlation between colors
in the training set and use a Cholesky decomposition to decorrelate them.
I have trouble understanding how to do that. I understand that for an arbitrary image I can calculate a correlation matrix by interpreting the image's shape as [channels, width*height] instead of [channels, height, width]. But how to take the whole dataset into account? It can be averaged over, but that doesn't have anything to do with Cholesky decomposition.
Inspecting the code confuses me even more (https://github.com/tensorflow/lucid/blob/master/lucid/optvis/param/color.py#L24). There's no code for calculating correlations, but there's a hard-coded version of the matrix (and the decorrelation happens by matrix multiplication with this matrix). The matrix is named color_correlation_svd_sqrt, which has svd inside of it, and SVD wasn't mentioned anywhere else. Also the matrix there is non-triangular, which means that it hasn't come from the Cholesky decomposition.
Clarifications on any points I've mentioned would be greatly appreciated.
I figured out the answer to your question here: How to calculate the 3x3 covariance matrix for RGB values across an image dataset?
In short, you calculate the RGB covariance matrix for the image dataset and then do the following calculations
U,S,V = torch.svd(dataset_rgb_cov_matrix)
epsilon = 1e-10
svd_sqrt = U # torch.diag(torch.sqrt(S + epsilon))
What is the procedure to calculate reprojected points, reprojected errors and mean reprojection error from the given world points (Original coordinates), intrinsic matrix, rotation matrices and translation vector?
Is there any inbuilt opencv function for that or we should calculate manuallay?
If we have to calculate manually, what is the best way to get reprojected points?
projectPoints projects 3D points to an image plane.
calibrateCamera returns the final re-projection error. calibrateCamera finds the camera intrinsic and extrinsic parameters from several views of a calibration pattern.
The function estimates the intrinsic camera parameters and extrinsic
parameters for each of the views. The algorithm is based on
[Zhang2000]1 and [BouguetMCT]2. The coordinates of 3D object points and
their corresponding 2D projections in each view must be specified.
That may be achieved by using an object with a known geometry and
easily detectable feature points. Such an object is called a
calibration rig or calibration pattern, and OpenCV has built-in
support for a chessboard as a calibration rig (see
findChessboardCorners() ).
The algorithm performs the following steps:
Compute the initial intrinsic parameters (the option only available
for planar calibration patterns) or read them from the input
parameters. The distortion coefficients are all set to zeros initially
unless some of CV_CALIB_FIX_K? are specified.
Estimate the initial
camera pose as if the intrinsic parameters have been already known.
This is done using solvePnP().
Run the global Levenberg-Marquardt
optimization algorithm to minimize the reprojection error, that is,
the total sum of squared distances between the observed feature points
imagePoints and the projected (using the current estimates for camera
parameters and the poses) object points objectPoints. See
projectPoints() for details. The function returns the final
re-projection error.
1ZHANG, Zhengyou. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2000, 22.11: 1330-1334.
2J.Y.Bouguet. MATLAB calibration tool. http://www.vision.caltech.edu/bouguetj/calib_doc/
I have 4 coplanar points in object coordinates and the correspoinding image points (on image plane). I want to compute the relative translation and rotation of the object plane with respect to the camera.
FindExtrinsicCameraParams2 is supposed to be the solution. But I'm having troubles with using it. Errors keep on showing when compiling
Has anyone successfully used this function in OpenCV?? Could I have some comments or sample code to use this function??
Thank you!
I would use the OpenCV function FindHomography() as it is simpler and you can converto easily from homography to extrinsic parameters.
You have to call the function like this
FindHomography(srcPoints, dstPoints, H, method, ransacReprojThreshold=3.0, status=None)
method is CV_RANSAC. If you pass more than 4 points, RANSAC will select the best 4-point set to satisfy the model.
You will get the homography in H, and if you want to convert it to extrinsic parameters you should do what I explain in this post.
Basically, the extrinsics matrix (Pose), has the first, second and fourth columns equal tp homography. The third column is redundant because it is the crossproduct of columns one and two.
After several days testing OpenCV functions related to 3D calibration, getting over all the errors, awkward output numbers, I finally get the correct outputs for these functions including findHomography, solvePnP (new version of FindExtrinsicCameraParams) and cvProjectPoints. Some of the tips have been discussed in use OpenCV cvProjectPoints2 function.
These tips are also applied for the error in this post. Specifically, in this post, my violation is passing float data to CV_64F Mat. All done now!!
You can use CalibrateCamera2.
objectPts - your 4 coplanar points
imagePts - your corresponding image points.
This method will compute instrinsic matrix and distortion coefficients, which tell you how the objectPts have been projected as the imagePts on to the camera's imaging plane.
There are no extrinsic parameters to compute here since you are using only 1 camera. If you used 2 cameras, then you are looking at computing Extrinsic Matrix using StereoCalibrate.
Ankur
When using OpenCV's findHomography function to estimate an homography between two sets of points, from different images, you will sometimes get a bad homography due to outliers within your input points, even if you use RANSAC or LMEDS.
// opencv java example:
Mat H = Calib3d.findHomography( src_points, dst_points, Calib3d.RANSAC, 10 );
How can you tell if the resulting 3x3 homography matrix is acceptable or not?
I have looked for an answer to this here in Stackoverflow and in Google and was unable to find it.
I found this article, but it is a bit cryptic to me:
"The geometric error for homographies"
The best way to tell if the homography is acceptable is.
1- Take the points of one image and reproject them using the computed homography.
//for one 3D point, this would be the projection
px' = H * px;
py' = H * py;
pz' = H * pz;
2- Calculate the euclidean distance between the reprojected points and the real points in the image.
Reprojection error for one point. p is the projected point and q is the real point.
3- Establish a treshold that decides if the reprojection error is acceptable.
For example, an error greater than one pixel wouldn't be acceptable for many tracking applications.