I've constructed a basic scene in Unity, taken some photos using Unity's Physical Camera object, and am now trying to use OpenCV to project known 3D points in the scene onto the image. For reference, the scene is just a few textured spheres and an intersecting plane (here is one of the images taken).
My method is:
Determine the camera intrinsic matrix from the Physical Camera parameters. As I understand, Unity Physical Cameras experience no distortion. I calculate the parameters in Unity as:
float pixelAspectRatio = (float)cam.pixelWidth / (float)cam.pixelHeight;
float f_x = cam.focalLength * ((float)cam.pixelWidth / cam.sensorSize.x);
float f_y = cam.focalLength * pixelAspectRatio * ((float)cam.pixelHeight / cam.sensorSize.y);
float x_0 = (float)cam.pixelWidth / 2;
float y_0 = (float)cam.pixelHeight / 2;
Apply change of basis matrices to convert the 3D points and Unity camera position vector/rotation matrix from Unity coordinates (left-handed) to the coordinates OpenCV uses (right-handed). As I understand it, my choice of change of basis matrix is somewhat arbitrary? It only needs to swap orientation, and I can apply another (orientation-preserving) matrix to the rotation matrix afterwards to ensure the Camera orientation is in OpenCV coordinates (following the convention here). In case I'm unclear, here's the code:
COB = np.array( # Change of basis matrix between Unity and OpenCV
[[1., 0., 0.],
[0., 0., 1.],
[0., 1., 0.]]
)
def tvec_unity_to_cv(tvec:np.ndarray): # tvec is a 3x1 position vector
return COB # tvec
ROT_ADJUSTMENT = np.array( # Secondary COB to ensure camera axes follow OpenCV convention after rotation
[[1., 0., 0.],
[0., 0., -1.],
[0., 1., 0.]]
)
def camera_rot_unity_to_cv(rot:np.ndarray): # rot is a 3x3 rotation matrix
trans = ROT_ADJUSTMENT # COB
return trans # rot # trans
Both COB and trans are idempotent, so I can use them in place of their inverses (I think?!).
Finally, I use cv.projectPoints(3d_points, rvec, tvec, K, distCoeffs=None) to project these points onto my image.
However, it all falls apart here: my projected points don't appear in the correct spots. Before you ask, I'm certain that I'm using the correct 3D positions (they are very easy to identify in Unity) - I'm currently trying to plot the plane corners and the top of the spheres, like this, but it comes out like this (excuse the colour shifting).
The task itself doesn't seem complex, so I'm wondering how I'm getting it wrong. Does anybody have any tips? Thanks!
Related
I am trying to repeat this code here to have real world coordinates on Python. But the results doesn't coinside.
My code :
uvPoint = np.matrix('222.84; 275.05; 1')
rots = cv2.Rodrigues(_rvecs[18])[0]
rot = np.linalg.inv(rots)
cam = np.linalg.inv(camera_matrix)
leftSideMat = rot * cam * uvPoint;
rightSideMat = rot * _tvecs[18];
s = np.true_divide((4.52 + rightSideMat), leftSideMat)
rot * (s * cam * uvPoint - _tvecs[18])
my camera matrix
array([[613.87755242, 0. , 359.6984484 ],
[ 0. , 609.35282925, 242.55955439],
[ 0. , 0. , 1. ]])
rotation matrix
array([[ 0.73824258, 0.03167042, 0.67379142],
[ 0.13296486, 0.97246553, -0.19139263],
[-0.66130042, 0.23088477, 0.71370441]])
and translation vector
array([[-243.00462163],
[ -95.97464544],
[ 935.8852482 ]])
I don't know what is Zconst, but whatever I try for z constant I can't even get close to real world coordinates which is (36, 144). What am I doing wrong here?
based on your comment. i think what you want is the pose estimation with known camera projection matrix.
You should check this link out for python implementation of what you want. https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_pose/py_pose.html
Edit
From your code. it seems your conversion of left and right matrix is correct. But are you sure the input rotation and translation is correct? can you try to plot a box following this tutorial on the original image based on this rotation and translation? If the box coincides with the 2D projected pattern in your data.
If possible. please post the original image that you are using.
Edit
Please check on the original post where scale s is calculated based on
element in which is the 3rd position in the output vector. rightSideMat.at(2,0))/leftSideMat.at(2,0))
and you are doing it as
output vector of rightSideMat), leftSideMat)
Try to do it with the same element operation. By right, scale S should be a float
element, not a matrix.
We have this camera, it is ELP 180 Degree Super Wide Angle Distortion Correction. We need to normalise image capture from this camera. But it is neither fish-eye or standard camera.
As far as I understand, it is barrel distortion. But if you notice straight lines are curved horizontally but vertical lines are not curved. The manufacturer of the camera says it is 'Distortion Corrected'. So let's assume they want to say vertical distortion is corrected but not horizontal.
We tried the following 2 ways to remove distortion but it is not perfect. Please help us to remove its distortion. Thanks a lot.
We tried OpenCV Camera Calibration to get camera Intrinsic Parameters and Distortion Coefficient.
Intrinsic Parameters
[673.9683892, 0., 343.68638231]
[0., 676.08466459, 245.31865398]
[0., 0., 1.]
Distortion
[5.44787247e-02, 1.23043244e-01, -4.52559581e-04, 5.47011732e-03, -6.83110234e-01]
Matlab Computer vision tool to get Intrinsic Parameters and Distortion Coefficient
Intrinsic Parameters
[291.11314081, 0.0, 289.772432415],
[0.0, 274.219315391, 223.73258747],
[0., 0., 1.0]
Distortion
[-3.0108207175179114e-01, 1.0803633903579697e-01, 4.3487318865386296e-03, -5.9566348399883859e-04, -1.8867490263403317e-02]
Result
Original image:
After Removing Distortion:
To me it looks like there may be a prism (or a digital prism-equivalent remapping filter) "squeezing" the image horizontally, which has the effect of visually accentuating the barrel in the horizontal direction.
If I am right, I don't think the standard OpenCv Heikkila-Silven model can fit it. You'll need to fit 2 separate higher order polynomials in (x, y), one for the horizontal component of the distortion and one for the vertical one.
Look up "anamorphic lens distortion"
Try using the fisheye camera model in the Camera Calibrator in matlab
I have an input depth image of rocks which I have to segment. I have found a good way to detect the edges. I want to apply the Watershed Algorithm to segment the rocks. I need to apply the distance transform now to know the distance to the edges. But the results are not as expected.
I tried various other options provided by opencv including CV_DIST_FAIR, CV_DIST_WELSCH.
But I need something of an inverse to the distance transform which shows highest intensity within the rocks. I can use this then as markers.
// Tried to inverse the binary but that doesn't work
//cv::bitwise_not(cannyedge_detected_image, cannyedge_detected_image);
// Finding the distance to the boundaries
cv::Mat distance_transformed_image, dst_distance_transform;
cv::distanceTransform(cannyedge_detected_image, distance_transformed_image, CV_DIST_L2, 3);
cv::normalize(distance_transformed_image, distance_transformed_image, 0, 1., cv::NORM_MINMAX);
cv::resize(distance_transformed_image, dst_distance_transform, cv::Size(image.cols * 3, image.rows * 3));
cv::imshow("Distance Transform", dst_distance_transform);
cv::waitKey(200);
The results look like :
I'm new here and if I broke any rule please help me to improve.
I'm doing some work on visual localization with a working radius about 300m. So I use a big camera with 4912*3684 resolution. But my camera calibration with a chessboard end up with a high reprojection error over 3.6 pix.
The camera_matrix is
[ 3.0126352098515147e+05, 0., 2456.,
0., 4.3598609578377334e+05, 1842.,
0., 0., 1. ]
I realized that fx is far from fy. And the nominal pixel size is 1.25um, the focal length is 755mm.
And I refer to some suggestion from this question FindChessboardCorners cannot detect chessboard on very large images by long focal length lens
The likely correct way to proceed is to start at a lower resolution (i.e. downsizing), then scale up the positions of the corners thus found, and use them as the initial estimates for a run of cvFindCornersSubpix at full resolution.
So I resize the input image before cv::findChessboardCorners() as the code below:
cv::Size msize(1228, 921); //for resolution 4912*3684
int downsize = 4; //downsize scale factor
cv::Mat small; // temp file to downsize the image
cv::resize(imageInput, small, msize);
bool ok = findChessboardCorners(small, board_size, image_points, CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_NORMALIZE_IMAGE);
if(ok){
//rectify the corner
for (size_t j = 0; j < image_points.size(); j++)
{
image_points[j].x = image_points[j].x * downsize;
image_points[j].y = image_points[j].y * downsize;
}
Mat view_gray;
cout << "imageInput.channels()=" << imageInput.channels() << endl;
cvtColor(imageInput, view_gray, CV_RGB2GRAY);
cv::cornerSubPix(view_gray, image_points, cv::Size(11, 11), cv::Size(-1, -1), cv::TermCriteria(CV_TERMCRIT_ITER + CV_TERMCRIT_EPS, 40, 0.01));
image_points_seq.push_back(image_points);
}
double err_first = calibrateCamera(object_points_seq, image_points_seq, image_size, cameraMatrix, distCoeffs, rvecsMat, tvecsMat, CV_CALIB_FIX_K3 | CALIB_FIX_PRINCIPAL_POINT);
And here are my input images:
images for calibration
Please tell me how to get an accurate calibration result!!!
For any calibration to be accurate, you should try considering the following things :
Ensure the focus is correct by verifying it with a simple focus chart.
Environment matters, the scene should be less reflective.
Calibration depends on the focus chart you use. So it is highly critical to have a focus chart to be flat. Any millimetre level bulges also would affect the calibration.
Consider covering the corners to get better distortion coefficients.
Use different pattern positions to cover the maximum of the field of view.
Apart from all these, get the calibration error for individual images and you can observe which image has got more error and which one is good. Unfocussed images and blurred images should be simply discarded for the calibration process. It is an easy process if you give your patient time. Have a good time calibrating.
I'm trying to verify my camera calibration, so I'd like to rectify the calibration images. I expect that this will involve using a call to warpPerspective but I do not see an obvious function that takes the camera matrix, and the rotation and translation vectors to generate the perspective matrix for this call.
Essentially I want to do the process described here (see especially the images towards the end) but starting with a known camera model and pose.
Is there a straightforward function call that takes the camera intrinsic and extrinsic parameters and computes the perspective matrix for use in warpPerspective?
I'll be calling warpPerspective after having called undistort on the image.
In principle, I could derive the solution by solving the system of equations defined at the top of the opencv camera calibration documentation after specifying the constraint Z=0, but I figure that there must be a canned routine that will allow me to orthorectify my test images.
In my seearches, I'm finding it hard to wade through all of the stereo calibration results -- I only have one camera, but want to rectify the image under the constraint that I'm only looking a a planar test pattern.
Actually there is no need to involve an orthographic camera. Here is how you can get the appropriate perspective transform.
If you calibrated the camera using cv::calibrateCamera, you obtained a camera matrix K a vector of lens distortion coefficients D for your camera and, for each image that you used, a rotation vector rvec (which you can convert to a 3x3 matrix R using cv::rodrigues, doc) and a translation vector T. Consider one of these images and the associated R and T. After you called cv::undistort using the distortion coefficients, the image will be like it was acquired by a camera of projection matrix K * [ R | T ].
Basically (as #DavidNilosek intuited), you want to cancel the rotation and get the image as if it was acquired by the projection matrix of form K * [ I | -C ] where C=-R.inv()*T is the camera position. For that, you have to apply the following transformation:
Hr = K * R.inv() * K.inv()
The only potential problem is that the warped image might go outside the visible part of the image plane. Hence, you can use an additional translation to solve that issue, as follows:
[ 1 0 | ]
Ht = [ 0 1 | -K*C/Cz ]
[ 0 0 | ]
where Cz is the component of C along the Oz axis.
Finally, with the definitions above, H = Ht * Hr is a rectifying perspective transform for the considered image.
This is a sketch of what I mean by "solving the system of equations" (in Python):
import cv2
import scipy # I use scipy by habit; numpy would be fine too
#rvec= the rotation vector
#tvec = the translation *emphasized text*matrix
#A = the camera intrinsic
def unit_vector(v):
return v/scipy.sqrt(scipy.sum(v*v))
(fx,fy)=(A[0,0], A[1,1])
Ainv=scipy.array( [ [1.0/fx, 0.0, -A[0,2]/fx],
[ 0.0, 1.0/fy, -A[1,2]/fy],
[ 0.0, 0.0, 1.0] ], dtype=scipy.float32 )
R=cv2.Rodrigues( rvec )
Rinv=scipy.transpose( R )
u=scipy.dot( Rinv, tvec ) # displacement between camera and world coordinate origin, in world coordinates
# corners of the image, for here hard coded
pixel_corners=[ scipy.array( c, dtype=scipy.float32 ) for c in [ (0+0.5,0+0.5,1), (0+0.5,640-0.5,1), (480-0.5,640-0.5,1), (480-0.5,0+0.5,1)] ]
scene_corners=[]
for c in pixel_corners:
lhat=scipy.dot( Rinv, scipy.dot( Ainv, c) ) #direction of the ray that the corner images, in world coordinates
s=u[2]/lhat[2]
# now we have the case that (s*lhat-u)[2]==0,
# i.e. s is how far along the line of sight that we need
# to move to get to the Z==0 plane.
g=s*lhat-u
scene_corners.append( (g[0], g[1]) )
# now we have: 4 pixel_corners (image coordinates), and 4 corresponding scene_coordinates
# can call cv2.getPerspectiveTransform on them and so on..