I need to calibrate a fish eye camera. To do this I have checkerboard images for which I can detect corners. These cover most of the effective area where the images lie.
I then used the python openCV library, and managed to get the intrinsic matrix K and distortion parameters D via the cv2.fisheye.calibrate module.
I got:
K = array([[220.18647104, 0. , 341.83122807],
[ 0. , 220.11052182, 343.39463313],
[ 0. , 0. , 1. ]])
D = array([[-0.00390754],
[-0.01050568],
[ 0.01015501],
[-0.00330259]])
So far so good, I can even use the cv2.fisheye.initUndistortRectifyMap with the K and D calculated, as well as the cv2.remap methods to obtain a good undistorted image, as shown here,
Example undistorted images
However, as one can see, not all of the field of view is accounted for, and my understanding is that K needs to be rescaled, and a new matrix needs to be calculated to include pixels outside of the best part of the image.
To do this I used the following (adapted from a tutorial found online),
img = cv2.imread(fname) # load the image
dim1 = img.shape[:2][::-1] # size of the img
dim2 = dim1 # wanting dim2 = dim1
dim3 = dim1 # wanting the entire undistorted image on the same size as the original img
scaled_K = K * dim1[0] / DIM[0] # Here, since dim1[0] = DIM[0], scaled_K = K
scaled_K[2][2] = 1.0 # keep the 1 on the third dimension
# That's where problem occurs, I would expect with balance = 0, and all dimensions
# set to dim1 to have new_K = K. But maybe I'm wrong here.
new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(scaled_K,
D,
dim2,
np.eye(3),
balance=0.)
# perform a remapping using the new_K
map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, D, np.eye(3), new_K, dim3, cv2.CV_16SC2)
undistorted_img = cv2.remap(img, map1, map2,interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
After doing this, I have,
new_K = array([[1.36845717e-03, 0.00000000e+00, 6.84232172e+02],
[0.00000000e+00, 1.36798515e-03, 6.83999949e+02],
[0.00000000e+00, 0.00000000e+00, 1.00000000e+00]])
which is way off, and as a result gives a bad undistorted image, as shown here,
wrong undistorted image
I have tried to cover more of the area with corners on the checkerboard, I have tried to change the balance keyword with values between 0 and 1, I have tried to use other dimensions for dim2 and dim3, I have tried with cropped images, where only the central part of the fish eye is retained,etc... all these without any successes....
Therefore any help would be much appreciated!
Thanks ;)
Related
I'm filming with 6 RGB cameras a scene that I want to reconstruct in 3D, kind of like in the following picture. And I forgot to take a calibration chessboard. So I used a blank rectangle board instead and filmed it, as I would film a regular chessboard.
First step, calibration --> OK.
I obviously couldn't use cv2.findChessboardCorners, so I made a small program that would allow me to click and store the location of each 4 corners. I calibrated from these 4 points for about 10-15 frames as a test.
Tl;Dr: It seemed to work great.
Next step, triangulation. --> NOT OK
I use direct linear transform (DLT) to triangulate my points from all 6 cameras.
Tl;Dr: It's not working so well.
Image and world coordinates are connected this way: ,
which can be written .
A singular value decomposition (SVD) gives
3 of the 4 points are correctly triangulated, but the blue one that should lie on the origin has a wrong x coordinate.
WHY?
Why only one point, and why only the x coordinate?
Does it have anything to do with the fact that I calibrate from a 4 points board?
If so, can you explain; and if not, what else could it be?
Update: I tried for an other frame while the board is somewhere else, and the triangulation is fine.
So there is the mystery: some points are randomly triangulated wrong (or at least the one at the origin), while most of the others are fine. Again, why?
My guess is that it comes from the triangulation rather than from the calibration, and that there is no connexion with my sloppy calibration process.
One common issue I came across is the ambiguity in the solutions found by DLT. Indeed, solving AQ = 0 or solving AC C-¹Q gives the same solution. See page 46 here. But I don't know what to do about it.
I'm now fairly sure this is not a calibration issue but I don't want to delete this part of my post.
I used ret, K, D, R, T = cv2.calibrateCamera(objpoints, imgpoints, imSize, None, None). It worked seamlessly, and the points where
perfectly reprojected on my original image with
cv2.projectPoints(objpoints, R, T, K, D).
I computed my projection matrix P as , and R, _ = cv2.Rodrigues(R)
How is it that I get a solution while I have only 4 points per image?
Wouldn't I need 6 of them at least? We have .We
can solve P by SVD under the form This is 2
equations per point, for 11 independent unknown P parameters. So 4
points make 8 equations, which shouldn't be enough. And yet
cv2.calibrateCamera still gives a solution. It must be using
another method? I came across Perspective-n-Point (PnP), is it what
opencv uses? In which case, is it directly optimizing K, R, and T and
thus needs less points?I could artificially add a few points
to get more than the 4 corner points of my board (for example, the
centers of the edges, or the center of the rectangle). But is it
really the issue?
When calibrating, one needs to decompose the projection matrix into
intrinsic and extrinsic matrices. But this decomposition is not
unique and has 4 solutions. See there section 'I'm seeing
double' and Chapt.21 of Hartley&Zisserman about Cheirality
for more information. It is not my issue since my camera points
are correctly reprojected to the image plane and my cameras are
correctly set up on my 3D scene.
I did not quite understand what you are asking, it is rather vague. However, I think you are miscalculating your projection matrix.
if I'm not mistaken, you will surely define 4 3D points representing your rectangle in real world space in this way for example:
pt_3D = [[ 0 0 0]
[ 0 1 0]
[ 1 1 0]
[ 1 0 0]]
you will then retrieve the corresponding 2D points (in order) of each image, and generate two vectors as follows:
objpoints = [pt_3D, pt_3D, ....] # N times
imgpoints = [pt_2D_img1, pt_3D_img2, ....] # N times ( N images )
You can then calibrate your camera and recover the camera poses as well as the projection matrices as follows:
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, imSize, None, None)
cv2.projectPoints(objpoints, rvecs, tvecs, K, dist)
for rvec, tvec in zip(rvecs, tvecs):
Rt, _ = cv2.Rodrigues(rvec)
R = Rt.T
T = - R # tvec
pose_Matrix = np.vstack(( np.hstack((R,T)) , [0, 0, 0, 1])) ( transformation matrix == camera pose )
Projection_Matrix = K # TransformationMatrix.T[:3, :4]
You don't have to apply the DLT or the triangulation (all is done in the cv2.calibrateCamera () function, and the 3D points remain what you define yourself
I've got an image from phone camera which have a paper inside it. In the image are also, some coordinates marked to get the distance between them. Since the aspect ratio of paper is known in advance (0.7072135785007072), I want to correct the distortion so that the whole image looks as if it's taken from the top view. I collect the four corners of the paper and apply opencv getPerspectiveTransform as follows:
pts1 = [[ 717., 664.],
[1112., 660.],
[1117., 1239.],
[ 730., 1238.]]
ratio=0.7072135785007072
cardH=math.sqrt((pts1[2][0]-pts1[1][0])*(pts1[2][0]-pts1[1][0])+(pts1[2][1]-pts1[1][1])*(pts1[2][1]-pts1[1][1]))
cardW=ratio*cardH;
pts2 = np.float32([[pts1[0][0],pts1[0][1]], [pts1[0][0]+cardW, pts1[0][1]], [pts1[0][0]+cardW, pts1[0][1]+cardH], [pts1[0][0], pts1[0][1]+cardH]])
M = cv2.getPerspectiveTransform(pts1,pts2)
with this matrix M I'm transforming the whole image as follows:
transformed = np.zeros((image.shape[1], image.shape[0]), dtype=np.uint8);
dst = cv2.warpPerspective(image, M, transformed.shape)
_ = cv2.rectangle(dst, (pts2[0][0], pts2[0][1]), (int(pts2[2][0]), int(pts2[2][1])), (0, 255, 0), 2)
The problem with this is that it's correcting the perspective of paper but distorting the overall image. I don't know why. The input image is this and the corresponding output image is this. In the input image point M and O and aligned horizontally, but to my surprise after transforming the overall image the point M and O are no longer aligned horizontally, why is that happening ?
Given a batch of images i have to find the images that fit together the best like in the example given below, but my solutions are not working:
Left image
Right image
I tried firstly with google cloud Vision API but it wasn't giving good results, then i trained a model over with ludwig but it will take forever to try all the possible combinations of images, as i have 2500 left images and 2500 right images.
is there a way to find this out or decrease the possible cases so that i can use it in my model.
This solution looks at a pair of images. The algorithm evaluates whether the shapes in the image will mesh like a key and a lock. My answer does not attempt to align the images.
The first step is to find the contours in the images:
left= cv2.imread('/home/stephen/Desktop/left.png')
right = cv2.imread('/home/stephen/Desktop/right.png')
# Resize
left = cv2.resize(left, (320,320))
gray = cv2.cvtColor(left, cv2.COLOR_BGR2GRAY)
_, left_contours, _ = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
# Approximate
left_contour = left_contours[0]
epsilon = 0.005*cv2.arcLength(left_contour,True)
left_contour = cv2.approxPolyDP(left_contour,epsilon,True)
What is a contour? A contour is just a list of points that lie on the perimeter of a shape. The contour for a triangle will have 3 points and a length of 3. The distance between the points will be the length of each leg in the triangle.
Similarly, the distances between the peaks and valleys will match in your images. To compute this distance, I found the distance between the contour points. Because of the way that the images are aligned I only used the horizontal distance.
left_dx = []
for point in range(len(left_contour)-1):
a = left_contour[point][0]
b = left_contour[point+1][0]
dist = a[0]-b[0]
left_dx.append(dist)
right_dx = []
for point in range(len(right_contour)-1):
a = right_contour[point][0]
b = right_contour[point+1][0]
# Use the - of the distance becuase this is the key hole, not the key
dist = -distance(a,b)
right_dx.append(dist)
# Reverse so they will fit
right_dx.reverse()
A this point you can sort of see that the contours line up. If you have better images, the contours will line up in this step. I used Scipy to iterpolate and check if the functions line up. If the two functions do line up, then the objects in the images will mesh.
left_x_values = []
for i in range(len(left_dx)): left_x_values.append(i)
x = np.array(left_x_values)
y = np.array(left_dx)
left_x_new = np.linspace(x.min(), x.max(),500)
f = interp1d(x, y, kind='quadratic')
left_y_smooth=f(left_x_new)
plt.plot (left_x_new, left_y_smooth,c = 'g')
I tried this again on a pair of images that I generated myself:
The contours:
The distances between contour points:
Fitting the contours:
I'm trying to verify my camera calibration, so I'd like to rectify the calibration images. I expect that this will involve using a call to warpPerspective but I do not see an obvious function that takes the camera matrix, and the rotation and translation vectors to generate the perspective matrix for this call.
Essentially I want to do the process described here (see especially the images towards the end) but starting with a known camera model and pose.
Is there a straightforward function call that takes the camera intrinsic and extrinsic parameters and computes the perspective matrix for use in warpPerspective?
I'll be calling warpPerspective after having called undistort on the image.
In principle, I could derive the solution by solving the system of equations defined at the top of the opencv camera calibration documentation after specifying the constraint Z=0, but I figure that there must be a canned routine that will allow me to orthorectify my test images.
In my seearches, I'm finding it hard to wade through all of the stereo calibration results -- I only have one camera, but want to rectify the image under the constraint that I'm only looking a a planar test pattern.
Actually there is no need to involve an orthographic camera. Here is how you can get the appropriate perspective transform.
If you calibrated the camera using cv::calibrateCamera, you obtained a camera matrix K a vector of lens distortion coefficients D for your camera and, for each image that you used, a rotation vector rvec (which you can convert to a 3x3 matrix R using cv::rodrigues, doc) and a translation vector T. Consider one of these images and the associated R and T. After you called cv::undistort using the distortion coefficients, the image will be like it was acquired by a camera of projection matrix K * [ R | T ].
Basically (as #DavidNilosek intuited), you want to cancel the rotation and get the image as if it was acquired by the projection matrix of form K * [ I | -C ] where C=-R.inv()*T is the camera position. For that, you have to apply the following transformation:
Hr = K * R.inv() * K.inv()
The only potential problem is that the warped image might go outside the visible part of the image plane. Hence, you can use an additional translation to solve that issue, as follows:
[ 1 0 | ]
Ht = [ 0 1 | -K*C/Cz ]
[ 0 0 | ]
where Cz is the component of C along the Oz axis.
Finally, with the definitions above, H = Ht * Hr is a rectifying perspective transform for the considered image.
This is a sketch of what I mean by "solving the system of equations" (in Python):
import cv2
import scipy # I use scipy by habit; numpy would be fine too
#rvec= the rotation vector
#tvec = the translation *emphasized text*matrix
#A = the camera intrinsic
def unit_vector(v):
return v/scipy.sqrt(scipy.sum(v*v))
(fx,fy)=(A[0,0], A[1,1])
Ainv=scipy.array( [ [1.0/fx, 0.0, -A[0,2]/fx],
[ 0.0, 1.0/fy, -A[1,2]/fy],
[ 0.0, 0.0, 1.0] ], dtype=scipy.float32 )
R=cv2.Rodrigues( rvec )
Rinv=scipy.transpose( R )
u=scipy.dot( Rinv, tvec ) # displacement between camera and world coordinate origin, in world coordinates
# corners of the image, for here hard coded
pixel_corners=[ scipy.array( c, dtype=scipy.float32 ) for c in [ (0+0.5,0+0.5,1), (0+0.5,640-0.5,1), (480-0.5,640-0.5,1), (480-0.5,0+0.5,1)] ]
scene_corners=[]
for c in pixel_corners:
lhat=scipy.dot( Rinv, scipy.dot( Ainv, c) ) #direction of the ray that the corner images, in world coordinates
s=u[2]/lhat[2]
# now we have the case that (s*lhat-u)[2]==0,
# i.e. s is how far along the line of sight that we need
# to move to get to the Z==0 plane.
g=s*lhat-u
scene_corners.append( (g[0], g[1]) )
# now we have: 4 pixel_corners (image coordinates), and 4 corresponding scene_coordinates
# can call cv2.getPerspectiveTransform on them and so on..
OpenCV users know that cvRemap is used for doing geometric transformations.
The mapx and mapy arguments are the data structures which give the mapping
information in the destination image.
Can I create two integer arrays holding random values from 1 to 1024 or from 1 to 768
if I deal with images (1024 X 768)
And then make mapx and mapy assigned with these values?
And then use them in cvRemap()?
Will it do the job or the only way to use mapx and mapy is get its value assigned by using the function cvUndistortMap()?
I want to know because I want to warp the images.
Just in case to tell you that I have already checked out Learning OpenCV book too.
I use cvRemap to apply distortion correction.
The map_x part is in image resolution and stores for each pixel the x-offset to be applied, while map_y part is the same for the y-offset.
in case of undistortion
# create map_x/map_y
self.map_x = cvCreateImage(cvGetSize(self.image), IPL_DEPTH_32F, 1)
self.map_y = cvCreateImage(cvGetSize(self.image), IPL_DEPTH_32F, 1)
# I know the camera intrisic already so create a distortion map out
# of it for each image pixel
# this defined where each pixel has to go so the image is no longer
# distorded
cvInitUndistortMap(self.intrinsic, self.distortion, self.map_x, self.map_y)
# later in the code:
# "image_raw" is the distorted image, i want to store the undistorted into
# "self.image"
cvRemap(image_raw, self.image, self.map_x, self.map_y)
Therefore: map_x/map_y are floating point values and in image resolution, like two images in 1024x768. What happens in cvRemap is basicly something like
orig_pixel = input_image[x,y]
new_x = map_x[x,y]
new_y = map_y[x,y]
output_image[new_x,new_y] = orig_pixel
What kind of geometric transformations do you want to do with this?