DISCLAIMER: Apologies for this very large question, as it could take a lot of your time.
I have a stereo setup consisting of two webcams these cameras possess auto-focus technology. The stereo setup is in canonical configuration where the cameras are separated by 10 cm distance.
I am using the stereo_calib.cpp program for stereo calibration, provided by the OpenCV sample programs.
Initially, I have captured the chessboard images by my stereo rig as the way it is shown in the sample cpp stereo images, and then tried to calibrate the setup, but the stereo rectification was either completely blank or the undistorted left and right images are tilted about 40 degrees.
As this was the case, then I have captured a set of 17 stereo pair chessboard images by keeping the Z distance constant, without any rotation, at this point the stereo images were correctly rectified in the process of stereo calibration. This Working Set contains the images of chessboard taken by the stereo setup along with the program and the image of how well the rectification has been achieved.
Later, when I was trying to calibrate the stereo setup again (as the cameras in the stereo setup was disturbed), with another new set of images of the chessboard taken by my stereo rig, the program is unable to rectify the stereo images. I am providing the Non Working Set where you can check the images taken by the stereo setup along with the images of the rectification.
As a picture is worth a thousand words.
You gotta see the output images of the provided program, which let you know much more than what I could say in my own words.
I am trying to find some new ways of stereo face recognition techniques.
Any help in this regard is highly appreciated.
And adding to this, I also need some existing techniques by which I could kick start my experimentation on new ways of face recognition using the stereo information.
Related
I have a vehicle with two cameras, left and right. Is there a difference between me calibrating each camera separately vs me performing "stereo calibration" ? I am asking because I noticed in the OpenCV documentation that there is a stereoCalibrate function, and also a stereo calibration tool for MATLAB. If I do separate camera calibration on each and then perform a depth calculation using the undistorted images of each camera, will the results be the same ?
I am not sure what the difference is between the two methods. I performed normal camera calibration for each camera separately.
For intrinsics, it doesn't matter. The added information ("pair of cameras") might make the calibration a little better though.
Stereo calibration gives you the extrinsics, i.e. transformation matrices between cameras. That's for... stereo vision. If you don't perform stereo calibration, you would lack the extrinsics, and then you can't do any depth estimation at all, because that requires the extrinsics.
TL;DR
You need stereo calibration if you want 3D points.
Long answer
There is a huge difference between single and stereo camera calibration.
The output of single camera calibration are intrinsic parameters only (i.e. the 3x3 camera matrix and a number of distortion coefficients, depending on the model used). In OpenCV this is accomplished by cv2.calibrateCamera. You may check my custom library that helps reducing the boilerplate.
When you do stereo calibration, its output is given by the intrinsics of both cameras and the extrinsic parameters.
In OpenCV this is done with cv2.stereoCalibrate. OpenCV fixes the world origin in the first camera and then you get a rotation matrix R and translation vector t to go from the first camera (origin) to the second one.
So, why do we need extrinsics? If you are using a stereo system for 3D scanning then you need those (and the intrinsics) to do triangulation, so to obtain 3D points in the space: if you know the projection of a general point p in the space on both cameras, then you can calculate its position.
To add something to what #Christoph correctly answered before, the intrinsics should be almost the same, however, cv2.stereoCalibrate may improve the calculation of the intrinsics if the flag CALIB_FIX_INTRINSIC is not set. This happens because the system composed by two cameras and the calibration board is solved as a whole by numerical optimization.
I have a stationary mono camera which captures a single image frame at some fps.
Assume the camera is not allowed to move,how do I generate a stereo image pair from the obtained single image frame? Is there any algorithms exists for this? If so, are they available in Open-CV?
To get a stereo image, you need a stereo camera, i.e. a camera with two calibrated lenses. So you cannot get a stereo image from a single camera with traditional techniques.
However, with the magic of deep learning, you can obtain the depth image from single camera.
And no, there's no builtin OpenCV function to do that.
The most common use of this kind of techniques is in 3D TVs, which often offer 2D-to-3D conversion, and thus mono to stereo conversion.
Various algorithms are used for this, you can look at this state of the art report.
There is also optical way for this.
If you can add binocular prisms/mirrors to your camera objective ... then you could obtain real stereoscopic image from single camera. That of coarse need access to the camera and setting up the optics. This also introduce some problems like wrong auto-focusing , need for image calibration, etc.
You can also merge Red/Cyan filtered images together to maintain the camera full resolution.
Here is a publication which might be helpful Stereo Panorama with a single Camera.
You might also want to have a look at the opencv camera calibration module and a look at this page.
What is the minimum number of chessboard image pairs in order to mathematically calibrate and rectify two cameras ? One pair is considered as a single view of the chessboard by each camera, ending with a left and right image of the same scene. As far as I know we need just one pair for a stereo system, as the stereo calibration seeks the relations between the tow cameras.
Stereo calibration seeks not only the rotation and translation between the two cameras, but also the intrinsic and distortion parameters of each camera. You need at least two images to calibrate each camera separately, just to get the intrinsics. If you have already calibrated each camera separately, then, yes, you can use a single pair of checkerboard images to get R and t. However, you will not get a very good accuracy.
As a rule of thumb, you need 10-20 image pairs. You need enough images to cover the field of view, and to have a good distribution of 3D orientations of the board.
To calibrate a stereo pair of cameras, you first calibrate the two cameras separately, and then you do another joint optimization of the parameters of both cameras plus the rotation and translation between them. So one pair of images will simply not work.
Edit:
The camera calibration algorithm used in OpenCV, Caltech Calibration Toolbox, and the Computer Vision System Toolbox for MATLAB is based on the work by Zhengyou Zhang. His paper explains it better than I ever could.
The crux of the issue here is that the points on the chessboard are co-planar, which is a degenerate configuration. You simply cannot solve for the intrinsics using just one view of a planar board. You need more than one view, with the board in different 3-D orientations. Views where the boards are in parallel planes do not add any information.
"One image with 3 corners give us 6 pieces of information can be used to solve both intrinsic and distortion. "
I think that this is your main error. These corners are not independent. A pattern with a 100x100 chessboard pattern does not provide more information than a 10x10 pattern in your perfect world as the points are on the same plane.
If you have a single view of a chessboard, a closer distance to the board can be compensated by the focus so that you are not (even in your perfect world) able to calibrate your camera's intrinsic AND extrinsic parameters.
I'm currently implementing the stereovision with OpenCV. Now I'm using the Stereo_Calib sample to remove the distortion en rectify the image. Removing the distortion works fine.
But when I apply rectification, the image is very warped.
This is the code to rectify the images. The parameters rmap are calculated in the same way as in the Stereo_calib example (see here)
void StereoCalibration::StereoRectify(Mat &imageLeft, Mat &imageRight)
{
Mat imLeft, imRight;
remap(imageLeft, imLeft,DistLeft.rmap[0], DistLeft.rmap[1], CV_INTER_CUBIC);
remap(imageRight,imRight, DistRight.rmap[0], DistRight.rmap[1], CV_INTER_CUBIC);
imageLeft = imLeft;
imageRight = imRight;
}
I realise this question is a few years old however, I have recently had a similar issue. Building on morynicz answer about "bad chessboard" patterns to calibrate stereo images, I found that even with a slight deformation in your chessboard pattern, for example that it isn't flat, can produce large warping in the stereo image pair on rectification. The algorithms in OpenCV, for instance, assume a flat chessboard pattern is being presented such that any physical deformation in that pattern will be wrongly attributed to distortions in the camera optics (or in the relative orientations of the two camera sensors). The algorithms will then try really hard to remove this false distortion leading to very warped images.
To avoid this problem, were possible, use a tablet (or other electronic screen) to display the chessboard pattern as it is then guaranteed to be flat.
Additionally, you should check that the images you are using to calibrate the stereo pair are in focus and have no motion blur or image tearing.
If using OpenCV to do the rectification do some experimentation with the flags used in the stereoCalibrate function as this may lead to a more "optimised" rectification for your particular application.
For anyone looking for help on this, I was dealing with very large scale resolution images and was getting very low reprojection error rate with good calibration images. I was getting very warped stereo pairs after rectification and a really bad depth map.
One thing to try is if your images are warped you might need to down-sample them.
Another thing to try is to combine the flags in stereoCalibrate instead of just choosing one.
Something like this worked for me :
cv2.stereoCalibrate(
object_points, image_points_left,image_points_right,
camera_matrix_left,dist_left,
camera_matrix_right, dist_right,
(5472,3648),None,None,None,None,
cv2.CALIB_FIX_ASPECT_RATIO + cv2.CALIB_ZERO_TANGENT_DIST + cv2.CALIB_USE_INTRINSIC_GUESS + cv2.CALIB_SAME_FOCAL_LENGTH + cv2.CALIB_RATIONAL_MODEL,criteria
)
I had the same problem, and I think that the issue was bad chessboard used to calibration or mixing up the maps.
I started working on opencv stereo image calibration and rectification recently and I was getting similar images. Although it is true to make sure the board is straight and it is true that we need to take multiple images on the corners and in the middle of the camera at different x,y,z and skew positions, what did the trick for me was the flags in stereoCalibrate. I used all the flags specified in the opencv docs except for INTRINSIC_GUESS and it started very nice undistorted and rectified images.
Say I plan to use OpenCV for 3D reconstruction using a stereo approach...and I do not have any special stereo camera but only webcams.
1.)How do I build a cheap stereo setup using a set of web cams?
2.)Is it possible to snap two images using web cams and convert them to stereo using openCV API?
I will use the stereo algorithm from the link below
Stereo vision with OpenCV
Using this approach I want to create a detailed mapping of an indoor environment.
(I would not like to use any projects like Insight3D which cannot be used for commercial purposes without distributing the source code)
You can find here a lot of resources including tutorials and stereo vision cameras
Firstly, ensure that your web cams don't have any inbuilt autofocus technology. As the cameras should have fixed focal length.
1) Align the cameras in canonical configuration with varying baseline distance. Then calibrate them using opencv's stereo_calib.cpp program. Usually, the distance will be 20-60cms. For some web cameras even 10cm will give you better results. If rms error and reprojection error are less than 0.5 then you could consider that the stereo setup is ready.
2) Yes, it is possible to capture stereo images from the setup which I just mentioned. Check out this link for capturing images from cameras.
OpenCV provides better algorithms from which one can do wonders with 3D vision.
Stereo is better suited for indoor environment as it is very sensitive to lighting variations.