I'm fairly new to Computer Vision and OpenCV. I'm working on a project in which we are building a robot that plays snooker. We installed a camera on top of the table, with the aim of detecting the balls. I originally assumed that getting rid of the barrel distortion would be quite straight forward. However, I'm not very satisfied with the results I'm getting. In the second image I attached, one can clearly see, that after applying the undistortion transformation, the sides of the table are not parallel to each other. Moreover, with respect to the first image, the balls are deformed into a sort of egg shape.
Does anyone have an idea of how I could fix this issues? I tried to take more picture of the chessboard patter is as many different positions as possible, without any visible changes. Also using more parameters to model the distortion didn't seem to yield any improvements.
Distorted Image
Undistorted Image
Related
I'm currently working on a project where I need to be able to very reliable get the positions of the balls on a pool table.
I'm using a Kinect v2 above the table as the source.
Initial image looks like this (after converting it to 8-bit from 16-bit by throwing away pixels which is not around table level):
Then a I subtract a reference image with the empty table from the current image.
After thresholding and equalization it looks like this: image
It's fairly easy to detect the individual balls on a single image, the problem is that I have to do it constantly with 30fps.
Difficulties:
Low resolution image (512*424), a ball is around 4-5 pixel in diameter
Kinect depth image has a lot of noise from this distance (2 meters)
Balls look different on the depth image, for example the black ball is kind of inverted compared to the others
If they touch each other then they can become one blob on the image, if I try to separate them with depth thresholding (only using the top of the balls) then some of the balls can disappear from the image
It's really important that anything other than balls should not be detected e.g.: cue, hands etc...
My process which kind of works but not reliable enough:
16bit to 8bit by thresholding
Subtracting sample image with empty table
Cropping
Thresholding
Equalizing
Eroding
Dilating
Binary threshold
Contour finder
Some further algorithms on the output coordinates
The problem is that a pool cue or hand can be detected as a ball and also if two ball touches then it can cause issues. Also tried with hough circles but with even less success. (Works nicely if the Kinect is closer but then it cant cover the whole table)
Any clues would be much appreciated.
Expanding comments above:
I recommend improving the IRL setup as much as possible.
Most of the time it's easier to ensure a reliable setup than to try to "fix" that user computer vision before even getting to detecting/tracking anything.
My suggestions are:
Move the camera closer to the table. (the image you posted can be 117% bigger and still cover the pockets)
Align the camera to be perfectly perpendicular to the table (and ensure the sensor stand is sturdy and well fixed): it will be easier to process a perfect top down view than a slightly tilted view (which is what the depth gradient shows). (sure the data can be rotated, but why waste CPU cycles when you can simply keep the sensor straight)
With a more reliable setup you should be able to threshold based on depth.
You can possible threshold to the centre of balls since the information bellow is occluded anyway. The balls do not deform, so it the radius decreases fast the ball probably went in a pocket.
One you have a clear threshold image, you can findContours() and minEnclosingCircle(). Additionally you should contrain the result based on min and max radius values to avoid other objects that may be in the view (hands, pool cues, etc.). Also have a look at moments() and be sure to read Adrian's excellent Ball Tracking with OpenCV article
It's using Python, but you should be able to find OpenCV equivalent call for the language you use.
In terms tracking
If you use OpenCV 2.4 you should look into OpenCV 2.4's tracking algorithms (such as Lucas-Kanade).
If you already use OpenCV 3.0, it has it's own list of contributed tracking algorithms (such as TLD).
I recommend starting with Moments first: use the simplest and least computationally expensive setup initially and see how robuts the results are before going into the more complex algorithms (which will take to understand and get the parameters right to get expected results out of)
I am trying to implement an application that projects an image onto a page of a notebook, using OpenCV, a webcam and a projector. To achieve that, I am doing the following steps:
I am using a webcam to detect the four corners points of a page.
A homography is learned between the four corner points of the camera image and their projections on my desk, as seen in the camera. By using the inverse transformation, I will be able to know where I should draw something on my camera image, so that the projection "ends up" at a desired location.
I am applying the inverse transformation to the detected four corners points of the page.
I am warping the desired image to the new, transformed set of points.
So far it works well, if the notebook is on my desk and wide open. Like in this picture:
But if I try to close one side (or both), the following happens:
See the problem? In the first picture the image is perfectly aligned with the edges of the page and remains so if you rotate or translate the notebook, while keeping it on the desk. But that doesn't happen in the second image, where the the top edge of the image is no longer parallel to the top edge of the page (the image becomes more and more skewed).
Can anyone explain why I get this projection problem or at least point me to some resources where I can read about it? I need to mention that the projector and the webcam are placed above and to the left of the notebook, not right above them.
Any tips or suggestions are welcome. Thank you!
You want an effect that is called a key stone correction. The problem you are experiencing is most probably due to the fact that optical axes, positions, and focal lengths of a web camera and a projector are different. I suggest to calibrate your setup so you would know their relative pose and incorporate it in your inverse Homography.
I took the example of code for calibrating a camera and undistorting images from this book: shop.oreilly.com/product/9780596516130.do
As far as I understood the usual camera calibration methods of OpenCV work perfectly for "normal" cameras.
When it comes to Fisheye-Lenses though we have to use a vector of 8 calibration parameters instead of 5 and also the flag CV_CALIB_RATIONAL_MODEL in the method cvCalibrateCamera2.
At least, that's what it says in the OpenCV documentary
So, when I use this on an array of images like this (Sample images from OCamCalib) I get the following results using cvInitUndistortMap: abload.de/img/rastere4u2w.jpg
Since the resulting images are cut out of the whole undistorted image, I went ahead and used cvInitUndistortRectifyMap (like it's described here stackoverflow.com/questions/8837478/opencv-cvremap-cropping-image). So I got the following results: abload.de/img/rasterxisps.jpg
And now my question is: Why is not the whole image undistorted? In some pics of my later results you can recognize that the laptop for example is still totally distorted. How can I acomplish even better results using the standard OpenCV methods?
I'm new to stackoverflow and I'm new to OpenCV as well, so please excuse any of my shortcomings when it comes to expressing my problems.
All chessboard corners should be visible to be found. The algorithm expect a certain size of chessboard such as 4x3 or 7x6 (for example). The white border around a chess board should be visible too or dark squares may not be defined precisely.
You still have high distortions at the image periphery after undistort() since distortions are radial (that is they increase with the radius) and your found coefficients are wrong. The latter are wrong since a calibration process minimizes the sum of squared errors in pixel coordinates and you did not represent the periphery with enough samples.
TODO: You have to have 20-40 chess board pattern images if you use 8 distCoeff. Slant your boards at different angles, put them at different distances and spread them around, especially at the periphery. Remember, the success of calibration depends on sampling and also on seeing vanishing points clearly from your chess board (hence slanting and tilting).
I got 2 pictures which is almost parallel, and not positioned very far from each other.
I'm using OpenCV to try to create a disparity map (Stereo correspondence).
Because I'm trying to use it in a real world scenario, the use of chessboard calibration is a bit un-practical.
Because of that, I'm using stereoRectifyUncalibrated().
I tried to compare the results, using 2 different sets of corresponding points for the rectification:
Points manually selected(point & click)
Points generated from SURF and filtered with RANSAC
Input image1:
http://i.stack.imgur.com/2TCi3.jpg
Input image2:
http://i.stack.imgur.com/j1fFA.jpg
(Note that I do undistortion on the images before using them for rectification etc)
Rectified images with SURF and RANSAC:(1 and 2 in that order):
http://i.stack.imgur.com/pwbUw.jpg
http://i.stack.imgur.com/mb7TM.jpg
Rectified images using the manually selected points(which is more inaccurate!):
http://i.stack.imgur.com/Bodeg.jpg
Now, the thing is, looking at the result we see that the surf-version is almost perfectly rectified.(The epipolar lines are quite well aligned).
While the manually selected point version is quite badly rectified...the epipolar lines are nowhere near aligned.
But when we look at the result of openCV's sgBM() using both our rectifications:
Manual point result:
http://i.stack.imgur.com/N8Cyp.png
SURF point result:
http://i.stack.imgur.com/tGsCN.jpg
The disparity/depth shown is more accurate/correct with the SURF-point(well rectified version). No surprise there.
However, the actual detected object-pixels and object-boundaries are actually a lot better on the badly rectified verison.
For instance, you can see that the pen is actually a pen and has the shape of a pen, in the bad rectified disparity map, but not in the well-rectified map.
Question is, why?
And how can I fix it?
(I tried fooling around with the sgBM() paramteres, making sure they are the same for both etc, but it does not have any effect. It is the different rectifications alone that makes the badly rectified image look good for some reason(with respect to object-boundaries)).
I intend to make a program which will take stereo pair images, taken by a single camera, and then correct and crop them so that when the images are viewed side by side with the parallel or cross eye method, the best 3D effect will be achieved. The left image will be the reference image, the right image will be modified for corrections. I believe OpenCV will be the best software for these purposes. So far I believe the processing will occur something like this:
Correct for rotation between images.
Correct for y axis shift.
Doing so will I imagine result in irregular black borders above and below the right image so:
Crop both images to the same height to remove borders.
Compute stereo-correspondence/disparity
Compute optimal disparity
Correct images for optimal disparity
Okay, so that's my take on what needs doing and the order it occurs in, what I'm asking is, does that seem right, is there anything I've missed, anything in the wrong order etc. Also, which specific functions of OpenCV would I need to use for all the necessary steps to complete this project? Or is OpenCV not the way to go? Much thanks.
OpenCV is great for this.
There is a whole chapter in:
And all the sample code for this in the book ships with the opencv distribution
edit: Roughly the steps are:
Remap each image to remove lens distortions and rotate/translate views to image center.
Crop pixels that don't appear in both views (optional)
Find matching objects in each view (stereoblock matching) create disparity map
Reproject disparity map into 3D model