EDIT: To further clarify if my question is not clear,
Input: The image below
Output: The points on edge 1, the points on edge 2, the points on edge 3, and the points on edge 4. (I do not have a problem finding contours. I am just unable to separate the points that lie on each of the four edges. I want to group those points into four separate edges so that I can fit four separate curves to them)
My problem here is to detect points and fit separate curves to each of the curved edges of objects like what is shown below (The image shown is one example. The actual shape of each object is different, but there will be either a sharp corner or change in slope from one edge to another):
One way to approach this is to separate out the points/pixels on each edge (the four lines in the above example) and fit polynomials on each of them. By searching a little bit, I learnt that Hough Transform is available for detecting straight edges in OpenCV, but not for curved edges. I also tried detecting contours, but it does not separate out edges of a closed shape. The main criterion for an edge to be considered separate from an adjacent one is that there is a sharp change in slope.
Could anyone give me ideas on how to achieve this? I prefer using C++ with OpenCV due to the other modules of my task.
What you are trying to do is essentially to find high curvature points in the outline. There are several methods for curvature estimation. Some are based on local derivatives of the intensity, and some are based on the arrangement of the pixels along the curve. This problem is very close to that of corner detection.
You may be interested by the following references: "A Comparative Study
on 2D Curvature Estimators, Simon Hermann and Reinhard Klette" or "Curvature estimation in noisy curves, Thanh Phuong Nguyen, Isabelle Debled-Rennesson". Notice that there is large litterature on the topic as curvature estimation in the digital domain is uneasy because it takes second order derivatives.
Related
I'm looking for an efficient way of selecting a relatively large portion of points (2D Euclidian graph) that are the furthest away from the center. This resembles the convex hull, but would include (many) more points. Further criteria:
The number of points in the selection / set ("K") must be within a specified range. Most likely it won't be very narrow, but it most work for different ranges (eg. 0.01*N < K < 0.05*N as well as 0.1*N < K < 0.2*N).
The algorithm must be able to balance distance from the center and "local density". If there are dense areas near the upper part of the graph range, but sparse areas near the lower part, then the algorithm must make sure to select some points from the lower part even if they are closer to the center than the points in the upper region. (See example below)
Bonus: rather than simple distance from center, taking into account distance to a specific point (or both a point and the center) would be perfect.
My attempts so far have focused on using "pigeon holing" (divide graph into CxR boxes, assign points to boxes based on coordinates) and selecting "outer" boxes until we have sufficient points in the set. However, I haven't been successful at balancing the selection (dense regions over-selected because of fixed box size) nor at using a selected point as reference instead of (only) the center.
I've (poorly) drawn an Example: The red dots are the points, the green shape is an example of what I want (outside the green = selected). For sparse regions, the bounding shape comes closer to the center to find suitable points (but doesn't necessarily find any, if they're too close to the center). The yellow box is an example of what my Pigeon Holing based algorithms does. Even when trying to adjust for sparser regions, it doesn't manage well.
Any and all ideas are welcome!
I don't think there are any standard algorithms that will give you what you want. You're going to have to get creative. Assuming your points are embedded in 2D Euclidean space here are some ideas:
Iteratively compute several convex hulls. For example, compute the convex hull, keep the points that are part of the convex hull, then compute another convex hull ignoring the points from the original convex hull. Continue to do this until you have a sufficient number of points, essentially plucking off points on the perimeter for each iteration. The only problem with this approach is that it will not work well for concavities in your data set (e.g., the one on the bottom of your sample you posted).
Fit a Gaussian to your data and keep everything > N standard
deviations away from the mean (where N is a value that you'd have to
choose). This should work pretty well if your data is Gaussian. If
it isn't, you could always model it with several Gaussians (instead
of one), and keep points with a joint probability less than some threshold. Using multiple Gaussians will probably handle concavities decently. References:
http://en.wikipedia.org/wiki/Gaussian_function
How to fit a gaussian to data in matlab/octave?\
Use Kernel Density Estimation - If you create a kernel density
surface, you could slice the surface at some height (e.g., turning
it into a plateau), giving you a perimeter shape (the shape of the
plateau) around the points. The trick would be to slice it at the
right location though, because you could end up getting no points
outside of the shape, but with the right selection you could easily
get the green shape you drew. This approach will work well and give you the green shape in your example if you choose the slice point wisely (which may be difficult to do). The big drawback of this approach is that it is very computationally expensive. More information:
http://en.wikipedia.org/wiki/Multivariate_kernel_density_estimation
Use alpha shapes to get a general shape the wraps tightly around
the outside perimeter of the point set. Then erode the shape a
little to force some points outside of the shape. I don't have a lot of experience with alpha shapes, but this approach will also be quite computationally expensive. More info:
http://doc.cgal.org/latest/Alpha_shapes_2/index.html
Could you suggest an approach for color-based segmentation for square or triangular shapes? I'm working on an iOS app for recognizing road signs and have implemented it for round signs but that approach doesn't seem to work with other forms. For the circles we do the following:
Detect the colors we need, e.g. red and white, through HSV/B.
Detect circle through the method called Fast Circle Detection Using Gradient Pair Vectors based on analysis of gradient direction vectors (description and code: http://rnd.azoft.com/applied-use-of-m2m-tchnology-in-ios-apps/)
Triangles and squares demand differed approach and we've stuck a bit.
Assuming you're looking for red lines...
Threshold just the red component of the image
Compute hough lines and look for line segments of an estimated length (if you know the length of the sides of the triangle/square you're looking for).
Once you have this list, find combinations of lines that form triangles and squares.
Verify each candidate triangle/square by checking that their areas are within expected ranges.
If you follow this method, it is likely that you will find multiple shapes within close proximity of each other i.e. the same triangle/square in the real world will be found multiple times by the algorithm depending on the thickness of the lines. In this case, cluster them by distance and only retain one shape per cluster.
Another option is
Threshold the red component of the image.
Find contours.
Check for closed contours.
For every close contour, check if the shape resembles an equilateral triangle or a square by plotting histograms of slopes of individual points on the contour. The histogram for a square will have two highly populated bins, while that of a triangle will have three highly populated bins.
I have studied on a school project for road sign detection and for our segmentation part, we really benefited from this paper.
http://vc.cs.nthu.edu.tw/home/paper/codfiles/cmwang/201201100409/110104%20Goal%20evaluation%20of%20segmentation%20algorithms%20for%20traffic%20sign%20recognition.pdf
It compares performance of many color based segmentation techniques and some non-color based approaches. Tests compared with different signs.
Unlike some survey paper in this area it explains threshold values for different methods.
Good luck.
Using GPUImage, I am able to detect corners of a book/page in an image. But sometimes, it will pass more than 4 points, in which case I will need to process and figure out the best rectangle out of these points. Here's an example:
What's the most efficient way to figure out the best rectangle in this case?
Thanks
If you're using a corner detection algorithm, then you can filter results based on the relative strength of the detected corner. The contrast at the book corners relative to your current background appears to be much stronger than the contrast at the point found in the wood grain. Are there relative magnitudes associated with each point, or do you just get the points? Setting thresholds for edge strengths can mean a lot of fiddling unless the intensities of the foreground and background are relatively constant.
Your sample image could be blurred or morphed. For example, the right morphological "close" on light pixels could eliminate the texture in the wood grain without having an effect on the size and shape of the book. (http://en.wikipedia.org/wiki/Mathematical_morphology)
Another possibility is to shrink the image to a much smaller size and then perform detection on that. Resizing the image will tend to wipe out tiny details such as whatever wood grain pattern is currently being detected.
Picking the right lens and lighting can make the image easier to process. Try to simplify the image as much as possible before processing it. As mentioned above, "dark field" lighting that would illuminate just the book edges would present a much simpler image for processing. Writing down the constraints can make it more obvious which solution will be most robust and simplest to implement. Finding any rectangle anywhere in an image is very difficult; it's much easier to find a light rectangle on a dark background if the rectangle is at least 100 x 100 pixels in size, rotated no more than 15 degrees from square to the image edges, etc.
More involved solutions can be split into two approaches:
Solving the problem using given only 4 or more (x,y) points.
Using a different image processing technique altogether for the sample image.
1. Solving the program given only the points
If you generally only have 5 or 6 points, and if you are confident that 4 of those points will belong to the corners of the rectangles that you want, then you can try this:
Find the convex hull of all points. The convex hull is the N-gon that completely encompasses all points. If the points were pegs sticking up, and if you stretched a rubber band around them and let it snap into place, then the final shape of the rubber band is a convex hull. Algorithms that find convex hulls typically return a list of points that ordered counterclockwise from the bottom leftmost point.
Make a copy of your point list and remove points from the copy until only four points remain. These four remaining points will still be ordered counterclockwise.
Calculate the angle formed by each set of three successive points: points 1, 2, 3, then 2, 3, 4, then 3, 4, 1, and so on.
If an angle is outside a reasonable tolerance--less than 70 degrees or greater than 110 degrees--skip back to step 2 and remove the next point (or set of points).
Store the min and max angles for each set of 4 points.
Repeat steps 2 - 6, removing a different point (or points) each time.
Track the set of points for which the min and max angles are closest to 90 degrees.
http://en.wikipedia.org/wiki/Convex_hull
There are a number of other checks and constraints that could be introduced. For example, if the point-to-point distances for 3 successive points in the convex hull (pts N to N+1, and N+1 to N+2) are close to the expected width and height of the book, then you might mark these as known good points and only test the remaining points to see which is the fourth point.
The technique above can get unwieldy if you get quite a few points, but it may work if two or three of the book corner points are expected to be found on the convex hull.
For any geometric problem, I always recommend checking out GeometricTools.com, which has a lot of great, optimized source code for all sorts of problems. It's very handy to have the book as well, especially if you can find a cheap copy using AddAll.com.
http://www.geometrictools.com/
2. Other image processing techniques for your sample image
Although I could be wrong, it appears that GPUImage doesn't have many general-purpose image processing algorithms. Some other image processing algorithms could make this problem much simpler to solve.
Though there isn't space to go into it here, one of the keys to successful image processing is appropriate lighting. Make sure you're lighting is consistent. A diffuse light that evenly illuminates the book and the background would work well. You can simplify the problem using funkier lighting: if you have four lights (or a special ring light), you can provide horizontal illumination from the top, bottom, left, and right that will cause the edges of the book to appear bright and other surfaces to appear dark.
http://www.benderassoc.com/mic/lighting/nerlite/Darkfield.htm
If you can use some other GPU libraries to do image processing, then one of the following techniques could work nicely:
Connected component labeling (a.k.a. finding blobs). It shouldn't be too hard to use either binary thresholding or a watershed algorithm to separate the white blob that is the book from the rest of the background. Once the blob for the book is identified, finding the corners is easier. (http://en.wikipedia.org/wiki/Connected-component_labeling) In OpenCV you can find the "contours."
Generate an list of edge points, then have four separate line-fitting tools search from top to bottom, right to left, bottom to top, and left to right to find the four strong (and mostly straight) edges associated with the book. In your sample image, though, either the book cover is slightly warped or the camera lens has introduced barrel distortion.
Use a corner detector designed to find light corners on a dark background. If you will always be looking for a white book on a wood grain background, you can create a detector to find white corners on a brown background.
Use a Hough technique to find the four strongest lines in the image. (http://en.wikipedia.org/wiki/Hough_transform)
The algorithmic technique that works best will depend on your constraints: are you looking for rectangles only of a certain size? is the contrast between foreground and background consistent? can you introduce lighting to simplify the appearance of the image? and so on.
I am currently working in SIFT, I had generated the difference of Gaussian and the extrema image layers. Can anyone explain to me how to use Hessian matrix to eliminate the low contrast keypoint?
A good keypoint is a corner. This comes from the Harris corner work and the Good features to track (KLT) papers first, then emphasized by the Mikolajczyk and Schmid paper.
Intuitively, a corner is a good feature because it is an intersection of two lines, while a single line segment can be moved along its direction, thus causing a less accurate localization.
A line segment is an edge, i.e., a first order derivative (gradient). A corner is an edge that changes its direction abruptly. This is measured by a second order derivative, hence the use of the Hessian matrix that contains the values of the directional second derivatives.
I'm trying to implement user-assisted edge detection using OpenCV.
Assume you have an image in which we need to find a polygonal shape. For the sake of discussion, let's say we need to find the top of a rectangular table in a picture. The user will click on the four corners of the table to help us narrow things down. Connecting those four points gives us a polygon, or four vectors.
But the user is not very accurate when clicking on those corners. So I'd like to use edge information from the image to increase the accuracy.
I'm using a Canny edge detector with a fairly high treshold to determine important edges in my image. (more precisely, I'm scaling down, blurring, converting to grayscale, then run Canny). How can I compute whether a vector aligns with an edge in my image? If I have a way to compute "alignment", my overal algorithm comes down to perturbating the location of the four edge points, computing the total "alignment" of my polygon with the edges in the image, until I find an optimum.
What is a good way to define and compute this "alignment" metric?
You may want to try to use FindContours to detect your table or any other contour. Then build a contour also from the user input points. After this you can read about Contour Moments by which you can compare contours. You can compare all the contours from the image with the one built from the user points and then select the closest match.