I have images that contain transparent rectangular overlay similar to the following images: Image1 Image2. My goal is to detect if these rectangular boxes exist (location doesn't matter). These boxes will always have edges parallel to the sides of the image.
Assumptions:
The transfer function of how the transparent rectangles are drawn is now known
The sides of the rectangles will also be parallel to the image
Attempted Solution 1: Color Detection
So far, I've tried color detection via cv.threshold as well as using band-pass filters with the cv2.inRange() on multiple color spaces (HSV, LUV, XYZ etc). The issue with color detection is that I am also capturing too much noise to effectively just tune for the pixels for the transparent area. I tried laying the masks using cv2.bitwiseAnd but still can't tune the noise down to a negligible state. I tried only isolating for large groups of pixels using morphological transformations but this still fails.
Attempted Solution 2: Edge Detection + Edge Validation
My second try at detecting the box involved applying cv2.bilateralFilter and then generating hough lines via cv2.Canny,cv2.HoughLinesP. Although I am detecting a significant number of edges related to the transparent box, I also get many miscellaneous edges.
To filter out false edges, I take each line segment and check a few sample pixels to the left and right sides. By applying something something similar to what I believe the transfer function is (cv2.addWeighted) I checked to see if if I can reproduce the similar values. Unfortunately, this also doesn't work well enough to tell the difference between edges from the transparent box vs "real edges." Result From Edge Detection
Any thoughts on how I might detect these boxes is highly appreciated!
Related
I am working on accurately segmenting objects from an image.
I have found contour lines by using a simple rectangular prism in HSV space as a color filter (followed by some morphological operations on the resulting mask to clear up noise). I found this approach to be better than applying canny edge detection to the whole image as that just picked up a lot of other edges I don't care about.
Is there a way to go about refining the contour line I have extracted such that it clips to the strongest local edge kind of like Adobe Photoshop's smart cropping utility?
Here's an image of what I mean
You can see a boundary between the sky blue and the gray. The dark blue is a drawn on contour. I'd like to somehow clip this to the nearby edge. It also looks like there are other lines in the grey region, so I think the algorithm should do some sort of more globalish optimisation to ensure that the "clipping" action doesn't jump randomly between my boundary of interest and the nearby lines.
Here are some ideas to try:
Morphological snakes: https://scikit-image.org/docs/dev/auto_examples/segmentation/plot_morphsnakes.html
Active contours: https://scikit-image.org/docs/dev/auto_examples/edges/plot_active_contours.html
Whatever livewire is doing under the hood: https://github.com/PyIFT/livewire-gui
Based on this comment, the last one is the most useful.
I hame some image with plane which have perspective transform.
I need to detect center of each white rhombus or rhombus itself.
Here is examples:
As I unserstand the problem can be solved by simple template matching if we rectify image, but I need to do it automatically.
Is there any functions in OpenCV suitable for this task? Any other ideas?
Here are two quick tests I just did without correcting the perspective issue.
Pure mathematical morphology:
Extract the red channel
Big white top-hat in order to detect all the bright areas, but without the big bright reflexion.
Small white top-hat in order to detect only the thin lines between the rhombus
Result of 2 minus result of 3. The lines between the rhombus are then thinner or even disappeared.
Opening to clean the final result.
Here are two results: Image1 and Image2. The main issue is that the rhombus do not have the same sizes (different magnification and perspective), which can be problematic with the mathematical morphology.
So here is an other solution using the Hough transform:
You start with the resulting image of the step 3 from the previous algorithm.
You apply a hough transform.
Here are the results: Hough1 and Hough2. Then you have to filter between lines touching a rhombus or not, but you can use my first algorithm for that. Even if all the rhombus are not detected by the first algorithm, most will be and it will be enough to detect the lines touching the Rhombus. Then the line intersections will be the centroids that your are looking for.
When I'm running Grabcut on an image,
I set the bounding box of grabcut to the edges of the image itself.
However, since the bottom of the image is a part of the foreground not background, it cuts out the lower part.
Is there any way of preventing this, such as setting the boundary box only to the top, left and right?
GrabCut needs the boundary to define what is "outside" so it can compute a background color model. Depending on your API/interface you might be able to define "outside" only from the right-top and left parts of the image, leaving the bottom "inside".
Assuming you are using cv::grabCut, you may define the initial rect to exceed the bottom part of the image, in that case the algorithm should not consider the lower part to be "obvious background".
As the background in this image is quite clean, you may try some rules to segment the foreground in this case. For examples, why don't you remove the “relatively” white regions. The color of background can be heuristically extracted from the boundary region of images.
If there is cluttered background in your data set, you may try to detect a tighter bounding box for the human by using detectors, such as DPM, or R-CNN. They already provide some powerful models for human detection. Based on the detected box, I suppose the result of grabcut should be better.
Anyway, it should be helpful to provide some more examples for analysis.
classic gab cut implemented in openCV has 6 Gaussian mixture models corresponding to 6 centroids in 3D color space - more than you need to model the image above. Thus your problem is in setting correct labels. You have to set not only bounding box or ROI but 3 regions with 3 labels - FG, BG, PROBABLY_BG (partition the image). The former two will contribute to the models of foreground and background on the basis of which the latter label will be the only region where the boundary is refined. In other words, you did not do correct initialization of grab cut with three labels. See this, for example How to set a mask image for grabCut in OpenCV?
The objective is to display the person on a different background (aka background removal).
I'm using the Kinect with Microsoft's Beta Kinect SDK to do so. With help of the depth, the background is filtered and we get only the image of the person.
This is pretty simple to do, and we can find the code that does that everywhere on the Internet. However, the depth signal is noisy, and we get pixels which do not belong to the person that are displayed.
I applied an edge detector to see if it was useful, and I currently get this:
Here's another without edge detection:
My question is: Which way can I get rid of these noisy white pixels around the person?
I tried morphological operations, but some parts of the body are erased and still leave white pixels behind.
The algorithm doesn't need to be real-time, I can just apply it when I press a 'Save image' button.
Edit 1:
I just tried to do background substraction with the closest frames on the shape border. The single pixels you see are flickering, which means it is noise and I can get easily get rid of them.
Edit 2:
The project is now over, and here's what we did: manual calibration of the Kinect by using the OpenNI driver, which provides directly the infrared image. The result is really good, but each calibration is specific to each Kinect.
Then, we applied a little transparency on the borders, and the result looks really nice! I can't provide pictures, however.
Your problem isn't just the noisy white pixels. You're missing significant parts of the person as well, e.g. part of his right hand. I'd recommend being more conservative with your thresholding of the depth data (allow more false positives). This would give you more noisy pixels, but at least you'd have the person in their entirety.
To get rid of the noisy pixels, I can think of a couple of things:
Feather the outer pixels (reduce them in intensity/increase their transparency if you're using an alpha channel)
Smooth the image, perform the edge detection on the smoothed image, then use these edges with your original sharp image.
Do some skin region detection to mark parts that definitely belong to a person. See skin detection in the YUV color space? and Skin Color Detection
For clothes, work with the hue and saturation image. If you know the color of the t-shirt (or that at least that it's not a neutral color), then this will stand out easily. If you don't know this information, then it may be worth building up a model of the person using the other frames (if there's a big gray blob that's moving around in your video, chances are that your subject is wearing a gray shirt)
The approaches aren't mutually exclusive so it may be worth trying to do them in combination. If I think of anything else, I'll post back here.
If there is no other way of resolving the jitter on the edges you could always try anti-alias as post-process.
I uploaded an example image for better understanding: http://www.imagebanana.com/view/kaja46ko/test.jpg
In the image you can see some scanlines and a marker (the white retangle with the circle in it). I want OpenCV to go along a specified area (in the example outlined trough the scanlines) that should be around 5x5. If that area contains a gradient from black to white, I want OpenCV to save the position of that area, so that I can work with it later.
The final result would be to differentiate between the marker and the other retangles separated trough black and white lines.
Is something like that possible? I googled a lot but I only found edge detectors but that's not what I want, I really need the detection of the black to white gradient only.
Thanks in advance.
it would be a good idea to filter out some of the areas by calculating their histogram.
You can use cvCalcHist for the task, then you can establish some threshold to determine if the black-white pixels percentage corresponds to that of a gradient. This will not solve the task but it will help you in reducing complexity.
Then, you can erode the image to merge all the white areas. After applying threshold, it would be possible to find connected components (using cvFindContours) that will separate images in black zones or white zones. You can then detect gradients by finding 5x5 areas that contain both a piece of a white zone and black zone simultaneously.
hope it helps.
Thanks for your answerer dnul, but it didn't really help me work this out. I though about a histogram to approach the problem but it's not quite what I want.
I solved this problem by creating a 40x40 matrix which holds 5x5 matrix's containing the raw pixel data in all 3 channels. I iterated trough each 40px-area and inside iterated trough each border of 5px-area. I checked each pixel and saved the ones which are darker then a certain threshold a storage.
After the iteration I had a rough idea of how many black pixels their are, so I checked each one of them for neighbors with white-pixels in all 3 channels. I then marked each of those pixels and saved them to another storage.
I then used the ransac algorithm to construct lines out of these points. It constructs about 5-20 lines per marker edge. I then looked at the lines which meet each other and saved the position of those that meet in a square angle.
The 4 points I get from that are the edges of the marker.
If you want to reproduce this you would have to filter the image in beforehand and apply a threshold to make it easier to distinguish between black and white pixels.
A sample picture, save after finding the points and before constructing the lines:
http://www.imagebanana.com/view/i6gfe6qi/9.jpg
What you are describing is edge detection. This is exactly how, say, the Canny edge detector works. It looks for dark pixels near light pixels, and based on a threshold that you pass in (There is also the adaptive canny, which figures out the threshold for you), and sets them to all black or all white (aka 'marks' them).
See here:
http://docs.opencv.org/doc/tutorials/imgproc/imgtrans/canny_detector/canny_detector.html