I'm using ArUco markers to correct perspective and calculate sizes in an image. In this image I know the exact distance between the outer edges of the markers and am using that to calculate the sizes of the black rectangles.
My problem is that aruco::detectMarkers doesn't always identify the true edges of the markers (as shown in the detail image). When I correct the perspective based on the corners of the markers, it causes distortion that affects the size calculations of the objects in the image.
Is there a way to improve the edge detection accuracy of aruco::detectMarkers?
Here's a scaled-down photo of the entire board:
Here's the detail of the lower-left marker showing the inaccuracy of the edge detection:
Here's the detail of the upper-right marker showing an accurate edge detection of the same marker ID:
It's hard to see in this shrunken image but the upper-left marker is accurate and the lower-right marker is inaccurate.
My function that calls detectMarkers:
bool findMarkers(const Mat image, Point2d outerMarkerCoordinates[], Point2d innerMarkerCoordinates[], Size2d *boardSize) {
Ptr<aruco::Dictionary> theDictionary = aruco::getPredefinedDictionary(aruco::DICT_4X4_1000);
vector<vector<Point2f> > markers;
vector<int> ids;
aruco::detectMarkers(image, theDictionary, markers, ids);
aruco::drawDetectedMarkers(image, markers, ids);
return true; //There's actually more code here that makes sure there are four markers.
}
Examination of the optional detectorParameters argument to detectMarkers showed a parameter called doCornerRefinement. Its description is "do subpixel refinement or not". Since the error I'm seeing is larger than a pixel, I didn't think this was applicable to my situation. I gave it a try anyway and experimented with the cornerRefinementWinSize value and found that it did indeed solve my problem. Now I'm thinking that "pixel" in the ArUco sense is the size of one of the squares within the marker, not an image pixel.
The modified call to detectMarkers:
bool findMarkers(const Mat image, Point2d outerMarkerCoordinates[], Point2d innerMarkerCoordinates[], Size2d *boardSize) {
Ptr<aruco::Dictionary> theDictionary = aruco::getPredefinedDictionary(aruco::DICT_4X4_1000);
vector<vector<Point2f> > markers;
vector<int> ids;
Ptr<aruco::DetectorParameters> detectorParameters = new aruco::DetectorParameters;
detectorParameters->doCornerRefinement = true;
detectorParameters->cornerRefinementWinSize = 11;
aruco::detectMarkers(image, theDictionary, markers, ids, detectorParameters);
aruco::drawDetectedMarkers(image, markers, ids);
return true; //There's actually more code here that makes sure there are four markers.
}
Success!
Related
Here is the picture I take with my USB camera. My camera has an angle with horizontal line, the target is on the bottom, with parallel and orthogonal lines delimiting rectangles. Post-it is a control marker of the center-rectangle.
Then I process several step-by-step processing in order to adjust the 'tilt' of the view and to extract lines.
Here is the line extraction without transform :
{"type":"toGray"} => mat.cvtColor( cv4.COLOR_BGR2GRAY);
{"type":"toBlur","size":10} => mat.gaussianBlur( new cv4.Size( size, size),0);
{"type":"toCanny","low":50,"high":150} => mat.canny( low_threshold, high_threshold);
{"type":"getLines","rho":1,"theta":0.017453292222222222,"threshold":15,"min_line_length":50,"max_line_gap":20 }] => let lines = mat.houghLinesP( rho, theta, threshold, min_line_length, max_line_gap);
Result is :
Now, I want to correct the tilt of view, using 'warpAffine' function, before extracting lines.
I select four points of the centered rectangle, in order to build two "three points array" (src, dst):
matTransf = cv4.getAffineTransform( srcPoints, dstPoints);
resultMat = mat.warpAffine( matTransf, new cv4.Size( mat.cols, mat.rows));
The result is the following:
Where is the mistake ?
I have tried too :
// four points at each corner of the rectangle, srcPoints for the picture, and dstPoints for the theoric shape
// With getPerspectiveTransform
matTransf = cv4.getPerspectiveTransform( srcPoints, dstPoints);
resultMat = mat.warpPerspective( matTransf, new cv4.Size( mat.cols, mat.rows));
// With findHomography
let result = cv4.findHomography( srcPoints, dstPoints);
matTransf = result.homography;
resultMat = mat.warpPerspective( matTransf, new cv4.Size( mat.cols, mat.rows));
Result is :
Best regards.
The transformation is not an affinity, it is a perspective described by a homography. Select in the image four corners of a physical rectangle, map them to points in a rectangle with the same aspect ratio as the physical one, estimate the homography from them (findHomography), finally warp (warpPerspective).
original picture
result picture
Here is my code:
private void button3_Click(object sender, EventArgs e)
{
string strFileName = string.Empty;
OpenFileDialog ofd = new OpenFileDialog();
if (ofd.ShowDialog() == DialogResult.OK)
{
Image<Bgr, byte> img1 = new Image<Bgr, byte>(ofd.FileName);
pictureBox1.Image = img1.ToBitmap();
Image<Gray, Byte> gray1 = img1.Convert<Gray, Byte>().PyrUp().PyrDown();
CircleF[] circles = gray1.HoughCircles(
new Gray(150),
new Gray(100),
2,
10,
0,
0)[0];
Image<Bgr, byte> imageCircles = img1.CopyBlank();
foreach (CircleF circle in circles)
{
imageCircles.Draw(circle, new Bgr(Color.Yellow), 5);
}
pictureBox4.Image = imageCircles.ToBitmap();
}
}
Are my parameters set correctly? Is there something I'm not understanding correctly?
Thank you!
EmguCV wraps the OpenCV stuff, so you can get mor information about how to use Emgu methods by looking at the OpenCV Doku. Here is he Hough transformation explained (could be that the parameter count or order varies but in most cases the parameter names match. At the explanation point 4 (Proceed to apply Hough Circle Transform:) you get all parameters explained:
dp = 1: The inverse ratio of resolution
min_dist = src_gray.rows/8: Minimum distance between detected centers
param_1 = 200: Upper threshold for the internal Canny edge detector
param_2 = 100*: Threshold for center detection.
min_radius = 0: Minimum radio to be detected. If unknown, put zero as default.
max_radius = 0: Maximum radius to be detected. If unknown, put zero as default
According to the Emgu Doku these values are used:
cannyThreshold
Type: TColor
The higher threshold of the two passed to Canny edge detector (the lower one will be twice smaller).
accumulatorThreshold
Type: TColor
Accumulator threshold at the center detection stage. The smaller it is, the more false circles may be detected. Circles, corresponding to the larger accumulator values, will be returned first
dp
Type: SystemDouble
Resolution of the accumulator used to detect centers of the circles. For example, if it is 1, the accumulator will have the same resolution as the input image, if it is 2 - accumulator will have twice smaller width and height, etc
minDist
Type: SystemDouble
Minimum distance between centers of the detected circles. If the parameter is too small, multiple neighbor circles may be falsely detected in addition to a true one. If it is too large, some circles may be missed
minRadius (Optional)
Type: SystemInt32
Minimal radius of the circles to search for
maxRadius (Optional)
Type: SystemInt32
Maximal radius of the circles to search for
So try to use a higher accumulatorThreshold (like 150 or 180) because your circles are pretty obvious. On large images a higher dp could help too, decreasing your waiting time.
Your dp is 10 ( I think it means 10pixils between detected centers) just take a much higher number depending on your image size. the OpenCV example uses an 8th of images height.
A minRadius of 0 should be ok since your image shows only large (yellow) circles.
A max Radios should depend on your image size too - this would remove those really large circles in lower image part.
Just try 5% of your image height as minRadius and ca 90% as maxRadius - or simply use sliders and an apply button to try it yourself
So, here is my situation. I have created a object detection program which is based on color object detection. My program detects the color red and it works perfectly. But here is the problems i am facing:-
Whenever there are more than one red object in the surrounding, my program detects them and it cannot really track one object at that time(i.e it tracks other red objects of various sizes in the background. It shows me the error that "too much noise in the background". As you can see in the "threshold image" attached, it detects the round object (which is my tracking object) and my cap which is red in color. I want my program to detect only my tracking object("which is a round shaped coke cap"). How can i achieve that? Please help me out. I have my engineering design contest in few days and i have to demo my program infront of my lecturers. My program should only be able to detect and track the object which i want. Thanks
My code for the objectdetection program is a little long. So, i am hereby explaining the code as follows- I captured a frame from the webcam frame-converted it to HSV- used HSV Inrange filter to filter out the other colors but red- applied morphological operations on the filtered image. This all goes in my main function
I am using a frame resolution of 1280*720 for my webcam frame. It kind of slows down my program but it was a trade off which i had to do for performing gesture controlled operations. Anyways here is my drawobjectfunction and trackfilteredobjectfunction.
int H_MIN = 0;
int H_MAX = 256;
int S_MIN = 0;
int S_MAX = 256;
int V_MIN = 0;
int V_MAX = 256;
//default capture width and height
const int FRAME_WIDTH = 1280;
const int FRAME_HEIGHT = 720;
//max number of objects to be detected in frame
const int MAX_NUM_OBJECTS=50;
//minimum and maximum object area
const int MIN_OBJECT_AREA = 20*20;
const int MAX_OBJECT_AREA = FRAME_HEIGHT*FRAME_WIDTH/1.5;
void drawObject(int x, int y,Mat &frame){
circle(frame,Point(x,y),20,Scalar(0,255,0),2);
if(y-25>0)
line(frame,Point(x,y),Point(x,y-25),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(x,0),Scalar(0,255,0),2);
if(y+25<FRAME_HEIGHT)
line(frame,Point(x,y),Point(x,y+25),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(x,FRAME_HEIGHT),Scalar(0,255,0),2);
if(x-25>0)
line(frame,Point(x,y),Point(x-25,y),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(0,y),Scalar(0,255,0),2);
if(x+25<FRAME_WIDTH)
line(frame,Point(x,y),Point(x+25,y),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(FRAME_WIDTH,y),Scalar(0,255,0),2);
putText(frame,intToString(x)+","+intToString(y),Point(x,y+30),1,1,Scalar(0,255,0),2);
}
void trackFilteredObject(int &x, int &y, Mat threshold, Mat &cameraFeed){
Mat temp;
threshold.copyTo(temp);
//these two vectors needed for output of findContours
vector< vector<Point> > contours;
vector<Vec4i> hierarchy;
//find contours of filtered image using openCV findContours function
findContours(temp,contours,hierarchy,CV_RETR_CCOMP,CV_CHAIN_APPROX_SIMPLE );
//use moments method to find our filtered object
double refArea = 0;
bool objectFound = false;
if (hierarchy.size() > 0) {
int numObjects = hierarchy.size();
//if number of objects greater than MAX_NUM_OBJECTS we have a noisy filter
if(numObjects<MAX_NUM_OBJECTS){
for (int index = 0; index >= 0; index = hierarchy[index][0]) {
Moments moment = moments((cv::Mat)contours[index]);
double area = moment.m00;
//if the area is less than 20 px by 20px then it is probably just noise
//if the area is the same as the 3/2 of the image size, probably just a bad filter
//we only want the object with the largest area so we safe a reference area each
//iteration and compare it to the area in the next iteration.
if(area>MIN_OBJECT_AREA && area<MAX_OBJECT_AREA && area>refArea){
x = moment.m10/area;
y = moment.m01/area;
objectFound = true;
refArea = area;
}else objectFound = false;
}
//let user know you found an object
if(objectFound ==true){
putText(cameraFeed,"Tracking Object",Point(0,50),2,1,Scalar(0,255,0),2);
//draw object location on screen
drawObject(x,y,cameraFeed);}
}else putText(cameraFeed,"TOO MUCH NOISE! ADJUST FILTER",Point(0,50),1,2,Scalar(0,0,255),2);
}
}
Here is the link of the image; as you can see it also detects the red hat in the background along with the red cap of the coke bottle.
My observations:- Here is what i think, to achieve my desired goal of not detecting objects of unknown sizes of red color. I think i have to edit the value of maximum object area which i declared in the above program as (const int MAX_OBJECT_AREA = FRAME_HEIGHT*FRAME_WIDTH/1.5;). I think i have to change this value, that might eliminate the detection of bigger continous red pictures. But also, there is another problem some objects are not completely red in color and they have patches of red and other colors. So, if the detected area is within the range specfied in my program then my program detects those red patches too. What i mean to say is i was wearing a tshirt which has mixed colors and when i tested my program by wearing that tshirt, my program was able to detect the red color out of the other colors. Now, how do i solve this issue?
I think you can try out the following procedure:
obtain a circular kernel having roughly the same area as your object of interest. You can do it like: Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(d, d));
where d is the diameter of the disk.
perform normalized-cross-correlation or convolution of the filtered regions image with this kernel (I think normalized-cross-correlation would be better. And add an empty boarder around the kernel).
the peak of the resulting image should give you the location of the circular region in your filtered image (if you are using normalized-cross-correlation, you'll have to add the shift).
To speed things up, you can perform this at a reduced resolution.
You can filter out non-circular shapes by detecting circles in your thresholded image. OpenCV provides a built-on method to detect circles using Hough transform, more info here. You can take advantage of this function to retain only circles that have a radius in a given range.
Another possibility is to implement connected component labeling (CCL) into your demo program.
I believe that it was removed at some point in verions 2.x of OpenCV, but a basic implementation of the two-pass version is straightforward from the Wikipedia page.
CCL will assign a unique ID for each object after thresholding. You then have to implement matching between the objects at frame (T-1) and objects in frame (T) (for example based on some nearest distance criterion) and possibly trajectory filtering or smoothing, but this would definitely give you some extra-points.
After some color detection and binary thresholding, I use the following code to find the contours and draw them onto the image:
using (MemStorage stor = new MemStorage())
{
Contour<Point> contours = img.FindContours(
Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_LIST,
stor);
for (; contours != null; contours = contours.HNext)
{
Contour<Point> currentContour = contours.ApproxPoly(contours.Perimeter * poly, stor);
img.Draw(currentContour,new Bgr(255,255,255),1);
Rectangle currentrect = currentContour.BoundingRectangle;
img.Draw(currentrect,new Bgr(255,255,255),2);
}
}
My problem is, as I expected, that if the contour is a rectangle but is rotated in the image, the bounding rectangle does not change its orientation to fit the rotation. Is their another way to accomplish this function? Any help would be greatly appreciated.
Yes, there is another way to accomplish this. You can use
contour.GetConvexHull(ORIENTATION.CV_CLOCKWISE);
using Moments, you can easily get the orientation and adjust the rectangle accordingly.
The method you are looking for is:
PointCollection.MinAreaRect(points);
Worked example is here:
http://www.emgu.com/wiki/index.php/Minimum_Area_Rectangle_in_CSharp
Complete documentation (which has little more than the above) is located here:
http://www.emgu.com/wiki/files/2.4.0/document/html/0d5fd148-0afb-fdbf-e995-6dace8c8848d.htm
So I have this Panel class. It's a little like a Window where you can resize, close, add buttons, sliders, etc. Much like the status screen in Morrowind if any of you remember. The behavior I want is that when a sprite is outside of the panel's bounds it doesn't get drawn and if it's partially outside only the part inside gets drawn.
So what it does right now is first get a rectangle that represents the bounds of the panel, and a rectangle for the sprite, it finds the rectangle of intersection between the two then translates that intersection to the local coordinates of the sprite rectangle and uses that for the source rectangle. It works and as clever as I feel the code is I can't shake the feeling that there's a better way to do this. Also, with this set up I cannot utilize a global transformation matrix for my 2D camera, everything in the "world" must be passed a camera argument to draw. Anyway, here's the code I have:
for the Intersection:
public static Rectangle? Intersection(Rectangle rectangle1, Rectangle rectangle2)
{
if (rectangle1.Intersects(rectangle2))
{
if (rectangle1.Contains(rectangle2))
{
return rectangle2;
}
else if (rectangle2.Contains(rectangle1))
{
return rectangle1;
}
else
{
int x = Math.Max(rectangle1.Left, rectangle2.Left);
int y = Math.Max(rectangle1.Top, rectangle2.Top);
int height = Math.Min(rectangle1.Bottom, rectangle2.Bottom) - Math.Max(rectangle1.Top, rectangle2.Top);
int width = Math.Min(rectangle1.Right, rectangle2.Right) - Math.Max(rectangle1.Left, rectangle2.Left);
return new Rectangle(x, y, width, height);
}
}
else
{
return null;
}
}
and for actually drawing on the panel:
public void DrawOnPanel(IDraw sprite, SpriteBatch spriteBatch)
{
Rectangle panelRectangle = new Rectangle(
(int)_position.X,
(int)_position.Y,
_width,
_height);
Rectangle drawRectangle = new Rectangle();
drawRectangle.X = (int)sprite.Position.X;
drawRectangle.Y = (int)sprite.Position.Y;
drawRectangle.Width = sprite.Width;
drawRectangle.Height = sprite.Height;
if (panelRectangle.Contains(drawRectangle))
{
sprite.Draw(
spriteBatch,
drawRectangle,
null);
}
else if (Intersection(panelRectangle, drawRectangle) == null)
{
return;
}
else if (Intersection(panelRectangle, drawRectangle).HasValue)
{
Rectangle intersection = Intersection(panelRectangle, drawRectangle).Value;
if (Intersection(panelRectangle, drawRectangle) == drawRectangle)
{
sprite.Draw(spriteBatch, intersection, intersection);
}
else
{
sprite.Draw(
spriteBatch,
intersection,
new Rectangle(
intersection.X - drawRectangle.X,
intersection.Y - drawRectangle.Y,
intersection.Width,
intersection.Height));
}
}
}
So I guess my question is, is there a better way to do this?
Update: Just found out about the ScissorRectangle property. This seems like a decent way to do this; it requires a RasterizerState object to be made and passed into the spritebatch.Begin overload that accepts it. Seems like this might be the best bet though. There's also the Viewport which I can apparently change around. Thoughts? :)
There are several ways to limit drawing to a portion of the screen. If the area is rectangular (which seems to be the case here), you could set the viewport (see GraphicsDevice) to the panel's surface.
For non-rectangular areas, you can use the stencil buffer or use some tricks with the depth buffer. Draw the shape of the surface in the stencil buffer or the depth buffer, set your render state to draw only pixels located in the shape you just rendered in the stencil/depth buffer, finally render your sprites.
One way of doing this is simple per-pixel collision. Although this is a bad idea if the sprites are large or numerous, this can be a very easy and fast way to get the job done with small sprites. First, do a bounding circle or bounding square collision check against the panel to see if you even need to do per-pixel detection.
Then, create a contains method that checks if the position, scale, and rotation of the sprite put it so far inside the panel that it must be totally enclosed by the panel, so you don't need per-pixel collision in that case. This can be done pretty easily by just creating a bounding square that has the width and height of the length of the sprite's diagonal, and checking for collision with that.
Finally, if both of these fail, we must do per-pixel collision. Go through and check against every pixel in the sprite to see if it is within the bounds of the panel. If it isn't set the alpha value of the pixel to 0.
Thats it.