In my project I need to recognize euro coins and someone advised me to use OpenCV classifiers and training algorithms. So I downloaded the 3.1 version of OpenCV and I trying to get it started. I would like to know some things that I don't understand from tutorials (the ones I am following are this, this from OpenCV official documentation and this).
First of all, is it mandatory to generate and consider negative samples? if yes, which kind of "objects" should I consider as negatives? In my app I should detect and recognize euro coins so...should I create negatives from any other random kind of objects?
Secondly, my app is supposed to recognize 2€, 1€ and 0.50€ coins. So, how many positive samples shall I generate with opencv_createsamples? One for each coin (front and back) or an unique one for all the 3 kind of coins? If I have understood well, then I will have some .xml files which I am supposed to enclose in my iOS app project, right?
Finally, will detectMultiScale() not only detect the coin but also its kind? That's why I was thinking that I would need more than a classifier file, to distinguish back from right side and to distinguish the value of the coin.
Hoping I have not written a too broad question, thank you for your attention.
Regarding negative (or, background) example consider on which background the euro coins will be in the image that your program has to process in the end. E.g., if they are on a table together with non-Euros, use table tops (without coins) as negatives. If you want to spot them on images where people hold them, use hands as negatives, etc.
Regarding the second question - I am also searching for advice on this. I guess it depends on the complexity of the features you need to recognize and also on the complexity of the background. I succeeded to train a classifier with 40 circle positives, but completely failed to train a classifier using 60 (aerial) cow images.
Regarding your third question, I think (not tested!) that depends on your recognition needs: If you just want to spot one of the Euro coins, you'll be fine to train one classifier with images from all coins from all sides. If you want to distinguish the coins, create different classifiers with images of each coin from all sides. If you even need to distinguish front from back, you'll have to create one classifier for every side of every coin.
yes, Negative samples are mandatory and again yes, you can have images of random objects but you need to have lots of them so i suggest use videos to extract their frames or website such as this
next, you need to have positive samples for your coins. Greater the number of positive coins, more accurate results will be. One thing you can do is to take 2-3 images of your coin and super impose them on your negative ones by changing an angle every time you superimpose. That way if you have say, 500 negative images and 2 positive images of a coin, you will have 1000 positive images after superimposing. opencv_createsamples script takes care of all of this.
Related
My xmas holiday project this year was to build a little Android app, which should be able to detect arbitrary Euro coins in a picture, recognize their value and sum the values up.
My assumptions/requirements for the picture for a good recognition are
uniform background
picture should be roughly the size of a DinA4 paper
coins may not overlap, but may touch each other
number-side of the coins must be up/visible
My initial thought was, that for the coin value-recognition later it would be best to first detect the actual coins/their regions in the picture. Any recognition then would run on only these regions of the picture, where actual coins are found.
So the first step was to find circles. This i have accomplished using this OpenCV 3 pipeline, as suggested in several books and SO postings:
convert to gray
CannyEdge detection
Gauss blurring
HoughCircle detection
filtering out inner/redundant circles
The detection works rather successfully IMHO, here a picture of the result:
Coins detected with HoughCircles with blue border
Up to the recognition now for every found coin!
I searched for solutions to this problem and came up with
template matching
feature detection
machine learning
The template matching seems very inappropriate for this problem, as the coins can be arbitrary rotated with respect to a template coin (and the template matching algorithm is not rotation-invariant! so i would have to rotate the coins!).
Also pixels of the template coin will never exactly match those of the region of the formerly detected coin. So any algorithm computing the similarity will produce only poor results, i think.
Then i looked into feature detection. This seemed more appropriate to me. I detected the features of a template-coin and the candidate-coin picture and drew the matches (combination of ORB and BRUTEFORCE_HAMMING). Unfortunately the features of the template-coin were also detected in the wrong candidate coins.
See the following picture, where the template or "feature" coin is on the left, a 20 Cents coin. To the right there are the candidate coin, where the left-most coin is a 20 Cents coin. I actually expected this coin to have the most matches, unfortunately not. So again, this seems not to be a viable way to recognize the value of coins.
Feature-matches drawn between a template coin and candidate coins
So machine learning is the third possible solution. From university i still now about neural networks, how they work, etc. Unfortunately my practical knowledge is rather poor AND i don't know Support Vector Machines (SVM) at all, which is the machine learning supported by OpenCV.
So my question is actually not source-code related, but more how to setup the learning process.
Should i learn on the plain coin-images or should i first extract features and learn on the features? (i think: features)
How much positives and negatives per coin should be given?
Would i have to learn also on rotated coins or would this rotation be handled "automagically" by the SVM? So would the SVM recognize rotated coins, even if i only trained it on non-rotated coins?
One of my picture-requirements above ("DinA4") limits the size of the coin to a certain size, e.g. 1/12 of the picture-height. Should i learn on coins of roughly the same size or different sizes? I think, that different sizes would result in different features, which would not help the learning process, what do you think?
Of course, if you have a different possible solution, this is also welcome!
Any help is appreciated! :-)
Bye & Thanks!
Answering your questions:
1- Should i learn on the plain coin-images or should i first extract features and learn on the features? (i think: features)
For many object classification tasks it's better to extract the features first and then train a classifier using a learning algorithm. (e.g the features can be HOG and the learning algorithm can be something like SVM or Adaboost). It's mainly due to the fact that the features have more meaningful information compared to the pixel values. (They can describe edges,shapes, texture, etc.) However, the algorithms like deep learning will extract the useful features as a part of learning procedure.
2 - How much positives and negatives per coin should be given?
You need to answer this question depending on the variation in the classes you want to recognize and the learning algorithm you use. For SVM , if you use HOG features and want to recognize specific numbers on coins you won't need much.
3- Would i have to learn also on rotated coins or would this rotation be handled "automagically" by the SVM? So would the SVM recognize rotated coins, even if i only trained it on non-rotated coins?
Again it depends on your final decision about the features(not SVM which is the learning algorithm) you're going to choose. HOG features are not rotation invariant but there are features like SIFT or SURF which are.
4-One of my picture-requirements above ("DinA4") limits the size of the coin to a certain size, e.g. 1/12 of the picture-height. Should i learn on coins of roughly the same size or different sizes? I think, that different sizes would result in different features, which would not help the learning process, what do you think?
Again, choose your algorithm , some of them ask you for a fixed/similar width/height ratio. You can find out about the specific requirements in related papers.
If you decide to use SVM take a look at this and also if you feel ok with Neural Network, using Tensorflow is a good idea.
I want to design an algorithm that would find matches in images of the same apartment, when put up by different real estate agents.
Photos are relatively taken in similar time so the interior of the rooms should not change that much but of course every guys takes different pictures from different angles, etc.
(TLDR; a apartment goes for sale, and different real estate guys come in and make their own pictures, and I want to know if the given pictures from various guys are of the same place)
I know that image processing and recognition algorithm selections highly depend on the use case, so could you point me in correct direction given my use-case?
http://reality.bazos.sk/inzerat/56232813/Prenajom-1-izb-bytu-v-sirsom-centre.php
http://reality.bazos.sk/inzerat/56371292/-PRENAJOM-krasny-1i-byt-rekonstr-Kupeckeho-Ruzinov-BA-II.php
You can actually use Clarifai's Custom Training API endpoint, fairly simple and straightforward. All you would have to do is train the initial image and then compare the second to it. If the probability is high, it is likely the same apartment. For example:
In javascript, to declare a positive it is:
clarifai.positive('http://example.com/apartment1.jpg', 'firstapartment', callback);
And a negative is:
clarifai.negative('http://example.com/notapartment1.jpg', 'firstapartment', callback);
You don't necessarily have to do a negative, but it could only help. Then, when you are comparing images to the first aparment, you do:
clarifai.predict('http://example.com/someotherapartment.jpg', 'firstapartment', callback);
This will give you a probability regarding the likeness of the photo to what you've trained ('firstapartment'). This API is basically doing machine learning without the hassle of the actual machine. Clarifai's API also has a tagging input that is extremely accurate with some basic tags. The API is free for a certain number of calls/month. Definitely worth it to check out for this case.
As user Shaked mentioned in a comment, this is a difficult problem. Even if you knew the position and orientation of each camera in space, and also the characteristics of each camera, it wouldn't be a trivial problem to match the images.
A "bag of words" (BoW) approach may be of use here. Rather than try to identify specific objects and/or deduce the original 3D scene, you determine what "feature descriptors" can distinguish objects from one another in your image sets.
https://en.wikipedia.org/wiki/Bag-of-words_model_in_computer_vision
Imagine you could describe the two images by the relative locations of textures and colors:
horizontal-ish line segments at far left
red blob near center left
green clumpy thing at bottom left
bright round object near top left
...
then for a reasonably constrained set of images (e.g. photos just within a certain zip code), you may be able to yield a good match between the two images above.
The Wikipedia article on BoW may look a bit daunting, but I think if you hunt around you'll find an article that describes "bag of words" for image processing clearly. I've seen a very good demo of a BoW approach used to identify objects such as boats and delivery vans in arbitrary video streams, and it worked impressively well. I wish I had a copy of the presentation to pass along.
If you don't suspect the image to change much, you could try the standard first step of any standard structure-from-motion algorithm to establish a notion of similarity between a pair of images. Any pair of images are similar if they contain a number of matching image features larger than a threshold which satisfy the geometrical constraint of the scene as well. For a general scene, that geometrical constraint is given by a Fundamental Matrix F computed using a subset of matching features.
Here are the steps. I have inserted the opencv method for each step, but you could write your methods too:
Read the pair of images. Use img = cv2.imread(filename).
Use SIFT/SURF to detect image features/descriptors in both images.
sift = cv2.xfeatures2d.SIFT_create()
kp, des = sift.detectAndCompute(img,None)
Match features using the descriptors.
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1,des2)
Use RANSAC to compute funamental matrix.
cv2.findFundamentalMatrix(pts1, pts2, cv2.FM_RANSAC, 3, 0.99, mask)
mask contains all the inliers. Simply count them to determine if the number of matches satisfying geometrical constraint is large enough.
CAUTION: In case of a planar scene, we use homography instead of a fundamental matrix and the steps described above work out pretty nicely because homography takes a point to a corresponding point in the other image. However, Fundamental matrix takes a point to the corresponding epipolar line in the other image, which makes the entire process a bit less stable. So I would recommend trying these steps a few more times with a little bit of jitter to the feature locations and collating the evidence over more than one trial to make the decision. You can also use more advanced steps to introduce robustness to this process but only if the steps described above don't yield the results you need.
I got some questions about the training of a cascade classifier:
On Some of my pictures half of the object is visible. Should I mark the visible part as region of interest, use the picture as negative sample or sort it out completely?
Is the classifier able to detect objects that are just partly visible (using Haar features)?
What should be the ratio of negative and positive samples? Often I read that you should use more negative samples. But for example in this thread it is mentioned that the ratio should be 2:1 (more positive samples).
My current classifier detects to much false positives. According to this tutorial you can either increase the number of stages or decrease the false alarm rate per stage. But I can't increase the number of stages without increasing the false alarm rate. If I just increase the number of stages, the training stops at some point because the classifier runs out of samples. Is the only way to reduce the false positives to increase the number of samples?
Would be glad if someone could answer one of my questions :)
In case of cascade classifier I would suggest to throw away the "half" objects. Since are they positive samples? no since they don't contain the object entirely, are they negative samples? no , because they are not something which have nothing to do with our object.
In my experience I started with training with almost similar number of negative and positive images, and I had the similar problem. Increasing the number of samples was the first step. You should probably increase the number of negative samples, note that you need to get different images, simply having 100 similar background images are almost the same as having only like 5-10 images. In my case the best ratio was positive:negative = 2:1. You still need to try out though it depends on the classifier you are trying to build. If your object is not something too fancy and comes in simple shapes and sizes (like a company logo or coin, or an orange) you don't have to get too many samples but if you are trying to build a classifier which checks for some complicated objects ( like a chair, yes.. chair is a serious object, since it comes in many different shapes and sizes) than you will need a lot of samples.
Hope this helps.
I have decided to train Haar classifier for 102 flower categories given here:(The dataset)
http://www.robots.ox.ac.uk/~vgg/data/flowers/102/categories.html
In the link you can see several categories. I am posting a few images of an individual flower to explain the question.
This flower belongs to a single class. I have 250 images as positives. There is a considerable variation in this flower's others images(of color, brightness, orientation, etc.). I am hunting for negative images right now. As you might have guessed, I didn't click these pictures so I can't go to the places where these were clicked to collect negative dataset. Instead, I have decided to extract frames from a video. Here is the link:
https://www.youtube.com/watch?v=x3zT1mJE0W0
Here are the images from the video:
It is a video of general garden with bushes and plants background.
My question is: Will this video(and other similar videos) suffice for being negative samples for successful detection? Is it safe to train the classifier for these flowers at all?(I mean with lot of variation in the background. I also plan to use the rest flowers category images as negatives that I am not detecting except the flower that I am trying to detect which in the case here is the Passion Flower).
This is my first training and I am asking this because the training is gonna eat my whole day and night. I am skeptical about it beforehand.
The trick with negative images is to use whatever you have, and as many as possible. The more difference, and quantity, of your negative images means that you will end up with a more robust classifier.
As for your specific question about whether the bushes are a good negative data set compared to the flowers I would say they will be ok. The background behind the bushes is relatively similar and you have quite a distinct flower pattern for your positive samples.
I am wanting to count the number of cars in aerial images of parking lots. After some research I believe that Haar Cascade Classifiers might be an option for this. An example of an image I will be using would be something similar to a zoomed in image of a parking lot from Google Maps.
My current plan to accomplish this is to train a custom Haar Classifier using cars that I crop out of images in only one orientation (up and down), and then attempt recognition multiple times while rotating the image in 15 degree increments. My specific questions are:
Is using a Haar Classifier a good approach here or is there something better?
Assuming this is a good approach, when cropping cars from larger images for training data would it be better to crop a larger area that could possibly contain small portions of cars in adjacent parking spaces (although some training images would obviously include solo cars, cars with only one car next to them, etc.) or would it be best to crop the cars as close to their outline as possible?
Again assuming I am taking this approach, how could I avoid double counting cars? If a car was recognized in one orientation, I don't want it to be counted again. Is there some way that I could mark a car as counted and have it ignored?
I think in your case I would not go for Haar features, you should search for something that is rotation invariant.
I would recommend to approach this task in the following order:
Create a solid training / testing data set and have a good look into papers about getting good negative samples. In my experience good negative samples have a great deal of influence on the resulting quality of your classifier. It makes your life a lot easier if all your samples are of the same image size. Add different types of negative samples, half cars, just pavement, grass, trees, people etc...
Before starting your search for a classifier make sure that you have your evaluation pipeline in order, do a 10 fold cross evaluation with the simplest Haar classifier possible. Now you have a baseline. Try to keep the software for all features you tested working in caseou find out that your data set needs adjustment. Ideally you can just execute a script and rerun your whole evaluation on the new data set automatically.
The problem of counting cars multiple times will not be of such importance when you can find a feature that is rotation invariant. Still non maximum suppression will be in order becaus you might not get a good recognition with simple thresholding.
As a tip, you might consider HOG features, I did have some good results on cars with them.