I am trying to detect potatoes using deep learning and sliding window approach to locate the potatoes. Since the CNN is not affected by orientation of the object that has been used to train the model, I don't see any problem in training the model but when it comes to detection there is huge issue. You see, the potatoes are more or less like cucumbers. And since I'm using sliding window technique, it is not possible to fit a potato which is in different orientation. For reference see the below image. What should I do for the detection part of the segmentation process?
This neural network is an example of how to handle object rotation. It uses a dataset where both object positions and orientations are labeled. Additionally, it used data augmentation with rotations. The rotation angle (more precisely, its sine and cosine) is added to the model output and the loss function.
So, the model detects the objects regardless of their orientation and also predicts their angles.
If you do not need to predict the angle, you can add rotated objects in the dataset and data augmentation without learning the angle. This will make the model rotationally invariant.
Related
I am working on a hand detection project. There are many good project on web to do this, but what I need is a specific hand pose detection. It needs a totally open palm and the whole palm face to outwards, like the image below:
The first hand faces to inwards, so it will not be detected, and the right one faces to outwards, it will be detected. Now I can detect hand with OpenCV. but how to tell the hand orientation?
Problem of matching with the forehand belongs to the texture classification, it's a classic pattern recognition problem. I suggest you to try one of the following methods:
Gabor filters: it is good to detect the orientation and pixel intensities (as forehand has different features), opencv has getGaborKernel function, the very important params of this function is theta (orientation) and lambd: (frequencies). To make it simple you can apply this process on a cropped zone of palm (as you have already detected it, it would be easy to crop for example the thumb, or a rectangular zone around the gravity center..etc). Then you can convolute it with a small database of images of the same zone to get the a rate of matching, or you can use the SVM classifier, where you have to train your SVM on a set of images by constructing the training matrix needed for SVM (check this question), this paper
Local Binary Patterns (LBP): it's an important feature descriptor used for texture matching, you can apply it on whole palm image or on a cropped zone or finger of image, it's easy to use in opencv, a lot of tutorials with codes are available for this method. I recommend you to read this paper talking about Invariant Texture Classification
with Local Binary Patterns. here is a good tutorial
Haralick Texture: I've read that it works perfectly when a set of features quantifies the entire image (Global Feature Descriptors). it's not implemented in opencv but easy to be implemented, check this useful tutorial
Training Models: I've already suggested a SVM classifier, to be coupled with some descriptor, that can works perfectly.
Opencv has an interesting FaceRecognizer class for face recognition, it could be an interesting idea to use it replacing the face images by the palm ones, (do resizing and rotation to get an unique pose of palm), this class has three methods can be used, one of them is Local Binary Patterns Histograms, which is recommended for texture recognition. and why not to try the other models (Eigenfaces and Fisherfaces ) , check this tutorial
well if you go for a MacGyver way you can notice that the left hand has bones sticking out in a certain direction, while the right hand has all finger lines and a few lines in the hand palms.
These lines are always sort of the same, so you could try to detect them with opencv edge detection or hough lines. Due to the dark color of the lines, you might even be able to threshold them out of it. Then gather the information from those lines, like angles, regressions, see which features you can collect and train a simple decision tree.
That was assuming you do not have enough data, if you have then you go into deeplearning, just take a basic inceptionV3 model and retrain the last dense layer to classify between two classes with a softmax, or to predict the probablity if the hand being up/down with sigmoid. Check this link, Tensorflow got your back on the training of this one, pure already ready code to execute.
Questions? Ask away
Take a look at what leap frog has done with the oculus rift. I'm not sure what they're using internally to segment hand poses, but there is another paper that produces hand poses effectively. If you have a stereo camera setup, you can use this paper's methods: https://arxiv.org/pdf/1610.07214.pdf.
The only promising solutions I've seen for mono camera train on large datasets.
use Haar-Cascade classifier,
you can get the classifier model file then use it here.
Just search for 'Haarcascade detection of Palm in Google' or use below code.
import cv2
cam=cv2.VideoCapture(0)
ccfr2=cv2.CascadeClassifier('haar-cascade-files-master/palm.xml')
while True:
retval,image=cam.read()
grey=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
palm=ccfr2.detectMultiScale(grey,scaleFactor=1.05,minNeighbors=3)
for x,y,w,h in palm:
image=cv2.rectangle(image,(x,y),(x+w,y+h),(256,256,256),2)
cv2.imshow("Window",image)
if cv2.waitKey(1) & 0xFF==ord('q'):
cv2.destroyAllWindows()
break
del(cam)
Best of Luck for your experience using HaarCascade.
I have a large dataset of vehicle images with the ground truth of their lengths (Over 100k samples). Is it possible to train a deep network to estimate vehicle length ?
I haven't seen any papers related to estimating object size using deep neural network.
[Update: I didn't notice computer-vision tag in the question, so my original answer was for a different question]:
Current convolutional neural networks are pretty good at identifying vehicle model from raw pixels. The technique is called transfer learning: take a general pre-trained model, such as VGGNet or AlexNet, and fine tune it on a vehicle data set. For example, here's a report of CS 231n course project that does exactly this (note: done by students, in 2015). No wonder there are apps out there already that do it in smartphone.
So it's more or less a solved problem. Once you know the model type, it's easy to look up it's size / length.
But if you're asking a more general question, when the vehicle isn't standard (e.g. has a trailer, or somehow modified), this is much more difficult, even for a human being. A slight change in perspective can result in significant error. Not to mention that some parts of the vehicle may be simply not visible. So the answer to this question is no.
Original answer (assumes the data is a table of general vehicle features, not the picture):
I don't see any difference between vehicle size prediction and, for instance, house price prediction. The process is the same (in the simplest setting): the model learns correlations between features and targets from the training data and then is able to predict the values for unseen data.
If you have good input features and big enough training set (100k will do),
you probably don't even need a deep network for this. In many cases that I've seen, a simplest linear regression produces very reasonable predictions, plus it can be trained almost instantly. So, in general, the answer is "yes", but it boils down to what particular data (features) you have.
You may do this under some strict conditions.
A brief introduction to Computer Vision / Multi-View Geometry:
Based on the basics of the Multi-View Geometry, the main problem of identifying of the object size is finding the conversion function from camera view to real world coordinates. By applying different conditions (i.e. capturing many sequential images - video / SfM -, taking same object's picture from different angles), we can estimate this conversion function. Hence, this is completely dependent on camera parameters like focal length, pixel width / height, distortion etc.
As soon as we have the camera to real world conversion function, it is super easy to calculate camera to point distance, hence the object's size.
So, based on your current task, you need to supply
image
camera's intrinsic parameters
(optionally) camera's extrinsic parameters
and get the output that you desire hopefully.
Alternatively, if you can fix the camera (same model, same intrinsic / extrinsic parameters), you can directly find the correlation between same camera's image and distance / object sizes just by giving the image as the only input. However, the NN will most probably will not work for different cameras.
I have a question regarding data augmentation for training the deep neural network for object detection.
I have quite limited data set (nearly 300 images). I augmented the data by rotating each image from 0-360 degrees with stepsize of 15 degree. Consequently I got 24 rotated images out of just one. So in total, I got around 7200 images. Then I drew bounding box around the object of interest in each augmented image.
Does it seem to be a reasonable approach to enhance the data?
Best Regards
In order to train a good model you need lots of representative data. Your augmentation is representative only for rotations, so yes, it is a good method, if you are concerned about having not enough object rotations. However, it will not help in any sense with generalization to other objects/transformations.
It seems like you are on the right track, rotation is usually a very useful transformation for augmenting the training data. I would suggest to try other transformations like shift (you most probably want to detect partially present objects), zoom (makes your model invariant to the scale), shear, flip, etc. By combining different transformations you can introduce additional diversity in your training data. Training set of 300 images is a very small number, so you would definitely need more than one transformation to augment so tiny training set.
This is a good approach as long as you don't implicitly change the labels when you do rotation. E.g. An image containing the digit 6 will become digit 9 on rotation of 180 deg. So, you've to pay some attention in such scenarios.
But, you could also do other geometric transformations like scaling, translation
Other augmentation that you can consider is using the pre-trained model such as ImageNet, if your problem domain has some resemblance to the ImageNet data. This will allow you to train deeper models even for your data scarce situation.
Even though rotation increases the representational complexity of your image, it might be not enough. Instead you probably need to add other types of augmentation as well.
Color augmentations are useful if they still represent the real distribution of your data.
Spatial augmentations work very good. Keep in mind that most modern systems use a lot of cropping, so that might help.
Actually I have a few scripts that I am trying to turn into a library that might work for you. Check them https://github.com/lozuwa/impy if you would like to.
I've been working on a project for some time, to detect and track (moving) vehicles in video captured from UAV's, currently I am using an SVM trained on bag-of-feature representations of local features extracted from vehicle and background images. I am then using a sliding window detection approach to try and localise vehicles in the images, which I would then like to track. The problem is that this approach is far to slow and my detector isn't as reliable as I would like so I'm getting quite a few false positives.
So I have been considering attempting to segment the cars from the background to find the approximate position so to reduce the search space before applying my classifier, but I am not sure how to go about this, and was hoping someone could help?
Additionally, I have been reading about motion segmentation with layers, using optical flow to segment the frame by flow model, does anyone have any experience with this method, if so could you offer some input to as whether you think this method would be applicable for my problem.
Below is two frames from a sample video
frame 0:
frame 5:
Assumimg your cars are moving, you could try to estimate the ground plane (road).
You may get a descent ground plane estimate by extracting features (SURF rather than SIFT, for speed), matching them over frame pairs, and solving for a homography using RANSAC, since plane in 3d moves according to a homography between two camera frames.
Once you have your ground plane you can identify the cars by looking at clusters of pixels that don't move according to the estimated homography.
A more sophisticated approach would be to do Structure from Motion on the terrain. This only presupposes that it is rigid, and not that it it planar.
Update
I was wondering if you could expand on how you would go about looking for clusters of pixels that don't move according to the estimated homography?
Sure. Say I and K are two video frames and H is the homography mapping features in I to features in K. First you warp I onto K according to H, i.e. you compute the warped image Iw as Iw( [x y]' )=I( inv(H)[x y]' ) (roughly Matlab notation). Then you look at the squared or absolute difference image Diff=(Iw-K)*(Iw-K). Image content that moves according to the homography H should give small differences (assuming constant illumination and exposure between the images). Image content that violates H such as moving cars should stand out.
For clustering high-error pixel groups in Diff I would start with simple thresholding ("every pixel difference in Diff larger than X is relevant", maybe using an adaptive threshold). The thresholded image can be cleaned up with morphological operations (dilation, erosion) and clustered with connected components. This may be too simplistic, but its easy to implement for a first try, and it should be fast. For something more fancy look at Clustering in Wikipedia. A 2D Gaussian Mixture Model may be interesting; when you initialize it with the detection result from the previous frame it should be pretty fast.
I did a little experiment with the two frames you provided, and I have to say I am somewhat surprised myself how well it works. :-) Left image: Difference (color coded) between the two frames you posted. Right image: Difference between the frames after matching them with a homography. The remaining differences clearly are the moving cars, and they are sufficiently strong for simple thresholding.
Thinking of the approach you currently use, it may be intersting combining it with my proposal:
You could try to learn and classify the cars in the difference image D instead of the original image. This would amount to learning what a car motion pattern looks like rather than what a car looks like, which could be more reliable.
You could get rid of the expensive window search and run the classifier only on regions of D with sufficiently high value.
Some additional remarks:
In theory, the cars should even stand out if they are not moving since they are not flat, but given your distance to the scene and camera resolution this effect may be too subtle.
You can replace the feature extraction / matching part of my proposal with Optical Flow, if you like. This amounts to identifying flow vectors that "stick out" from a consistent frame-to-frame motion of the ground. It may be prone to outliers in the optical flow, however. You can also try to get the homography from the flow vectors.
This is important: Regardless of which method you use, once you have found cars in one frame you should use this information to robustify your search of these cars in consecutive frame, giving a higher likelyhood to detections close to the old ones (Kalman filter, etc). That's what tracking is all about!
If the number of cars in your field of view always remain the same but move around then you can use optical flow...it will give you good results against a still background...if the number of cars are changing then you need to call goodFeaturestoTrack function in OpenCV after certain number of frames and again track the cars using optical flow.
You can use background modelling to model the background and hence the cars are always your foreground.The simplest example is frame differentiation...subtract the previous frame current frame. diff(x,y,k) = I(x,y,k) - I(x,y,k-1) .As your cars are moving in each frame you will get their position..
Both the process will work fine since you have a still background I presume..check this link to find what Optical flow can do.
I'd like to create an object detector based on cascade classifier, the only problem is that LBP and Haar features are not rotation invariant. The first thing that comes to my mind is to rotate the training sample at different angles, but I doubt that the resulting classifier would have good quality, moreover, the object could have stretched proportions. There are many rotation invariant detectors, for example, iPhone recognizes faces in real time in any orientation, so I wonder how do they achieve this? I would prefer to use OpenCV for this.
Check out the object detection framework available at https://github.com/nenadmarkus/pico.
The framework enables you to learn a custom object detector (for example, for finding frontal, upright faces) and then use it at runtime for rotation invariant detection.
This is achieved by scanning the image with a rotated version of the object detector at a number of different orientations. Note that this can be done without cascade retraining or image resampling, and it should work in real-time on modern machines (the provided face detection demo does).
The details are given in the paper available at http://arxiv.org/abs/1305.4537.
Fourier descriptors are rotation invariants (and translation as well as scaling invariants); the idea then would be to train whatever classifier your confortable with on the Fourier Descriptor result (PCA on Fourier descriptor, associated with a SVM seems to be a logical choice).
See Fourier Descriptors (Wolfram)
for matching logos I think this is what you need: http://www.ijera.com/papers/Vol2_issue5/JW2517421747.pdf
What about some simple solution....
Object Detection using SURF