I'm implementing color quantization based on k-means clustering method on some RGB images. Then, I will determine the performance the algorithm. I found some information about training and testing. As I understand, I should divide the samples of images for training and testing.
But I am confused about the terms training and testing. What does these mean ? And how to implement with a rank value ?
Training and testing are two common concepts in machine learning. Training and testing are more easily explained in the framework of supervised learning; where you have a training dataset for which you know both input data as well as additional attributes that you want to predict. Training consists in learning a relation between data and attributes from a fraction of the training dataset, and testing consists in testing predictions of this relation on another part of the dataset (since you know the prediction, you can compare the output of the relation and the real attributes). A good introductory tutorial using these concepts can be found on http://scikit-learn.org/stable/tutorial/basic/tutorial.html
However, clustering is a class of unsupervised learning, that is, you just have some input data (here, the RGB values of pixels, if I understand well), without any corresponding target values. Therefore, you can run a k-means clustering algorithm in order to find classes of pixels with similar colors, without the need to train and test the algorithm.
In image processing, training and testing is for example used for classifying pixels in order to segment different objects. A common example is to use a random forest classifier: the user selects pixels belonging to the different objects of interest (eg background and object), the classifier is trained on this set of pixels, and then the remainder of the pixels are attributed to one of the classes by the classifier. ilastik (http://ilastik.org/) is an example of software that performs interactive image classification and segmentation.
I don't know which programming language you're using, but k-means is already implemented in various libraries. For Python, both SciPy (http://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.vq.kmeans2.html#scipy.cluster.vq.kmeans2) and scikit-learn (http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) have an implementation of K-means. Also note that, depending on your application, you may be interested in clustering pixels together using not only pixels values, but also spatial proximity of pixels. See for example the scikit-image gallery example http://scikit-image.org/docs/dev/auto_examples/plot_rag_mean_color.html
Related
I have a set of labeled training data, and I am training a ML algorithm to predict the label. However, some of my data points are more important than others. Or, analogously, these points have less uncertainty than the others.
Is there a general method to include an importance-representing weight to each training point in the model? Are there instead some specific models which are capable of this while others are not?
I can imagine duplicating these points (and perhaps smearing their features slightly to avoid exact duplicates), or downsampling the less important points. Is there a more elegant way to approach this problem?
Scikit-learn allows you to pass an array of sample weights while fitting the model. Vowpal Wabbit (an online ML library) also has this option.
I have a question about some basic concepts of machine learning. The examples, I observed, were giving a brief overview .For training the system, feature vector is given as input. In case of supervised learning, the dataset is labelled. I have confusion about labelling. For example if I have to distinguish between two types of pictures, I will provide a feature vector and on output side for testing, I'll provide 1 for type A and 2 for type B. But if I want to extract a region of interest from a dataset of images. How will I label my data to extract ROI using SVM. I hope I am able to convey my confusion. Thanks in anticipation.
In supervised learning, such as SVMs, the dataset should be composed as follows:
<i-th feature vector><i-th label>
where i goes from 1 to the number of patterns (also examples or observations) in your training set so this represents a single record in your training set which can be used to train the SVM classifier.
So you basically have a set composed by such tuples and if you do have just 2 labels (binary classification problem) you can easily use a SVM. Indeed the SVM model will be trained thanks to the training set and the training labels and once the training phase has finished you can use another set (called Validation Set or Test Set), which is structured in the same way as the training set, to test the accuracy of your SVMs.
In other words the SVM workflow should be structured as follows:
train the SVM using the training set and the training labels
predict the labels for the validation set using the model trained in the previous step
if you know what the actual validation labels are, you can match the predicted labels with the actual labels and check how many labels have been correctly predicted. The ratio between the number of correctly predicted labels and the total number of labels in the validation set returns a scalar between [0;1] and it's called the accuracy of your SVM model.
if you're interested in the ROI, you might want to check the trained SVM parameters (mainly the weights and bias) to reconstruct the separation hyperplane
It is also important to know that the training set records should be correctly, a priori labelled: if the training labels are not correct, the SVM will never be able to correctly predict the output for previously unseen patterns. You do not have to label your data according to the ROI you want to extract, the data must be correctly labelled a priori: the SVM will have the entire set of type A pictures and the set of type B pictures and will learn the decision boundary to separate pictures of type A and pictures of type B. You do not have to trick the labels: if you do, you're not doing classification and/or machine learning and/or pattern recognition. You're basically tricking the results.
I've got a set of F features e.g. Lab color space, entropy. By concatenating all features together, I obtain a feature vector of dimension d (between 12 and 50, depending on which features selected.
I usually get between 1000 and 5000 new samples, denoted x. A Gaussian Mixture Model is then trained with the vectors, but I don't know which class the features are from. What I know though, is that there are only 2 classes. Based on the GMM prediction I get a probability of that feature vector belonging to class 1 or 2.
My question now is: How do I obtain the best subset of features, for instance only entropy and normalized rgb, that will give me the best classification accuracy? I guess this is achieved, if the class separability is increased, due to the feature subset selection.
Maybe I can utilize Fisher's linear discriminant analysis? Since I already have the mean and covariance matrices obtained from the GMM. But wouldn't I have to calculate the score for each combination of features then?
Would be nice to get some help if this is a unrewarding approach and I'm on the wrong track and/or any other suggestions?
One way of finding "informative" features is to use the features that will maximise the log likelihood. You could do this with cross validation.
https://www.cs.cmu.edu/~kdeng/thesis/feature.pdf
Another idea might be to use another unsupervised algorithm that automatically selects features such as an clustering forest
http://research.microsoft.com/pubs/155552/decisionForests_MSR_TR_2011_114.pdf
In that case the clustering algorithm will automatically split the data based on information gain.
Fisher LDA will not select features but project your original data into a lower dimensional subspace. If you are looking into the subspace method
another interesting approach might be spectral clustering, which also happens
in a subspace or unsupervised neural networks such as auto encoder.
What system to use for Anomaly detection?
I see that systems like Mahout do not list anomaly detection, but problems like classification, clustering, recommendation...
Any recommendations as well as tutorials and code examples would be great, since I haven't done it before.
There is an anomaly detection implementation in scikit-learn, which is based on One-class SVM. You can also check out the ELKI project which has spatial outlier detection implemented.
In addition to "anomaly detection", you can also expand your search with "outlier detection", "fraud detection", "intrusion detection" to get some more results.
There are three categories of outlier detection approaches, namely, supervised, semi-supervised, and unsupervised.
Supervised: Requires fully labeled training and testing datasets. An ordinary classifier is trained first and applied afterward.
Semi-supervised: Uses training and test datasets, whereas training data only consists of normal data without any outliers. A model of the normal class is learned and outliers can be detected afterward by deviating from that model.
Unsupervised: Does not require any labels; there is no distinction between a training and a test dataset Data is scored solely based on intrinsic properties of the dataset.
If you have unlabeled data the following unsupervised anomaly detection approaches can be used to detect abnormal data:
Use Autoencoder that captures a feature representation of the features present in the data and flags as outliers data points that are not well explained using the new representation. Outlier score for a data point is calculated based on reconstruction error (i.e., squared distance between the original data and its projection) You can find implementations in H2O and Tensorflow
Use a clustering method, such as Self Organizing Map (SOM) and k-prototypes to cluster your unlabeled data into multiple groups. You can detect external and internal outliers in the data. External outliers are defined as the records positioned at the smallest cluster. Internal outliers are defined as the records distantly positioned inside a cluster. You can find codes for SOM and k-prototypes.
If you have labeled data, there are plenty of supervised classification approaches that you can try to detect outliers. Examples are Neural Networks, Decision Tree, and SVM.
I am doing remote sensing image classification. I am using the object-oriented method: first I segmented the image to different regions, then I extract the features from regions such as color, shape and texture. The number of all features in a region may be 30 and commonly there are 2000 regions in all, and I will choose 5 classes with 15 samples for every class.
In summary:
Sample data 1530
Test data 197530
How do I choose the proper classifier? If there are 3 classifiers (ANN, SVM, and KNN), which should I choose for better classification?
KNN is the most basic machine learning algorithm to paramtise and implement, but as alluded to by #etov, would likely be outperformed by SVM due to the small training data sizes. ANNs have been observed to be limited by insufficient training data also. However, KNN makes the least number of assumptions regarding your data, other than that accurate training data should form relatively discrete clusters. ANN and SVM are notoriously difficult to paramtise, especially if you wish to repeat the process using multiple datasets and rely upon certain assumptions, such as that your data is linearly separable (SVM).
I would also recommend the Random Forests algorithm as this is easy to implement and is relatively insensitive to training data size, but I would advise against using very small training data sizes.
The scikit-learn module contains these algorithms and is able to cope with large training data sizes, so you could increase the number of training data samples. the best way to know for sure would be to investigate them yourself, as suggested by #etov
If your "sample data" is the train set, it seems very small. I'd first suggest using more than 15 examples per class.
As said in the comments, it's best to match the algorithm to the problem, so you can simply test to see which algorithm works better. But to start with, I'd suggest SVM: it works better than KNN with small train sets, and generally easier to train then ANN, as there are less choices to make.
Have a look at below mind map
KNN: KNN performs well when sample size < 100K records, for non textual data. If accuracy is not high, immediately move to SVC ( Support Vector Classifier of SVM)
SVM: When sample size > 100K records, go for SVM with SGDClassifier.
ANN: ANN has evolved overtime and they are powerful. You can use both ANN and SVM in combination to classify images
More details are available #semanticscholar.org