From the documentation scikit-learn implements SVC, NuSVC and LinearSVC which are classes capable of performing multi-class classification on a dataset. By the other hand I also read about that scikit learn also uses libsvm for support vector machine algorithm. I'm a bit confused about what's the difference between SVC and libsvm versions, by now I guess the difference is that SVC is the support vector machine algorithm fot the multiclass problem and libsvm is for the binary class problem. Could anybody help me to understad the difference between this?.
They are just different implementations of the same algorithm. The SVM module (SVC, NuSVC, etc) is a wrapper around the libsvm library and supports different kernels while LinearSVC is based on liblinear and only supports a linear kernel. So:
SVC(kernel = 'linear')
is in theory "equivalent" to:
LinearSVC()
Because the implementations are different in practice you will get different results, the most important ones being that LinearSVC only supports a linear kernel, is faster and can scale a lot better.
This is a snapshot from the book Hands-on Machine Learning
Related
I am trying to read the documentation of Vowpal Wabbit and it doesn't specify how to select specific learning algorithms (Not loss) like SVM,NN, Decision trees, etc. How does one select a specific learning algorithm?
Or does it select the algorithm itself depending on problem type (regression/classification like an automl type or low-code ML library?
There are some blogs showing to use Neural networks with -nn command but that isn't part of documentation--is this because it doesn't focus on specific algorithm, as noted above? If so, What is Vowpal Wabbit in essence?
Vowpal Wabbit is based on online learning (SGD-like updates, but there is also --bfgs if you really need batch optimization) and (machine learning) reductions. See some of the tutorials or papers to understand the idea of reductions. Many VW papers are also about Contextual Bandit, which is implemented as a reduction to a cost-sensitive one-against-all (OAA) classification (which is further reduced to regression). See a simple intro into reductions or a simple example how binary classification is reduced into regression.
As far as I know, VowpalWabbit does not support Decision trees nor ensembles, but see --boosting and --bootstrap. It does not support SVM, but see --loss_function hinge (hinge loss is one of the two key concepts of SVM) and --ksvm. It does not support NN, but --nn (and related options) provides a very limited support simulating a single hidden layer (feed-forward with tanh activation function), which can be added into the reduction stack.
I was watching a YouTube video to learn about Support Vector Machines (SVM).
In the video, he mentions that an SVM finds Support Vector Classifiers (SVC) for dividing the data as one step in their classifying process.
I have used LinearSVC from scikit-learn for classification, but I have a hard time understanding if the implementation of LinearSVC in scikit-learn is an SVM or an SVC, or if the description in the video is incorrect. I find contradicting descriptions on different sites.
The accepted answer in this question states that LinearSVC is not an SVM, but either it does not say that it is an SVC.
On the description page of LinearSVC it says "Linear Support Vector Classification", but under "See also" on this page, it says that LinearSVC is "Scalable Linear Support Vector Machine for classification implemented using liblinear".
From what I can understand, LinearSVC and SVC(kernel='linear') are not the same, but that is not the question.
Thanks!
In terms of Machine Learning concepts LinearSVC is both because:
SVM is a model/algorithm used to find a plane that splits the space of samples
this can be applied for both classification (SVC) and regression (SVR) - both SVC and SVR are kinds of SVMs
So, an SVC would be a kind of SVM and LinearSVC looks like a specific kind of SVC, although not extending a base SVC class in scikit-learn.
If you mean sklearn source code - the LinearSVC is in the svm module... so it's an SVM. It doesn't extend the SVC or BaseSVC classes but to me this is an implementation issue/detail and I'd rather think of it as an SVC.
Many packages such as sklearn use RBF kernel as default for SVM classification. I wonder if there is a proof/explanation why this is the "best" kernel for a general use.
Is there some analysis for the richness of boundaries that can be written as a_1*K(x,x_1)+...+a_m K(x,x_m)=0 for different kernels?
I am looking for some references.
Thank you.
I'm looking for a SVM implementation with support for non-linear kernels and one-vs-rest scenario, to perform a multi-label classification. Preferably, written in Python, or that I can call from Python with wrappers.
I was looking into sklearn, and there are two implementations to use SVM for classification:
sklearn.svm.LinearSVC
- supports multi-label classification with a one-vs.-rest scenario, but it's based on liblinear, and therefore only supports linear kernels.
sklearn.svm.SVC
- based on libsvm, supports non-linear kernels, but multi-label classification is done under a one-vs.-one reduction, it trains K (K − 1) / 2 binary classifiers for a K-way multiclass problem.
More info also here:
http://scikit-learn.org/stable/modules/multiclass.html
Does anyone knows any other SVM implementations directly supporting multi-label classification and non-linear kernels ?
One possible solution could also be to adapt the code based on sklearn.svm.SVC, to perform One-vs-Rest, was this already attempted before?
Binary Relevance problem transformation method uses one-vs-rest approach for doing multi-label classification. One could easily implement SVM with non-linear kernels using scikit-multilearn library. The following is the sample python code to do the same, where each row of train_y is a one-hot vector representing multiple labels (For instance, [0,0,1,0,1,0])
from skmultilearn.problem_transform.br import BinaryRelevance
from sklearn.svm import SVC
# Non-linear kernel
svm = SVC(kernel='rbf')
cls = BinaryRelevance(classifier=svm)
cls.fit(train_x, train_y)
predictions = cls.predict(test_x)
I am a beginner to SVM which i have successfully implemented one-class classification.Now i want to know about multi-class classification which am very much confused about.
I went through How to do multi class classification using Support Vector Machines (SVM) which i want the exact same output but the link does not have a specific example using windows.If anyone can help me out with an example in windows for both “ONE-AGAINST-ONE”,”ONE-AGAINST-ALL” methods of multi-class classification
Thanks
Using libLinear you will not be able to have a similar output because it cannot predict probabilities. You should use libSVM for that.
LibLinear does not support multi-class classification by default, but you can download this tool from the official site and it can do the job.
If you want multi-class probability estimate, you can take a look at this tool
I think the other answer is talking about multi-label classification or something because liblinear does support multi-class classification by default (Choose -s to be between 0 and 7, there are 8 different modes https://github.com/cjlin1/liblinear/blob/master/README)
For the input, you can use the same as binary classification, but just set the label to be the index of the class (between 0 and (# of classes - 1)) instead of ±1.