Convert sklearn.svm SVC classifier to Keras implementation - machine-learning

Question

I'm trying to convert some old code from using sklearn to Keras implementation. Since it is crucial to maintain the same way of operation, I want to understand if I'm doing it correctly.
I've converted most of the code already, however I'm having trouble with sklearn.svm SVC classifier conversion. Here is how it looks right now:
from sklearn.svm import SVC
model = SVC(kernel='linear', probability=True)
model.fit(X, Y_labels)
Super easy, right. However, I couldn't find the analog of SVC classifier in Keras. So, what I've tried is this:
from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='softmax'))
model.compile(loss='squared_hinge',
optimizer='adadelta',
metrics=['accuracy'])
model.fit(X, Y_labels)
But, I think that it is not correct by any means. Could you, please, help me find an alternative of the SVC classifier from sklearn in Keras?
Thank you.

Answer 1

If you are making a classifier, you need squared_hinge and regularizer, to get the complete SVM loss function as can be seen here. So you will also need to break your last layer to add regularization parameter before performing activation, I have added the code here.
These changes should give you the output
from keras.regularizers import l2
from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1), kernel_regularizer=l2(0.01))
model.add(activation('softmax'))
model.compile(loss='squared_hinge',
optimizer='adadelta',
metrics=['accuracy'])
model.fit(X, Y_labels)
Also hinge is implemented in keras for binary classification, so if you are working on a binary classification model, use the code below.
from keras.regularizers import l2
from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1), kernel_regularizer=l2(0.01))
model.add(activation('linear'))
model.compile(loss='hinge',
optimizer='adadelta',
metrics=['accuracy'])
model.fit(X, Y_labels)
If you cannot understand the article or have issues with the code, feel free to comment.
I had this same issue a while back, and this GitHub thread helped me understand, maybe go through it too, some of the ideas here are directly from here https://github.com/keras-team/keras/issues/2588

Answer 2

If you are using Keras 2.0 then you need to change the following lines of anand v sing's answer.
W_regularizer -> kernel_regularizer
Github link
model.add(Dense(nb_classes, kernel_regularizer=regularizers.l2(0.0001)))
model.add(Activation('linear'))
model.compile(loss='squared_hinge',
optimizer='adadelta', metrics=['accuracy'])
Or You can use follow
top_model = bottom_model.output
top_model = Flatten()(top_model)
top_model = Dropout(0.5)(top_model)
top_model = Dense(64, activation='relu')(top_model)
top_model = Dense(2, kernel_regularizer=l2(0.0001))(top_model)
top_model = Activation('linear')(top_model)
model = Model(bottom_model.input, top_model)
model.compile(loss='squared_hinge',
optimizer='adadelta', metrics=['accuracy'])

Answer 3

You can use SVM with Keras implementation suing scikeras. It is a Scikit-Learn API wrapper for Keras. It was first release in May 2020. Below I have attached the official documentation link for it. I hope you will find your answer over there.
https://pypi.org/project/scikeras/#description