Grid search problems with SVC - how to troubleshoot? - machine-learning

Question

I am trying to find the best hyperparameters for Support Vector classification. So far, Grid Search worked fine for tasks like that, but with the SVCs it seems to be hitting walls everywhere.
A minimal attempt with only a few suggestions for the C parameter works and produces results:
param_grid = {
'C' : [0.01, 0.1, 1, 10],
}
classifier = SVC()
grid_search = GridSearchCV(estimator=classifier, param_grid=param_grid, scoring='f1',
error_score=0, n_jobs=-1, verbose=42)
grid_search.fit(data[0], np.ravel(data[1]))
Similarly, other parameters like gamma, coef0 or shrinking don't create any problems.
However, anything involving searching for a kernel function just seems to go on processing infinitely. Even just adding one other choice leads to Python hogging all available processors for some ominous work that does not finish (at least not within 10 minutes or so).
param_grid = {
'C' : [0.01, 0.1, 1, 10],
'kernel': ['rbf', 'linear'],
}
What's really confusing me is that it starts out alright, producing good output within the first minute, then just seemingly stops doing anything while still making the coolers run in full speed. The verbose output looks like this:
Fitting 3 folds for each of 8 candidates, totalling 24 fits
[CV] kernel=rbf, C=0.01 ..............................................
[CV] kernel=rbf, C=0.01 ..............................................
[CV] kernel=rbf, C=0.01 ..............................................
[CV] kernel=linear, C=0.01 ...........................................
[...]/python3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.
'precision', 'predicted', average, warn_for)
[CV] ............... kernel=rbf, C=0.01, score=0.564932, total= 0.9s
[CV] kernel=linear, C=0.01 ...........................................
[CV] ............... kernel=rbf, C=0.01, score=0.574120, total= 0.8s
[CV] kernel=linear, C=0.01 ...........................................
[CV] ............... kernel=rbf, C=0.01, score=0.000000, total= 0.9s
[CV] kernel=rbf, C=0.1 ...............................................
[Parallel(n_jobs=-1)]: Done 1 tasks | elapsed: 1.3s
[Parallel(n_jobs=-1)]: Done 2 tasks | elapsed: 1.4s
[Parallel(n_jobs=-1)]: Done 3 tasks | elapsed: 1.5s
[CV] ................ kernel=rbf, C=0.1, score=0.555556, total= 1.0s
[CV] kernel=rbf, C=0.1 ...............................................
[Parallel(n_jobs=-1)]: Done 4 tasks | elapsed: 2.9s
[CV] ................ kernel=rbf, C=0.1, score=0.564932, total= 1.1s
[CV] kernel=rbf, C=0.1 ...............................................
[Parallel(n_jobs=-1)]: Done 5 tasks | elapsed: 4.5s
[CV] ................ kernel=rbf, C=0.1, score=0.574120, total= 1.0s
[CV] kernel=linear, C=0.1 ............................................
[Parallel(n_jobs=-1)]: Done 6 tasks | elapsed: 5.9s
Setting njobs to any other number leads to similar results: A part of the computation is done quickly and without complaints, then it seems to get stuck and uses all available CPU without any visible progress.
Furthermore, giving only one kernel choice has different results. While rbf and sigmoid work fine and finish within seconds, poly and linear apparently get stuck.
I'm at a loss - what is the problem here, and how can I run grid search usefully? My data consists of a bit over 5000 instances with 12 numerical features each. The classes are either 0 or 1, in equal distribution. Is that too much, maybe? If so, why would some searches work just fine and the trouble only start for certain kernel functions?
EDIT It looks like this is a problem with the data I'm using. The only thing that helped, so far, was normalizing the features (all values in a range from 0 to 1).
Now, normalization is generally recommended for Support Vector approaches, as they are not scale invariant, so I was going for it in any case. But I thought of it as a way to improve performance, not a necessary precaution for it to work, like it seems to be in this case.
I'll be able to work with this, for now, but I'm still curious whether anybody knows just what might be wrong with the data and how it could stil be fed to a support vector algorithm.

Answer 1

I used my data and I modified a bit you code. It runs fine the following code for me in Windows 8.
Code:
import pandas as pd
import numpy as np
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
if __name__=='__main__':
data= pd.read_csv('Prior Decompo2.csv', header=None)
X, y = data.iloc[0:, 0:26].values, data.iloc[0:,26].values
param_grid = {'C' : [0.01, 0.1, 1, 10], 'kernel': ('rbf', 'linear')}
classifier = SVC()
grid_search = GridSearchCV(estimator=classifier, param_grid=param_grid, scoring='accuracy', n_jobs=-1, verbose=42)
grid_search.fit(X,y)
Change #1:
I added the if __name__=='__main__':
Change #2:
Use parenthesis for the kernel:
param_grid = {'C' : [0.01, 0.1, 1, 10], 'kernel': ('rbf', 'linear')}
Important:
In your code that you posted after 'kernel': ['rbf', 'linear'], you have a comma that is not needed at all !
Change #3:
Inside GridSearchCV use another scoring e.g. scoring='accuracy':
grid_search = GridSearchCV(estimator=classifier, param_grid=param_grid,scoring='accuracy', n_jobs=-1, verbose=42)
The results is:
You can clearly see in the image that both linear and rbf are tested.