I'm looking for metrics to compare various regressions models (e.g. SVM, Decision Tree, Neural Network etc), to decide the merits of each for solving a specific problem.
For my problem I have just over 80,000 training samples with 12 variables, all of which are independent and identically distributed.
I've done most of my research into neural networks but I'm drawing a blank when trying to compare them against other models.
Any input (including reading suggestions) would be greatly appreciated, thanks!
You can compare regression models by calculating the mean squared error for each model over a test set. The best model will simply be the one with the least error.
Sadly, there ist nothing like roc curves for regression models. Except your output is a binary variable like with logistic regression.
Related
I read this line today :
Every regression gets better with the addition of more features or variables... But adding more features increases complexity and reduces interpretability of the model as well.
I am unable to understand what is interpretability? (searched it on google but still did not get it)
Please help thank you
I would say that interpretability in a regression problems is when you can explain the result of your model to non statistician / domain experts.
For example: you try to predict the size of people depending on many variable, including sex. If you use linear regression, you will be able to say that the model will add 20cm (again, for example) to the predicted size if the person is a man (compared to a woman). The domain expert will understand the relationship between explanatory variable and the predicted result, without understanding statistics or how a linear regression works.
In addition, I disagree with the fact that the addition of more features or variables always improve regression result.
What is a better regression ? Improvement in choosen metrics ? For training or test set ? A "better regression" doesn't mean anything...
If we assume that a better regression is a regression which is better to predict the target for a new dataset, more variable doesn't always improve prediction power, especially when there is no regularization, if the added feature contains futures variables or many others cases.
I am trying to use machine learning to predict a dataset. It is a regression problem with 180 input features and 1 continuously-valued output. I try to compare deep neural networks, random forest regression, and linear regression.
As I expect, 3-hidden-layer deep neural networks outperform other two approaches with a root mean square error (RMSE) of 0.1. However, I unexpected to see that random forest even performs worse than linear regression (RMSE 0.29 vs. 0.27). In my expectation, the random forest can discover more complex dependencies between features to decrease error. I have tried to tune the parameters of random forest (number of trees, maximum features, max_depth, etc.). I also tried different K-cross validation, but the performance is still less than linear regression.
I searched online, and one answer says linear regression may perform better if features have a smooth, nearly linear dependence on the covariates. I do not fully get the point because if that is the case, should not deep neural networks give much performance gain?
I am struggling to give an explanation. Under what situation, random forest is worse than linear regression, but deep neural networks can perform much better?
If your features explain linear relation to the target variable then a Linear Model usually performs well than a Random Forest Model. It totally depends on the linear relations between your features.
That said, Linear models are not superior or the Random Forest is any inferior one.
Try scaling and transforming the data using MinMaxScaler() from scikit-learn to see if the linear model improves further
Pro Tips
If linear model is working like a charm you need to ask your self Why? and How? And get into the basics of both the models to understand why it worked on your data. These questions will lead you to feature engineer better. And as a matter of fact, Kaggle Grand Masters do use Linear Models in stacking to get that top 1% score by capturing the linear relations in the dataset.
So at the end of the day, linear models could wonders too.
I've read a few papers which consider using classifiers besides logistic regression to determine propensity scores, but none of them explain how exactly to get the propensity scores from the model. Specifically, I'm interested in using J48, Bagging, and AdaBoostM1 (the latter two most likely either with logistic regression or J48). For J48 I was thinking I could just use the probability of that node being classified as yes to determine propensity and for the other two, take the probability from each tree or equation and average them. But I'm wondering if there is a correct way to do this. I have very little statistical background and no ML background. Any good resources would be greatly appreciated. Thanks.
I have read these lines in one of the IEEE Transaction on software learning
"Researchers have adopted a myriad of different techniques to construct software fault prediction models. These include various statistical techniques such as logistic regression and Naive Bayes which explicitly construct an underlying probability model. Furthermore, different machine learning techniques such as decision trees, models based on the notion of perceptrons, support vector machines, and techniques that do not explicitly construct a prediction model but instead look at a set of most similar
known cases have also been investigated.
Can anyone can explain what they are really want to convey.
Please give example.
Thanx in advance.
The authors seem to distinguish probabilistic vs non-probabilistic models, that is models that produce a distribution p(output | data) vs those that just produce an output output = f(data).
The description of the non-probabilistic algorithms is a bits odd to my taste, though. The difference between a (linear) support vector machine, a perceptron and logistic regression from the model and algorithmic perspective is not super large. Implying the former "look at a set of most similar known cases" and the latter doesn't seems strange.
The authors seem to be distinguishing models which compute per-class probabilities (from which you can derive a classification rule to assign an input to the most probable class, or, more complicated, assign an input to the class which has the least misclassification cost) and those which directly assign inputs to classes without passing through the per-class probability as an intermediate result.
A classification task can be viewed as a decision problem; in this case one needs per-class probabilities and a misclassification cost matrix. I think this approach is described in many current texts on machine learning, e.g., Brian Ripley's "Pattern Recognition and Neural Networks" and Hastie, Tibshirani, and Friedman, "Elements of Statistical Learning".
As a meta-comment, you might get more traction for this question on stats.stackexchange.com.
I was learning about different techniques for classification, like probablistic classifiers etc , and stubled upon the question Why cant we implement a binary classifier as a Regression function of all the attributes and classify on the basis of the output of the function , say if the output is less than a certain value it belongs to class A , else in class B . Is there any limitation to this method compared to probablistic approach ?
You can do this and it is often done in practice, for example in Logistic Regression. It is not even limited to binary classes. There is no inherent limitation compared to a probabilistic approach, although you should keep in mind that both are fundamentally different approaches and hard to compare.
I think you have some misunderstanding in classification. No matter what kind of classifier you are using (svm, or logistic regression), you can always view the output model as
f(x)>b ===> positive
f(x) negative
This applies to both probabilistic model and non-probabilistic model. In fact, this is something related to risk minimization which results the cut-off branch naturally.
Yes, this is possible. For example, a perceptron does exactly that.
However, it is limited in its use to linearly separable problems. But multiple of them can be combined to solve arbitrarily complex problems in general neural networks.
Another machine learning technique, SVM, works in a similar way. It first transforms the input data into some high dimensional space and then separates it via a linear function.