what's the difference between performance metrics and evaluation metrics in evaluating a classifier?
I have searched and i am not sure if they are different.
"Performance metrics" and "evaluation metrics" mean the same thing. They can be used interchangeably.
Related
I am trying to model a classifier using XGBoost on a highly imbalanced data-set, with a limited number of positive samples and practically infinite number of negative samples.
Is it possible that having too many negative samples (making the data-set even more imbalanced) will weaken the model's predictive power? Is there a reason to limit the number of negative samples aside from running time?
I am aware of the scale_pos_weight parameter which should address the issue but my intuition says even this method has its limits.
To answer your question directly: adding more negative examples will likely decrease the decision power of the trained classifier. For the negative class choose the most representative examples and discard the rest.
Learning from imbalanced dataset can influence the predictive power and even an ability of a classifier to converge at all. Generally recommended strategy is to maintain similar sizes of training examples per each of the classes. Imbalance of classes effect on learning depends on the shape of the decision space and the width of boundaries between classes. The wider they are, and the simpler the decision space the more successful training even for imbalanced datasets.
TL;DR
For a quick overview of the methods of imbalanced learning I recommend these two articles:
SMOTE and AdaSyn by example
How to Handle Imbalanced Data: An Overview
Dealing with Imbalanced Classes in Machine Learning
Learning from Imbalanced Data by Prof. Haibo He (more scientific)
There is a Python package called imbalanced-learn which has an extensive documentation of algorithms that I recommend for in-depth review.
When stacking Boltzmann machines to generatively pre-train a deep neural net, how accurate do the reconstructions need to be? If they are too accurate, can overfitting be a concern? Or is excessively high accuracy only a red flag when doing discriminative fine-tuning?
What is a concern is not burning in the markov chains enough to suppress high energy areas in training set which are far from the initial values. This is typical using CD (1) or any low order contrastive divergence. That said, these methods will always typically intialise weights far from local optima that non-pre-trained nets would get stuck in.
RBMs are also trained with simulated annealing so are more likely to explore more of the parameter space.
I also recommend you read the paper Understanding deep learning requires rethinking generalization by Zhang et al. It basically shows how these networks practically completely memorise the probabiliy distributions and can still generalise.
I'm working on replicating results from a paper and when the authors are describing their setup for SVM they say this:
To increase the dimensionality of our feature vectors to be better
suited to SVMs, we expanded the feature space by taking the polynomial
combinations of degree less than or equal to 2, of all features. This
increased the number of features from 12 to 91.
How would you do this in the gui version of weka?
I really can't figure out what setting they changed to increase the number of attributes by 79. I've looked through the internet and through the weka documentation and even just clicking around on the gui but I can't seem to find any functionality that would do this.
Thank you for your help!
It seems the authors of the paper do not really understand how SVM works. Simply train SVM with polynomial kernel of degree 2 and you will get the same expressive power.
I used Logistic Regression as a classifier. I have six features, I want to know the important features in this classifier that influence the result more than other features. I used Information Gain but it seems that it doesn't depend on the used classifier. Is there any method to rank the features according to their importance based on specific classifier (like Logistic Regression)?
any help would be highly appreciated.
You could use Random Forest Classifier to give you a ranking of your features. You could then select the top x features from this and use it for logistic regression, although Random Forest would work perfectly fine as well.
Check out variable importance at https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
One way to do this is by null hypothesis significance testing. Basically, for each feature, you test for evidence that the coefficient of that feature is nonzero. Most statistical software reports the results of these tests by default in the model summary (Scikit-learn and other machine-learning oriented tools tend to not do so). With a small number of features, you can use this information and stepwise regression to rank the importance of the features.
How can I decide if a given distance metric is a good?
Classification or clustering?
For classification, use whichever gives the highest cross-validated accuracy (or whichever quality measure you need).
For clustering, you need to pick a criteria for cluster quality, and choose the one that gives you the best cluster quality:
http://www.jstor.org/discover/10.2307/2284239?uid=3739448&uid=2&uid=3737720&uid=4&sid=21101924573361 (jstor, not free)
The metric is good, if it works for your problem.
In classification, it's as simple as that. If it gives good results, go with it. Otherwise, try a different metric.