This might be simple, but why is the blassoAveraged model labeled as "Bayesian Ridge Regression (Model Averaged)"? Shouldn't it be Bayesian Lasso? Or the average of several L1 normalization models has an end-effect similar to L2?
Thanks,
Gustavo
Related
I understand how Lasso or Ridge are trying to make slopes flatter. But do all the coefficients of different features punished in a same rate? Like if without Lasso the regression is y=x+2z, then both of the coefficients will become half of the previous: y=0.5x+z? Or like the features with a less correlation with the dependent variable are punished more?
Lasso and Ridge regression use what's called L1 and L2 regularization.
Read more here: https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c
I am working on a document which should contain the key differences between using Naive Bayes (generative) and Logistic Regression (discriminative) models for text classification.
During my research, I ran into this definition for Naive Bayes model: https://nlp.stanford.edu/IR-book/html/htmledition/naive-bayes-text-classification-1.html
The probability of a document d being in class c is computed as ... where p(tk|c) is the conditional probability of term tk occurring in a document of class c...
When I got to the part of comparing Generative and Discriminative models, I found this explanation on StackOverflow as accepted: What is the difference between a Generative and Discriminative Algorithm?
A generative model learns the joint probability distribution p(x,y) and a discriminative model learns the conditional probability distribution p(y|x) - which you should read as "the probability of y given x".
At this point I got confused: Naive Bayes is a generative model and uses conditional probabilities, but at the same time the discriminative models were described as if they learned the conditional probabilities as opposed to the joint probabilities of the generative models.
Can someone shed some light on this please?
Thank you!
It is generative in the sense that you don't directly model the posterior p(y|x) but rather you learn the model of the joint probability p(x,y) which can be also expressed as p(x|y) * p(y) (likelihood times prior) and then through the Bayes rule you seek to find the most probable y.
A good read I can recommend in this context is: "On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes"
(Ng & Jordan 2004)
I was running Multi-label classification on text data I noticed TFIDF outperformed LDA by a large margin. TFIDF accuracy was aorund 50% and LDA was around 29%.
Is this expected or should LDA do better than this?
LDA is normally used for unsupervised learning, not for classification. It provides a generative model, not a discriminative model (What is the difference between a Generative and Discriminative Algorithm?), which makes it less optimal for classification. LDA can also be sensitive to data preprocessing and model parameters.
I am trying to perform logistic regression for my data. I came to know about glm. What is the actual difference between glm and regular logistic regression?
What are the pros and cons of it?
Logistic Regression is a special case of Generalized Linear Models. GLMs is a class of models, parametrized by a link function. If you choose logit link function, you'll get Logistic Regression.
The main benefit of GLM over logistic regression is overfitting avoidance. GLM usually try to extract linearity between input variables and then avoid overfitting of your model. Overfitting means very good performance on training data and poor performance on test data.
I need some help in understanding the difference between regression trees and linear model tree.
Regards
Shahzad
A linear model tree is a decision tree with a linear functional model in each leaf, whereas in classical regression tree (e.g., CART) it is the sample mean of the response variable for statistical units in each leaf (hence, a constant) that is being considered. Linear model trees can be seen as a a form of locally weighted regression, while regression tree are piecewise-constant regression.
For more information on linear model trees, you can consult
Torgo, L. Functional models for regression tree
leaves.
In Proceedings of the 14th International Conference on Machine
Learning, pp. 385–393. Morgan Kaufmann, 1997.