I am having the same problem as previous post from dmartin but the solution presented has not being working to my dataset.
trying to fit:
model<-glmer(nb~habitat*stigmatype+(1|sitecode/stigmaspecies),
family=Gamma(link=log))
Warning message:
In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :
Model failed to converge with max|grad| = 0.00436052 (tol = 0.001)
I uploaded my R version to R-3.1.0 for Windows (32/64 bit) in order to run glmmADB package as a way to apply a post hoc test on the interaction factors.
Before that, I was using glmer, in previous R version which was working fine for at least the glmer, which gave me the following output:
> summary(nbnew)
Generalized linear mixed model fit by maximum likelihood ['glmerMod']
Family: Gamma ( log )
Formula: n ~ habitat * stigmatype + (1 | sitecode/stigmaspecies)
AIC BIC logLik deviance
3030.101 3066.737 -1506.050 3012.101
Random effects:
Groups Name Variance Std.Dev.
stigmaspecies:sitecode (Intercept) 5.209e+00 2.2822436
sitecode (Intercept) 2.498e-07 0.0004998
Residual 2.070e+00 1.4388273
Number of obs: 433, groups: stigmaspecies:sitecode, 109; sitecode, 20
Fixed effects:
Estimate Std. Error t value Pr(>|z|)
(Intercept) 2.3824 0.4080 5.839 5.26e-09 ***
habitatnon-invaded -1.8270 0.6425 -2.843 0.00446 **
stigmatypesemidry -1.7531 0.7573 -2.315 0.02061 *
stigmatypewet -1.7210 0.8944 -1.924 0.05434 .
habitatnon-invaded:stigmatypesemidry 2.0774 1.1440 1.816 0.06938 .
habitatnon-invaded:stigmatypewet 1.3120 1.4741 0.890 0.37346
---
`Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 `
Correlation of Fixed Effects:
(Intr) hbttn- stgmtyps stgmtypw hbttnn-nvdd:stgmtyps`
hbttnn-nvdd -0.635
stgmtypsmdr -0.539 0.342
stigmatypwt -0.456 0.290 0.246
hbttnn-nvdd:stgmtyps 0.357 -0.562 -0.662 -0.163
hbttnn-nvdd:stgmtypw 0.277 -0.436 -0.149 -0.607 0.245
Since I am interested in difference between each level of habitat and stigma type as well as on the interactions, I applied ghlt from multicomp:
model<-glmer(log(nb+1)~habitat*stigmatype+
(1|sitecode/stigmaspecies),
family=Gamma(link=log))
av<-anova(model)`
nb.habstigma<-interaction(nb$habitat, nb$stigmatype,drop=T)
m1<-glmer(nbnb.habstigma+(1|sitecode/stigmaspecies),family=Gamma(link=log))
stigmatest<-glht(m1, linfct = mcp(nb.habstigma = "Tukey"))
and:
Error: pwrssUpdate did not converge in (30) iterations
from here, I switched to R latest version to install glmmADMB, and got the message:
Warning message:
In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :
Model failed to converge with max|grad| = 0.00436052 (tol = 0.001)
I followed the instructions from Ben Bolker (response to dmartin) trying to refit with
control=glmerControl(optimizer="bobyqa")
but
Warning messages:
1: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :
Model failed to converge with max|grad| = 52.2329 (tol = 0.001)
2: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :
Model failed to converge: degenerate Hessian with 1 negative eigenvalues
Any ideas- please??
Thank you!
Related
I've been trying out different models for my count data that is zero inflated and it looks like after looking at the Akaike information criterion (AIC) and the rootgram of different models, the generalized linear model negative binomial is the best fit. I'd like to do a post-hoc on my data but I keep running into some errors and I'm not sure the best way to frame the code in order to get what I'm looking for.
I used the following code for the model, where "flwrs" is my response variable (the count data), "tre" is a treatment group (which there are four of - F400, F800, GE400, GE800, and "wa" is for my weeks after transplant (ranging from 1-26).
negbin <- glm.nb(flwrs ~ tre*wa + (1 | replic), data = test)
summary(negbin)
Anova(negbin)
With the anova I see p < 2.2e-16 for tre, wa, and tre:wa. I'd like to explore more the significance between the tre and wa in particular.
Analysis of Deviance Table (Type II tests)
Response: flwrs
LR Chisq Df Pr(>Chisq)
tre 263.6 3 < 2.2e-16 ***
wa 4423.8 25 < 2.2e-16 ***
1 | replic 0
tre:wa 531.1 72 < 2.2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
When I try to do a post-hoc with emmeans though I keep running into errors that I can't seem to work through.
This is what I tried based on what I read about emmeans in the help tab of R
emm <- emmeans(negbin, ~ tre | wa)
pairs(emm, adjust = "tukey")
The reponse is this :
> emm <- emmeans(negbin, ~ tre | wa)
Error in `[.data.frame`(tbl, , vars, drop = FALSE) :
undefined columns selected
Error in (function (object, at, cov.reduce = mean, cov.keep = get_emm_option("cov.keep"), :
Perhaps a 'data' or 'params' argument is needed
> pairs(emm, adjust = "tukey")
Error in if (nc < 2L) stop("only one column in the argument to 'pairs'") :
argument is of length zero
So I'm not very sure how to specify parameters or define the columns. Mainly I'm hoping to get the p values for the different tre and wa so that I can add significance values to a plot I created. I'm very new to statistical modelling in R so any suggestions to what I'm doing are welcome!
Suppose you want to evaluate a simple glm model to forecast an economic data series.
Consider the following code:
library(caret)
library(ggplot2)
data(economics)
h <- 7
myTimeControl <- trainControl(method = "timeslice",
initialWindow = 24*h,
horizon = 12,
fixedWindow = TRUE)
fit.glm <- train(unemploy ~ pce + pop + psavert,
data = economics,
method = "glm",
preProc = c("center", "scale","BoxCox"),
trControl = myTimeControl)
Suppose that the covariates used into the train formula are predictions of values obtained by some other model.
This simple model gives the following results:
Generalized Linear Model
574 samples
3 predictor
Pre-processing: centered (3), scaled (3), Box-Cox transformation (3)
Resampling: Rolling Forecasting Origin Resampling (12 held-out with a fixed
window)
Summary of sample sizes: 168, 168, 168, 168, 168, 168, ...
Resampling results:
RMSE Rsquared
1446.335 0.2958317
Apart from the bad results obtained (this is only an example).
I wonder if it is correct:
To consider the above results as results obtained, on the entire dataset, by a GLM trained using only 24*h=24*7 samples and retrained after every horizon=12 samples
How evaluate RMSE as horizon grows from 1 to 12 (as reported here http://robjhyndman.com/hyndsight/tscvexample/ )?
if I show fit.glm summary I obtain:
Call:
NULL
Deviance Residuals:
Min 1Q Median 3Q Max
-5090.0 -1025.5 -208.1 833.4 4948.4
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 7771.56 64.93 119.688 < 2e-16 ***
pce 5750.27 1153.03 4.987 8.15e-07 ***
pop -1483.01 1117.06 -1.328 0.185
psavert 2932.38 144.56 20.286 < 2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for gaussian family taken to be 2420081)
Null deviance: 3999514594 on 573 degrees of freedom
Residual deviance: 1379446256 on 570 degrees of freedom
AIC: 10072
Number of Fisher Scoring iterations: 2
The parameters showed refer to the last trained GLM or are "average" paramters?
I hope I've been clear enough.
This resampling method is like any others. The RMSE is estimated using different subsets of the training data. Note that it says "Summary of sample sizes: 168, 168, 168, 168, 168, 168, ...". The final model uses all of the training data set.
The difference between Rob's results and these are primarily due to the difference between Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE)
I am using rfe in caret to perform feature selection based on the ROC metric from twoClassSummary for a logistic regression model created from an imbalanced dataset (approx 25:1). More often than not I get an error message. Sometimes however I do not.
On the two occasions that the code has run without error (giving a believable result), I have run the exact same rfe line again immediately, and it has failed with this error message:
Error in { : task 1 failed - "undefined columns selected"
(Note that the task number can also vary up to 4.)
myLRFuncs <- lrFuncs
myLRFuncs$summary <- twoClassSummary
rfe.ctrl <- rfeControl(functions = myLRFuncs,
method = "cv",
number = 5,
verbose = TRUE)
train.ctrl <- trainControl(method="none",
classProbs=TRUE,
summaryFunction=twoClassSummary,
verbose=TRUE)
glm_rfe_ROC <- rfe(x=train[,-c("outcome")],y=train$outcome,
sizes=c(1:5, 10, 15, 20, 25),
rfeControl=rfe.ctrl,
method="glm",
metric="ROC",
trControl=train.ctrl)
I am aware that I could use lasso or gradient boosted regression, and so avoid rfe, but I plan to use this approach for a wide range of additional algorithms, so would really like to have this working reliably.
The error seems to be related to how you are subsetting your predictors:
> train <- data.frame(outcome = 1:10, x1 = 1:10, x2 = 1:10)
> train[,-c("outcome")]
Error in -c("outcome") : invalid argument to unary operator
> train(x = train[,-c("outcome")], y = train$outcome)
Error in -c("outcome") : invalid argument to unary operator
Max
I obtained the following code from this Stack Overflow question. caret train() predicts very different then predict.glm()
The following code is producing an error.
I am using caret 6.0-52.
library(car); library(caret); library(e1071)
#data import and preparation
data(Chile)
chile <- na.omit(Chile) #remove "na's"
chile <- chile[chile$vote == "Y" | chile$vote == "N" , ] #only "Y" and "N" required
chile$vote <- factor(chile$vote) #required to remove unwanted levels
chile$income <- factor(chile$income) # treat income as a factor
tc <- trainControl("cv", 2, savePredictions=T, classProbs=TRUE,
summaryFunction=twoClassSummary) #"cv" = cross-validation, 10-fold
fit <- train(chile$vote ~ chile$sex +
chile$education +
chile$statusquo ,
data = chile ,
method = "glm" ,
family = binomial ,
metric = "ROC",
trControl = tc)
Running this code produces the following error.
Something is wrong; all the ROC metric values are missing:
ROC Sens Spec
Min. : NA Min. :0.9354 Min. :0.9187
1st Qu.: NA 1st Qu.:0.9354 1st Qu.:0.9187
Median : NA Median :0.9354 Median :0.9187
Mean :NaN Mean :0.9354 Mean :0.9187
3rd Qu.: NA 3rd Qu.:0.9354 3rd Qu.:0.9187
Max. : NA Max. :0.9354 Max. :0.9187
NA's :1
Error in train.default(x, y, weights = w, ...) : Stopping
In addition: Warning message:
In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo, :
There were missing values in resampled performance measures.
Would anyone know what the issue is or can reproduce/ not reproduce this error. I've seen other answers to this error message that says this has to do with not having representation of classes in each cross validation fold but this isn't the issue as the number of folds is set to 2.
Looks like I needed to install and load the pROC package.
install.packages("pROC")
library(pROC)
You should install using
install.packages("caret", dependencies = c("Imports", "Depends", "Suggests"))
That gets most of the default packages. If there are specific modeling packages that are missing, the code usually prompts you to install them.
I know I'm late to the party, but I think you need to set classProbs = TRUE in train control.
You are using logistic regression when using the parameters method = "glm", family = binomial.
In this case, you must make sure that the target variable (chile$vote) has only 2 factor levels, because logistic regression only performs binary classification.
If the target has more than two labels, then you must set family = "multinomial"
I'm new to WEKA and advanced statistics, starting from scratch to understand the WEKA measures. I've done all the #rushdi-shams examples, which are great resources.
On Wikipedia the http://en.wikipedia.org/wiki/Precision_and_recall examples explains with an simple example about a video software recognition of 7 dogs detection in a group of 9 real dogs and some cats.
I perfectly understand the example, and the recall calculation.
So my first step, let see in Weka how to reproduce with this data.
How do I create such a .ARFF file?
With this file I have a wrong Confusion Matrix, and the wrong Accuracy By Class
Recall is not 1, it should be 4/9 (0.4444)
#relation 'dogs and cat detection'
#attribute 'realanimal' {dog,cat}
#attribute 'detected' {dog,cat}
#attribute 'class' {correct,wrong}
#data
dog,dog,correct
dog,dog,correct
dog,dog,correct
dog,dog,correct
cat,dog,wrong
cat,dog,wrong
cat,dog,wrong
dog,?,?
dog,?,?
dog,?,?
dog,?,?
dog,?,?
cat,?,?
cat,?,?
Output Weka (without filters)
=== Run information ===
Scheme:weka.classifiers.rules.ZeroR
Relation: dogs and cat detection
Instances: 14
Attributes: 3
realanimal
detected
class
Test mode:10-fold cross-validation
=== Classifier model (full training set) ===
ZeroR predicts class value: correct
Time taken to build model: 0 seconds
=== Stratified cross-validation ===
=== Summary ===
Correctly Classified Instances 4 57.1429 %
Incorrectly Classified Instances 3 42.8571 %
Kappa statistic 0
Mean absolute error 0.5
Root mean squared error 0.5044
Relative absolute error 100 %
Root relative squared error 100 %
Total Number of Instances 7
Ignored Class Unknown Instances 7
=== Detailed Accuracy By Class ===
TP Rate FP Rate Precision Recall F-Measure ROC Area Class
1 1 0.571 1 0.727 0.65 correct
0 0 0 0 0 0.136 wrong
Weighted Avg. 0.571 0.571 0.327 0.571 0.416 0.43
=== Confusion Matrix ===
a b <-- classified as
4 0 | a = correct
3 0 | b = wrong
There must be something wrong with the False Negative dogs,
or is my ARFF approach totally wrong and do I need another kind of attributes?
Thanks
Lets start with the basic definition of Precision and Recall.
Precision = TP/(TP+FP)
Recall = TP/(TP+FN)
Where TP is True Positive, FP is False Positive, and FN is False Negative.
In the above dog.arff file, Weka took into account only the first 7 tuples, it ignored the remaining 7. It can be seen from the above output that it has classified all the 7 tuples as correct(4 correct tuples + 3 wrong tuples).
Lets calculate the precision for correct and wrong class.
First for the correct class:
Prec = 4/(4+3) = 0.571428571
Recall = 4/(4+0) = 1.
For wrong class:
Prec = 0/(0+0)= 0
recall =0/(0+3) = 0