Stack Overflow
Stack Overflow

Questions tagged [model-comparison]

A task in statistics and machine learning for comparing models that are fit to the same data with the aim of determining which best explains the data. Questions about how to use model comparision to evaluate models are likely more appropriate for CrossValidated (https://stats.stackexchange.com)!

Model comparison is a task in model selection involving the application of a measure (criterion) to the fits of a set of candidate models to data. The resulting measurements enable a quantitative comparison of appropriateness of the models in explaining the data.

See the Model Selection Wikipedia entry.

98 questions
35
votes
3 answers

What is a threshold in a Precision-Recall curve?

I am aware of the concept of Precision as well as the concept of Recall. But I am finding it very hard to understand the idea of a 'threshold' which makes any P-R curve possible. Imagine I have a model to build that predicts the re-occurrence (yes…
Mr.A
  • 373
  • 1
  • 3
  • 5
9
votes
1 answer

Compute a kernel ridge regression in R for model selection

I have a dataframe df df<-structure(list(P = c(794.102395099402, 1299.01021921817, 1219.80731174175, 1403.00786976395, 742.749487463385, 340.246973543409, 90.3220586792255, 195.85557320714, 199.390867672674, 191.4970921278, 334.452413539092,…
SimonB
  • 670
  • 1
  • 10
  • 25
6
votes
0 answers

Structure of the random effects in glmmLasso

I want to perform model selection among ~150 fixed-effect and 7 random-effect variables, on a set of 360 observations. I decided to use the Lasso procedure for mixed models, with the glmmLasso. I did a lost of researches to find some examples of…
T. Ravessoud
  • 61
  • 4
5
votes
2 answers

AIC different between biglm and lm

I have been trying to use biglm to run linear regressions on a large dataset (approx 60,000,000 lines). I want to use AIC for model selection. However I discovered when playing with biglm on smaller datasets that the AIC variables returned by…
user3303687
  • 53
  • 2
4
votes
0 answers

Python stepwise regression with AIC?

what is the Python equivalent for R step() function of stepwise regression with AIC as criteria? Is there an existing function in statsmodels.api?
Chenying Gao
  • 310
  • 4
  • 14
3
votes
2 answers

Model comparison for breakpoint time series model in R strucchange

I want to test whether a time series contains structural changes or not. Using this simulated example creates a series with two breaks after 30 and 80 observations. set.seed(42) sim_data = data.frame(outcome = c(rnorm(30, 10, 1), rnorm(50, 20, 2),…
ben_aaron
  • 1,504
  • 2
  • 19
  • 39
3
votes
1 answer

Model selection in R, all models giving the same AIC and BIC

So this is the head of my data, thickness grains resistivity 1 25.1 14.9 0.0270 2 368.4 58.1 0.0267 3 540.4 77.3 0.0160 4 712.1 95.6 0.0105 5 883.7 113.0 0.0090 6 1055.7 130.0 …
Emma Smith
  • 381
  • 1
  • 4
  • 9
3
votes
1 answer

how to loop through multiple models to put in a list in R

I have 117 models, named m1, m2, m3, ..., m117. I need to compare their AIC using model.sel in the MuMIn package. The syntax for model comparison is model.sel(object, ...) in which the object is the list of models. Thus, I tried to create a loop to…
le nghiem
  • 33
  • 1
  • 4
3
votes
0 answers

Using BIC,AIC for estimating number of clusters in document clustering using Kmeans

In my approach I am trying to find the optimal value of 'k' for clustering a set of documents using KMEANS algorithm. I wanted to use 'AIC' and 'BIC' information criterion function for finding the best model. I used this resource…
Raghav
  • 41
  • 4
3
votes
1 answer

Rank a list of models based on AIC values

After applying a model between one response variable and several exlanatory variables across a dataframe, I would like to rank each model by the AIC score. I have encountered a very similar question that does exactly what I want to do. Using…
James White
  • 705
  • 2
  • 7
  • 20
3
votes
1 answer

Subsetting in dredge (MuMIn) - must include interaction if main effects are present

I'm doing some exploratory work where I use dredge{MuMIn}. In this procedure there are two variables that I want to set to be allowed together ONLY when the interaction between them is present, i.e. they can not be present together only as main…
ego_
  • 1,409
  • 6
  • 21
  • 31
2
votes
0 answers

How to compare GAMs that include random effects and factors?

I am trying to make a model comparison (say, for hypothesis testing) of two GAMs (mgcv package), where both models include random effects smooth term (s(bs="re")), and the second model additionally include a factor variable. So: gm0 <- gam(y ~…
Kamil Bartoń
  • 1,482
  • 9
  • 10
2
votes
2 answers

Histogram of AIC for each models

Hello How can I create a histogram for the difference of the AICs of each models to the AIC of the full model.? #AIC of the full model Y <- modelTT$aic #AICs for each of the n models. X <- lapply(listOfModels,function(xx) xx$aic) so basically I…
Bustergun
  • 977
  • 3
  • 11
  • 17
2
votes
1 answer

AIC in R: differences in manual vs. internal value when using weighted data

I am attempting to use R for model selection based on the AIC statistic. When comparing linear models with or without weighting, my code in R informs me that weighting is preferable compared to no-weighting, and these results are confirmed in other…
Ron
  • 80
  • 1
  • 1
  • 6
2
votes
1 answer

step function matching AIC of full model

Shouldn't the AIC(full) = 275.93 match the output of of the AIC when the step() function runs with the full model which is -9.86 below Start: AIC=-9.86 y ~ x + x2 Df Sum of Sq RSS AIC - x2 1 0.03672 85.372 -11.8147 - x 1 …
user3022875
  • 8,598
  • 26
  • 103
  • 167
1
2 3 4 5 6 7