2

I encounter some discrepancies when comparing the deviance of a weighted and unweigthed model with the AIC values. A general example (from ‘nls’):

DNase1 <- subset(DNase, Run == 1)
fm1DNase1 <- nls(density ~ SSlogis(log(conc), Asym, xmid, scal), DNase1)

This is the unweighted fit, in the code of ‘nls’ one can see that ‘nls’ generates a vector wts <- rep(1, n).

Now for a weighted fit:

 fm2DNase1 <- nls(density ~ SSlogis(log(conc), Asym, xmid, scal), 
                 DNase1, weights = rep(1:8, each = 2))

in which I assign increasing weights for each of the 8 concentrations with 2 replicates.

Now with deviance I get:

deviance(fm1DNase1)
[1] 0.004789569

> deviance(fm2DNase1)
[1] 0.0164259

telling me that the weighted fit has a significantly higher deviance (is a worse fit).

Now with AIC (or BIC) I get

> AIC(fm1DNase1)
[1] -76.41642

> AIC(fm2DNase1)
[1] -372.5437

which tells me that the second fit is by orders of magnitude the better one (lower AIC). Why so?

If I define AIC based on residual sum-of-squares as found in the textbooks

RSS <- function (object) 
{
    w <- object$weights
    r <- residuals(object)
    if (is.null(w)) 
        w <- rep(1, length(r))
    sum(w * residuals(object)^2)
}

AICrss <- function(object)
{
  n <- nobs(object)
  k <- length(coef(object))
  rss <- RSS(object)
  n * log((2 * pi)/n) + n + 2 + n * log(rss) + 2 * k
}

I get

> AICrss(fm1DNase1)
[1] -76.41642

which is the same value as the above AIC (stats:::AIC.logLik) based on log-likelihood

but

> AICrss(fm2DNase1)
[1] -56.69772

which is higher and fits perfectly to the also higher deviance of the second model.

Could anyone enlighten me? Is the standard AIC implementation for ‘nls’ models not applicable in case of weighted fitting?

Cheers, Andrej

merv
  • 67,214
  • 13
  • 180
  • 245
A.N. Spiess
  • 21
  • 1
  • 2
  • 1
    I have been [advised on Cross Validated](http://stats.stackexchange.com/questions/35601/how-to-read-the-the-goodness-of-fit-on-nls-of-r#comment71402_35601) that AIC is not generally applicable to `nls` fits. – Roland Sep 24 '12 at 14:47
  • I didn't understand why you thought two models with deviances of 0.004789569 and 0.0164259 would be "significantly different". Is the deviance function even meaningful for nls fits? And if it is meaningful, wouldn't you need to have a difference of greater than 3.84 (and maybe more if the degrees of freedom were higher) to reach significance? – IRTFM Sep 24 '12 at 17:31
  • 1
    what happens if you scale your weights so they sum to 1? I suspect that R is (rightly or wrongly) treating the weights as though they correspond to having more data ... – Ben Bolker Sep 24 '12 at 17:48
  • Aaah, thanks guys, when I standardize so all weights sum up to 1 I get with stats:::AIC.logLik AIC(fm1DNAse) = 544, AIC(fm2DNase1) = 589, which also favors the unweighted model. Should this not be noted in the 'nls' doc that you have to use weights that sum up to one? Thanks again! – A.N. Spiess Sep 25 '12 at 06:38
  • 2
    move comment to answer, please! – te_ar Nov 07 '12 at 10:11
  • Intuitively, if you weight something, it must lead to a better, fit isn't it? – Eric Mar 13 '18 at 14:06

0 Answers0