6

I am using the 'bife' package to run the fixed effect logit model in R. However, I cannot compute any goodness-of-fit to measure the model's overall fit given the result I have below. I would appreciate if I can know how to measure the goodness-of-fit given this limited information. I prefer chi-square test but still cannot find a way to implement this either.

    ---------------------------------------------------------------                 
    Fixed effects logit model                   
    with analytical bias-correction                 

    Estimated model:                    
    Y ~ X1 +X2 + X3 + X4 + X5 | Z                   

    Log-Likelihood= -9153.165                   
    n= 20383, number of events= 5104                    
    Demeaning converged after 6 iteration(s)                    
    Offset converged after 3 iteration(s)                   

    Corrected structural parameter(s):                  

        Estimate    Std. error  t-value Pr(> t) 
    X1  -8.67E-02   2.80E-03    -31.001 < 2e-16 ***
    X2  1.79E+00    8.49E-02    21.084  < 2e-16 ***
    X3  -1.14E-01   1.91E-02    -5.982  2.24E-09    ***
    X4  -2.41E-04   2.37E-05    -10.171 < 2e-16 ***
    X5  1.24E-01    3.33E-03    37.37   < 2e-16 ***
    ---                 
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1                  

    AIC=  18730.33 , BIC=  20409.89                     


    Average individual fixed effects= 1.6716                    
    ---------------------------------------------------------------                 
Julius Vainora
  • 47,421
  • 9
  • 90
  • 102
Eric
  • 528
  • 1
  • 8
  • 26
  • 1
    Exactly what kind of goodness-of-fit measures are you after? It's possible to extract residuals from `bife` objects and you may also estimate different specifications. So you are not so restricted after all. – Julius Vainora Nov 10 '18 at 16:45
  • Julius Vainora: I prefer chi-square test. – Eric Nov 11 '18 at 17:45

1 Answers1

8

Let the DGP be

n <- 1000
x <- rnorm(n)
id <- rep(1:2, each = n / 2)
y <- 1 * (rnorm(n) > 0)

so that we will be under the null hypothesis. As it says in ?bife, when there is no bias-correction, everything is the same as with glm, except for the speed. So let's start with glm.

modGLM <- glm(y ~ 1 + x + factor(id), family = binomial())
modGLM0 <- glm(y ~ 1, family = binomial())

One way to perform the LR test is with

library(lmtest)
lrtest(modGLM0, modGLM)
# Likelihood ratio test
#
# Model 1: y ~ 1
# Model 2: y ~ 1 + x + factor(id)
#   #Df  LogLik Df  Chisq Pr(>Chisq)
# 1   1 -692.70                     
# 2   3 -692.29  2 0.8063     0.6682

But we may also do it manually,

1 - pchisq(c((-2 * logLik(modGLM0)) - (-2 * logLik(modGLM))),
           modGLM0$df.residual - modGLM$df.residual)
# [1] 0.6682207

Now let's proceed with bife.

library(bife)
modBife <- bife(y ~ x | id)
modBife0 <- bife(y ~ 1 | id)

Here modBife is the full specification and modBife0 is only with fixed effects. For convenience, let

logLik.bife <- function(object, ...) object$logl_info$loglik

for loglikelihood extraction. Then we may compare modBife0 with modBife as in

1 - pchisq((-2 * logLik(modBife0)) - (-2 * logLik(modBife)), length(modBife$par$beta))
# [1] 1

while modGLM0 and modBife can be compared by running

1 - pchisq(c((-2 * logLik(modGLM0)) - (-2 * logLik(modBife))), 
           length(modBife$par$beta) + length(unique(id)) - 1)
# [1] 0.6682207

which gives the same result as before, even though with bife we, by default, have bias correction.

Lastly, as a bonus, we may simulate data and see it the test works as it's supposed to. 1000 iterations below show that both test (since two tests are the same) indeed reject as often as they are supposed to under the null.

enter image description here

Julius Vainora
  • 47,421
  • 9
  • 90
  • 102
  • Thank you. I know how the degrees of freedom is extracted to be used in glm but not sure how it works the same for bife function's length(modBife$par$beta) you specified. If I am trying to just take out the loglikelihood values simply to input in the pchisq function without using any buildup function you hypothetically created, from which part shall I take out the values for length(modBife$par$beta)? – Eric Nov 11 '18 at 21:19
  • 1
    The vector `modBife$par$beta` contains all the beta coefficients (not fixed effects, no intercept). When testing `modBife0` (full model) vs. `modBife` (only fixed effects), it is exactly those beta coefficients that we set to zero. So, if I understand your question correctly, `length(modBife$par$beta)` in the the regular output would correspond to the number of variables: 5 in your example (`X1`, ..., `X5`). – Julius Vainora Nov 11 '18 at 21:24
  • 1
    I'll also jump ahead to explain `length(modBife$par$beta) + length(unique(id)) - 1`. Here we are testing the full model against only the intercept. Then the reason for `length(modBife$par$beta)` remains the same. Next, we set all the fixed effects to zero, and there are `length(unique(id))` of them. But in the full model we also don't have the intercept. So, from `length(unique(id))` non-beta coefficients we go to 1 (intercept), hence the `length(unique(id)) - 1` degrees of freedom. – Julius Vainora Nov 11 '18 at 21:31
  • Thank you. If length(unique(id)) goes to 1 as it is the intercept as you said and subtract 1 again to get the degrees of freedom, wouldn't that be again length(modBife$par$beta) itself as before? If so, I think the answers should be the same? – Eric Nov 11 '18 at 21:38
  • 1
    In my example, for instance, we have a1*id1 + a2*id2 + b1*x in the full model (where a1 and a2 are fixed effects, id1 and id2 are individual dummy variables). Then the minimal model would be just intercept*1. So, the number of degrees of freedom = 2 = 1 (beta) + 2 (fixed effects) - 1 (intercept is coming back). In other words, while there are `length(unique(id))` fixed effects, we lose one degree of freedom due to the restriction that all those dummy variables always sum to 1. – Julius Vainora Nov 11 '18 at 21:42
  • My id (so called Z in my example) is not part of my regressors. It is treated as time variable which is not included in any beta coefficients' measures. When I run length(regress$par$beta) over my model called 'regress' as in my example above, I get 5. However, as you can see from my example, the Z which corresponds to id what you make is not included in the regressors. So if I run on the null model that sets Y~1, using length(unique(Z)) I get 207 which does not make any sense. Or it make sense to you? – Eric Nov 11 '18 at 21:53
  • Anyway what ever it is, I get 0 if I follow your bottom method which seems to be a good sign. Then I just would like to check whether ((-2 * logLik(modGLM0)) - (-2 * logLik(modBife))) is indeed the chi-square value I can report. For my case I get 2711.83 for this chi-square value and wonder whether this can be ok to report. Probably length(unique(Z)) = 207 makes sense then. – Eric Nov 11 '18 at 21:59
  • Then lastly may I know how I can get the R2 value for this as well please? – Eric Nov 11 '18 at 22:00
  • Anyway I think length(unique(Z))=207 can make sense since my unique id length is indeed 207 although Z is not part of my regressors as in your bife example above while you put in your glm example with a plus sign as part of the regressors. – Eric Nov 11 '18 at 22:08
  • 2
    @Eric, re 1st comment: `length(unique(Z))` being 207 should make sense (I'll add test simulation results today or tomorrow in this case). Re 2nd comment: right, that's a value to report, and indeed chi square takes larger value with more degrees of freedom. My id is just like your Z: they are fixed effects (dummy variables for each individual or, in your case, each time period) with estimated values given at `modBife$par$alpha`. Re R^2: in logistic regression there no longer is a clear R^2; there are multiple proposals. One is McFadden's R^2, given by `c(1- logLik(modBife) / logLik(modGLM0))`. – Julius Vainora Nov 11 '18 at 22:29
  • 1
    @Eric, coming back to `length(unique(Z))`, everything is fine, except you need to keep in mind the ratio `n / length(unique(Z))`. The larger it is the better. Otherwise the test may perform poorly. – Julius Vainora Nov 12 '18 at 18:30
  • Thank you so much again Julius! – Eric Nov 12 '18 at 18:52
  • Your second bife models yields the warning "Error in `[[<-.data.frame`(`*tmp*`, "x", value = raw(0)) : replacement has 0 rows, data has 1000" Was there an update due to which this code doesn't work anymore? – Jakob Sep 11 '19 at 10:35
  • @PeterPan, apparently yes. – Julius Vainora Sep 12 '19 at 16:12
  • @JuliusVainora alright, lets give the authors some time, its still pre-version 1. Maybe the now omitted statistics show up in the next update again :) – Jakob Sep 16 '19 at 11:39