0

I am analysing a dataset with over 450k rows about 100k rows in one of the columns I am looking at (pa1min_) has NA values, due to non-responses and other random factors. This column deals with workout times in minutes.

I don't think it makes sense to fill the NA values with the mean or median given that it's nearly a quarter of the data and the biases that could potentially create. I would like to impute the missing observations with a linear regression. However, I receive an error message:

Error: vector memory exhausted (limit reached?)
In addition: There were 50 or more warnings (use warnings() to see the first 50)

This is my code:

# imputing using multiple imputation deterministic regression
imp_model <- mice(brfss2013, method="norm.predict", m=1)
# store data
data_imp <- complete(imp_model)
# multiple imputation
imp_model <- mice(brfss2013, m=5)
# building predictive mode
fit <- with(data=imp_model, lm(y ~ x + z))
# combining results
combined <- pool(fit)

Here is a link to the data (compressed) Data

Note: I really just want to fill impute for one column...the other columns in the dataframe are a mixture of characters, integers and factors, some with more than 2 levels.

slamballais
  • 3,161
  • 3
  • 18
  • 29
Emm
  • 2,367
  • 3
  • 24
  • 50
  • Sounds like you just need a computers with more RAM. Hard to say though without some sort of [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) – MrFlick Feb 20 '19 at 19:31
  • @MrFlick added a link to the compressed data – Emm Feb 20 '19 at 20:00

1 Answers1

2

Similar to what MrFlick mentioned, you are somewhat short in RAM.

Try running the algorithm on 1% of your data, and if you succeed, you should try checking out the bigmemory package for doing in-disk computations.

I also encourage you to check if the model you fit on your data is actually good without bayesian imputation, because the fact of trying to have perfect data could not be much more beneficial than just imputating mean/median/first/last values on your data.

Hope this helps.

Just Burfi
  • 159
  • 9
  • Thanks, tried running the algo on under 1% of my data and got this error message -> Error in solve.default(xtx + diag(pen)) : system is computationally singular: reciprocal condition number = 1.56244e-26 In addition: There were 50 or more warnings (use warnings() to see the first 50) – Emm Feb 20 '19 at 20:39
  • @Emm I recommend you to check this post :) https://stackoverflow.com/questions/50928796/system-is-computationally-singular-reciprocal-condition-number-in-r – Just Burfi Feb 20 '19 at 20:45
  • What I try to explain to you with the above post is that the problem cannot be solved because the matrix isn't invertible (check answer) due to the determinant of the matrix being zero according to machine precision – Just Burfi Feb 20 '19 at 20:47