0

I'm trying to run quantreg with 2 independent variables over the 12 555 029 cases. I have 16GB RAM on computer, 64Bit OS. Command memory.limit() returned 16 264. Command sessionInfo() returned:

R version 3.5.2 (2018-12-20)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2

During the execution of quantreg only up to 4GB is used. I saw this in task manager. It is not finished for more than one hour and I stopped it.

Why quantreg is not using other RAM available? How I can increase using of RAM memory?

Boriana
  • 1
  • 6

1 Answers1

2

R cannot use the all RAM available on your system. There are some tricks to overcome limited RAM issue in R. R stores all data on the RAM so this is why the size of the data can be analyzed is limited to the amount of the RAM. What we should do is to reduce the size of the data and choose more efficient data types and some tricks to overcome this problem.

1 - Depends on the type of your data, you may change it's type. For instance, numeric values take more memory than integers.

2 - Use factors instead of characters where it is possible. Factors are more efficient because they store the levels instead of the whole vector.

3 - Use bit vectors if you have any logical variable without NA value. They take up only 1 bit while logical variables take 4 to 32 bits.

4 - Reuse your objects instead of creating new ones where it is possible and make sure that you call gc() when you delete some objects.

5 - You can save your big sized data, delete it and reload it when it is necessary.

6 - Have a look at bigmemory and ff packages.

7 - Close all other processes on your system because they are taking some amount of your RAM.

And you can have a look at the book titled 'R High Performance Programming' by Lim and Tjhi.

Hope it helps.

boyaronur
  • 521
  • 6
  • 18
  • I have only numeric variables 2 independent and one dependent.They are not logical. The packages for big memory is not suitable for quantreg. In first 10 minutes function is used around 4GB and continue for a long time with less than 1GB. There aren't the result yet. – Boriana Jan 21 '19 at 13:12
  • is it data frame? if so change it to matrix. – boyaronur Jan 21 '19 at 13:14
  • I tried but it didn't work. I do it in Python. There are no problem with memory. – Boriana Jan 23 '19 at 12:29