1

I'm facing a pretty expected problem while I'm running irritatingly the below code which creates all possible combinations for a specified sequence and then it stores them in the final.grid variable. The thing is that there is no only one sequence but about hundred of thousands of them and each one could have enough combinations. 

for()...
   combs = get.all.combs(sequence)
   final.grid = rbind(final.grid, combs)

Anyway. Tried to run my code in a windows PC with 4GB RAM and after 4 hours (not even half of the combinations being calculated) R returned this error

Error: cannot allocate vector of size 4.0 Gb

What i was though as solution is to write after each iteration the final.grid to a file , free the allocated memory and continue. The truth is that I have not experience on such implementations with R and I don't know which solution to choose and if there are some of them that will do better and more efficiently. Have in mind that probably my final grid will need some GBs.

Somewhere in the stack exchange I read about ff package but there was not enough discussion on the subject (at least I didn't found it) and preferred to ask here for your opinions.

Thanks

J. Doe
  • 619
  • 4
  • 16

2 Answers2

0

I cannot understand very well your question, because the piece of code that you put is not clear to figure it out your problem.

But, you can try saving your results as .RData or .nc files, depending on the nature of your data. However, it could be better if you are more explicit about your problem, for instance showing what code is behind get.all.combs function or sequence data.

Wencheng Lau-Medrano
  • 630
  • 4
  • 12
  • I'dont think that posting the whole code here will help solve the problem. Function get.all.combs() returns a huge matrix with 4 columns and a lot of rows. This matrix then is appended (fore each sequence) to the final.grid matrix which can reach some GB's. – J. Doe Jul 15 '17 at 15:18
0

One thing you can try is the memory.limit() function to see if you can allocate enough memory for your work. This may not work if your Windows OS is 32 bit.

If you have large data object that you don't need for some parts of your program, you could first save them, and them remove using 'rm', and when you need them again you can load the objects.

The link below has more info that could be useful to you. Increasing (or decreasing) the memory available to R processes

EDIT: You can use object.size function to see memory requirement for objects you have. If they are too big, try loading them only when you need them.

It is possible one of the functions you use try to allocate more memory than you have. See if you can try to find where exactly the program crashes.

kangaroo_cliff
  • 6,067
  • 3
  • 29
  • 42
  • I'll give a try to the memory.limit() by setting it at 3GB's and see what is going to happen. – J. Doe Jul 15 '17 at 15:26
  • If you are setting it at 3 Gb, then that means you are reducing the size of memory available to R. That's isn't going to help. – kangaroo_cliff Jul 16 '17 at 02:03
  • You were right. Nothing changed. About your edit : I don't want to load anything. I'm just having an object(matrix) that through iterations is getting bigger and bigger until it fills the memory. Now I'm going to try split this object into many parts of 2Gb. – J. Doe Jul 16 '17 at 10:25
  • By loading, I meant keep them (big objects) in the memory only when you need them. You can save and remove them until you need them, and then load them. – kangaroo_cliff Jul 16 '17 at 11:21