comparing a huge vector to a fixed number

Question

I want assess how many components of v=rnorm(10^8) are less than or equal to 0.5. So I wrote this

v=rnorm(10^8)
sum(v<=0.5)

Unfortunately, I'm getting this message

> v=rnorm(10^8)
Error: cannot allocate vector of size 762.9 Mb
> sum(v<=0.5)
Error: object 'v' not found
R(3446,0xa046b540) malloc: *** mmap(size=800002048) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(3446,0xa046b540) malloc: *** mmap(size=800002048) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

Is there any way to do this given that I can't reduce the size of vector v? Thanks

What is our `memory.limit()`? And how many RAM do you have? Cause 1GB should be enough (`memory.limit(size=1000)`) to create vector and 1.5GB to do `sum`. — Marek, May 10 '11 at 08:17

James · Answer 1 · 2011-05-10T15:14:08.437

Since you work through the random number sequence in turn, you can break the vector into chunks, eg:

> set.seed(1)
> rnorm(10)
 [1] -0.6264538  0.1836433 -0.8356286  1.5952808  0.3295078 -0.8204684
 [7]  0.4874291  0.7383247  0.5757814 -0.3053884
> set.seed(1)
> rnorm(5)
[1] -0.6264538  0.1836433 -0.8356286  1.5952808  0.3295078
> rnorm(5)
[1] -0.8204684  0.4874291  0.7383247  0.5757814 -0.3053884

So:

ans <- 0
for (i in 1:10){
    x <- rnorm(10^7)
    ans <- ans + sum(x<=0.5)
    }

At the end of each loop iteration the smaller x should be removed, so you will only need around 76 Mb of memory.

For info, I got:

> ans
[1] 69142375

which compares to:

> pnorm(0.5)
[1] 0.6914625

score 3 · Answer 2 · edited May 23 '17 at 10:24

3

See, among many others: this question, and from within R: ?Memory.

The problem is simply that the vector you want cannot be allocated (on your system), let alone calculate things from it.

edited May 23 '17 at 10:24

Community

1
1

answered May 10 '11 at 07:02

Nick Sabbe

11,684
1
43
57

score 2 · Answer 3 · answered May 10 '11 at 11:29

2

Buy more memory and use 64 bit R.

answered May 10 '11 at 11:29

hadley

102,019
32
183
245

score 2 · Answer 4 · answered May 10 '11 at 12:26

You have not offered details regarding either your OS or your version of R or the number and tasks of other running programs. You should NOT be getting multi-line malloc error reports with instructions on where to set breakpoints with any of the current versions of R. So you may have an out-of-date version of R and should consider getting current if that is the case. When you do not have sufficient memory you should just be getting a single line message as you did when you failed to create v. There does need to be contiguous memory to hold the 10^8 elements and this would involve about 8*10^8 bytes, and with over head might be 0.9 GB.

Try restarting your OS and not loading any other programs. Start a fresh session of R without loading any saved workspace. Generally you will need at least twice the memory of your large object so I would suggest (if on Windows) trying (memory.limit(size=2000)). Then repeat your test.

comparing a huge vector to a fixed number

4 Answers4