1

How to allocate a huge distance matrix in an appropriate way to avoid "allocation is unable" error. Imagine you have a 100.000 points randomly spreaded over some space. How can one cleverly create a matrix or "dist"-object, which represents the the half of DistMatrix. Maybe it should be another object, which will be able efficiently allocate the large number of distances.

You can get the polygonial object from the following link: https://www.dropbox.com/sh/65c3rke0gi4d8pb/LAKJWhwm-l

# Load required packages
library(sp)
library(maptools)
library(maps)

# Load the polygonal object
x <- readShapePoly("vg250_gem.shp")

# Sample or Pick up a large Number of Points
# this command needs some minutes to be done. 
# "coord" is SpatialPoints Object
n <- 1e5
coord <- spsample(x, n, "random")
# Try to measure the distances by dist() 

DistMatrix <- dist(coord@coords)
Error: negative length vectors are not allowed

# Try to measure the distances by spDists()
DistMatrix <- spDists(coord)
Error: cannot allocate vector of size (some number) MB

# It seems that the problem lies on large matrix to be created.

How is this problem solvable in R for great numbers of "n".

And_R
  • 1,647
  • 3
  • 18
  • 32

1 Answers1

0

At this point R cannot allocate the random number of megabytes of RAM. At this point, your computer is using all of its memory somewhere else and there just isn't (some number) of MBytes available for your process to continue. You have several solutions at this point. Among them, get a machine with more RAM, close programs, or do your distance calculations in smaller batches. Try a smaller n; and when it works just repeat the process several times until you have your whole matrix of distances.

Wilmer E. Henao
  • 4,094
  • 2
  • 31
  • 39
  • Saying to do it in small batches you just mean using the spDistsN1() ? Please if you can clarify your idea, it can be useful. Thanks in advance. – And_R Jan 23 '14 at 16:38
  • Hi. I am not familiar with that approach. But if it works for you fine. I actually meant to divide your distance matrix into submatrices and to solve each submatrix one by one. Saving the result each time on your hard drive. Freeing your RAM. – Wilmer E. Henao Jan 23 '14 at 19:12
  • you know some package which speed up simple computitional tasks like sum, sqrt and so on. I have heard that there is a package allowing that, but I couldn't find any. – And_R Jan 27 '14 at 15:34
  • bigmemory package maybe? – Wilmer E. Henao Jan 27 '14 at 17:45