1

I am trying to build a large inverse-distance matrix with 227,973 coordinates. Ultimately, I wish to use the results of this as the spatial weights matrix input to a spatial durbin model.

This is my code:

dist_matrix <- as.matrix(dist(cbind(full_df$Longitude,full_df$Latitude)))
  dist_matrix_inv <- 1/dist_matrix
  diag(dist_matrix_inv) <- 0
  ilw <- mat2listw(dist_matrix_inv, style="W")

However, I am getting this error on the first line of the code:

Error: vector memory exhausted (limit reached?)

I am assuming that the problem here lies with storing the large matrix in RAM?

As such, I have tried increasing the memory limit that R has placed, referencing some solutions here: Error: vector memory exhausted (limit reached?) R 3.5.0 macOS -- particularly with setting the parameter: R_MAX_VSIZE=16Gb

However, I still encounter the same error.

Someone has suggested solving submatrices first and then piecing them together at the end to obtain the large matrix here: How to create a Large Distance Matrix?

However, I am not sure how to do that in code. I am also unsure if this would solve the problem of storing a large matrix in the RAM (assuming that this is the problem).

Can anyone advise?

Here is my system, if this is useful to know:

R version 3.5.2 (2018-12-20)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.5

Thank you!

imguessing
  • 377
  • 1
  • 3
  • 9
  • The distance matrix for 227,973 coordinates has a number for each pairwise combination, which amounts to 227973^2 numbers. If each number uses 8 bytes, that would need 48GiB of RAM give or take. Maybe [bigmemory](https://cran.r-project.org/web/packages/bigmemory/index.html) could help, but I'm fairly certain you'd have to code the distance matrix calculation yourself. – Alexis Jun 19 '19 at 22:22
  • @Alexis Thanks for your advice! Have been trying to figure out how to use the bigmemory package. Seems like there isnt much resources out there on this! Any ideas on how one could go about solving the submatrices? Thanks alot for your advice!! – imguessing Jun 20 '19 at 09:33
  • I'd try to use [`Rcpp` with `bigmemory`](https://gallery.rcpp.org/articles/using-bigmemory-with-rcpp/), where you allocate a file-backed matrix and do the Euclidean distance calculations in C++, check that link for an example. – Alexis Jun 20 '19 at 10:54

0 Answers0