0

Suppose I have a very large matrix that I read in and I want to transpose it in R. If I try to transpose it I get memory errors. Is there any library in R that allows you cache the original dataset? Would taking "chunks of data", transposing each dataset, and then appending it be an efficient approach?

data = fread("C:/Users/ee2/Desktop/Science/data.tsv")
data_t=data[, .N, by = list(A, V)]
Warning messages:
1: 
  Reached total allocation of 16274Mb: see help(memory.size)
smci
  • 32,567
  • 20
  • 113
  • 146
lord12
  • 2,837
  • 9
  • 35
  • 47
  • Does this help http://stackoverflow.com/questions/7262485/options-for-caching-memoization-hashing-in-r ? – symbolrush Jan 11 '16 at 20:00
  • It gives a good introduction on how to perform caching with simple operations but how can I apply this to data transposing? – lord12 Jan 12 '16 at 03:57
  • What are its dimensions? Ultimately what are you trying to compute? If the matrix is this large then surely it'll cause grief when you use it in computation. You could look into sparse matrix representations. If the only issue is memory used by `data.table::fread()` (and it almost surely isn't), then make sure to use colClasses ; also stringsAsFactors=F. – smci May 11 '16 at 00:04
  • Anyway, please give us reproducible example code (with random integers and seed). – smci May 11 '16 at 00:05
  • Can you transpose it at all by any means necessary? – djechlin May 11 '16 at 00:13

0 Answers0