0

I am trying to read a large space-delimited file (14Gb) of 49,376 rows and 73,625 columns into R for analysis.

I have tried using fread from the data.table package, as suggested here.

I receive the error 

Error: segfault from C stack overflow

Is there another approach that could be used here? Any other packages or some kind of work around for this error? My R session info is below.

R version 3.0.2 (2013-09-25)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US       LC_NUMERIC=C         LC_TIME=en_US       
 [4] LC_COLLATE=en_US     LC_MONETARY=en_US    LC_MESSAGES=en_US   
 [7] LC_PAPER=en_US       LC_NAME=C            LC_ADDRESS=C        
[10] LC_TELEPHONE=C       LC_MEASUREMENT=en_US LC_IDENTIFICATION=C 

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.9.4

loaded via a namespace (and not attached):
[1] chron_2.3-45 tools_3.0.2
Community
  • 1
  • 1
user3745089
  • 139
  • 1
  • 9
  • Do you get stack overflow using [1.9.5, the current development version](https://github.com/Rdatatable/data.table/wiki/Installation) as well? If so, could you please file an issue there? Thanks. – Arun Jan 14 '15 at 15:24
  • Perhaps also update your R version to the latest as an aside..? – talat Jan 14 '15 at 15:32
  • Unfortunately this is running on a server, and I do not have the ability to install a new R version. The development version does not fix the issue. – user3745089 Jan 14 '15 at 15:36
  • You could look into the [package ff](http://cran.r-project.org/web/packages/ff/index.html) which is aimed at handling large amounts of data. – nrussell Jan 14 '15 at 15:38
  • 1
    The problem was memory limit on the server. Once increased, fread worked as expected. – user3745089 Jan 14 '15 at 16:12
  • 1
    @user3745089, could you answer and accept it so that it remains answered? Thanks. – Arun Jan 14 '15 at 16:50

1 Answers1

0

The error was occurring due to insufficient memory. Once I increased the memory limit, fread worked as expected.

user3745089
  • 139
  • 1
  • 9