2

I have been trying to upload many files into R using several different methods that have worked with me in the past, but for some reason are not here. I have read many posts on the forum that address the different ways this can be done but none of them seem to work for my problem; the files are larger I suppose.

Here are the different things I have tried:

files <- list.files(pattern = ".txt")

listOfFiles <- list()

for(i in 1:length(files)){
 listOfFiles[[i]] <- read.table(files[i], header = TRUE, sep = "\t", stringsAsFactors = FALSE)
}

However, when I run this, my computer just freezes and ceases to work. This has led me to believe that it may be a memory issue however, I have tried changing the memory.limit() to about 12000 and it still does not run.

There is a posting here that sort of addresses the issue at hand: Quickly reading very large tables as dataframes. Reasons why it differs is that I know that the scripts I have uploaded work, just not on many files totaling more than 2GB. I believe this is a memory issue because, when I ran it again I got the error:

Error: cannot allocate vector of size 7.8 Mb

I have read other posts on the forum that use lapply, so thought I'd try it out however, it has also failed to work.

Here is what I did:

listo <- lapply(files, read.table)

This on the other hand runs, but when I try and open the list listo it gives me the error:

Error: object 'listo' not found

Any help would be much appreciated.

h3ab74
  • 318
  • 3
  • 16
  • 1
    Maybe fread from package Data.table could help here? It should be faster than read.table. – TinglTanglBob Mar 15 '19 at 14:38
  • 1
    Possible duplicate of [Quickly reading very large tables as dataframes](https://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes) – divibisan Mar 15 '19 at 15:01
  • I am trying fread now after changing the memory limit again, will let you know if it works! Thank you all for your time though. – h3ab74 Mar 15 '19 at 15:20
  • @TinglTanglBob It has worked with fread! Thank you so much. – h3ab74 Mar 15 '19 at 15:54
  • 1
    If you've figured it out, it's better to post that as an answer (you can answer your own questions, and even accept your own answer) than as an edit to the question – camille Mar 15 '19 at 22:01

1 Answers1

0

Thank you @TinglTanglBob for your help in solving this question.

Here is the how I solved it:

memory.limit(size = 12000)
files <- list.files(pattern = ".txt")
YFV_list <- list()

for(i in 1:length(files)){
  YFV_list[[i]] <- fread(files[i], sep = "\t", header = TRUE, stringsAsFactors = FALSE)
}

So I'm assuming was a memory issue. Using fread from the Data.table package helped overcome this problem as it was not working earlier with read.table. However, some tweaking needed to be done to the memory.limit for this to work regardless.

h3ab74
  • 318
  • 3
  • 16