Split big data in R

Question

I have a big data file (~1GB) and I want to split it into smaller ones. I have R in hand and plan to use it.

Loading the whole into memory cannot be done as I would get the "cannot allocate memory for vector of xxx" error message.

Then I want to use the read.table() function with the parameters skip and nrows to read only parts of the file in. Then save out to individual files.

To do this, I'd like to know the number of lines in the big file first so I can workout how many rows should I set to individual files and how many files should I split into.

My question is: how can I get the number of lines from the big data file without fully loading it into R?

Suppose I can only use R. So cannot use any other programming languages.

Thank you.

Anything done in R is fine. Just not using Java or Python or other programming language. — Steve, Jul 03 '15 at 00:46

score 1 · Answer 1 · answered Jul 03 '15 at 00:50

1

Counting the lines should be pretty easy -- check this tutorial http://www.exegetic.biz/blog/2013/11/iterators-in-r/ (the "iterating through lines part). The gist is to use ireadLines to open an iterator over the file

answered Jul 03 '15 at 00:50

Yana K.

1,926
4
19
27

2

This is a good suggestion. But please write a piece of example code that will find out the number of lines in the file, without linking to an external tutorial. – Alex Jul 03 '15 at 00:52

score 1 · Answer 2 · edited May 23 '17 at 11:43

1

For Windows, something like this should work

fname <- "blah.R"  # example file
res <- system(paste("find /v /c \"\"", fname), intern=T)[[2]]
regmatches(res, gregexpr("[0-9]+$", res))[[1]]
# [1] "39"

edited May 23 '17 at 11:43

Community

1
1

answered Jul 03 '15 at 01:04

Rorschach

31,301
5
78
129

Split big data in R

2 Answers2