dealing with large CSV files (20G) in ruby

Question

I am working on little problem and would have some advice on how to solve it: Given a csv file with an unknown number of columns and rows, output a list of columns with values and the number of times each value was repeated. without using any library.

if the file is small this shouldn't be a problem, but when it is a few Gigs, i get NoMemoryError: failed to allocate memory. is there a way to create a hash and read from the disk instead of loading the file to Memory? you can do that in perl with tied Hashes

EDIT: will IO#foreach load the file into memory? how about File.open(filename).each?

Just wondering... did you not accept an answer because none of the solutions helped? Or was there another reason? This question just turned up in my activity again and I wondered. — marcus erronius, Apr 27 '15 at 02:45

score 21 · Answer 1 · answered Dec 12 '12 at 22:46

21

Read the file one line at a time, discarding each line as you go:

open("big.csv") do |csv|
  csv.each_line do |line|
    values = line.split(",")
    # process the values
  end
end

Using this method, you should never run out of memory.

answered Dec 12 '12 at 22:46

marcus erronius

3,613
1
16
32

Jan · Answer 2 · 2012-12-13T07:34:54.513

6

Do you read the whole file at once? Reading it on a per-line basis, i.e. using ruby -pe, ruby -ne or $stdin.each should reduce the memory usage by garbage collecting lines which were processed.

data = {}
$stdin.each do |line|
  # Process line, store results in the data hash.
end

Save it as script.rb and pipe the huge CSV file into this script's standard input:

ruby script.rb < data.csv

If you don't feel like reading from the standard input we'll need a small change.

data = {}
File.open("data.csv").each do |line|
  # Process line, store results in the data hash.
end

edited Dec 13 '12 at 07:34

answered Dec 12 '12 at 22:22

Jan

11,636
38
47

i am looking for a pure ruby version, but this is very helpful – fenec Dec 12 '12 at 22:37
2

@fenec, what makes you consider this version impure? – Jan Dec 12 '12 at 22:41
hahahah maybe because i don't know how to use it, how would i read the file – fenec Dec 12 '12 at 22:42
@fenec, by all definitions, this is a pure ruby version. – sunnyrjuneja Dec 13 '12 at 00:35
Piping in the CSV file might be the part that makes it not pure Ruby. But this method can be applied equally well to a `File.open` initiated data stream – marcus erronius Dec 13 '12 at 02:25

score 1 · Answer 3 · answered Aug 11 '20 at 10:21

1

For future reference, in such cases you want to use CSV.foreach('big_file.csv', headers: true) do |row|

This will read the file line by line from the IO object with minimal memory footprint (should be below 1MB regardless of file size).

answered Aug 11 '20 at 10:21

Krzysztof Karski

749
1
12
19

dealing with large CSV files (20G) in ruby

3 Answers3

Linked