Building a TSV from a file with hundreds of millions of rows

Question

I have an application that needs to build a tsv file from a data file that has a couple hundred million rows.

Right now my code to build the tsv file looks something like this:

File.open(data, 'rb').each { |line|
        row = to_tsv_row(line)) # this formats the row to be delimited by a tab
        open(tsv_path, "a+") { |f| f << row }
      }

This seems like it will be a rather slow way to build a tsv file (perhaps inefficient). Is there a library out there that could do this quickly and efficiently?

how about using `sed` from the command line? http://stackoverflow.com/questions/525592/find-and-replace-inside-a-text-file-from-a-bash-command — Uri Agassi, Jun 20 '14 at 20:17

score 0 · Answer 1 · answered Jun 20 '14 at 21:26

You can use Ruby's CSV library:

require 'csv'
CSV.open(tsv_path, 'w', col_sep: "\t") do |tsv|
  File.open(data, 'rb').each do |line|
    row = to_row(line) # this only needs to convert line to an array of Objects responding to to_s
    tsv << row
  end
end

This takes about 10 seconds to do a million rows on my system, though I think that depends a lot on your data and hardware. A TSV with a hundred million rows sounds like a bad idea in the first place.

Building a TSV from a file with hundreds of millions of rows

1 Answers1