Rails how to get the row count from CSV fast

Question

I have tried length method to get the row count and it took at least 2 mins to return the row count. Length method is fast up to 10 000 and it's too slow when it exceeds 100 000 rows. Is there a way to get the row count with in 2 sec even file contains over 100k rows?

I have used the following code.

file = params[:file]
f = File.open(file.path, 'r:iso-8859-1:utf-8')
csv_file = CSV.parse(f.read, :headers => true, :converters => :all, :header_converters => lambda {|h| h.to_s.include?(' ') ? h.downcase.parameterize.underscore : h})
csv_file.length

https://stackoverflow.com/questions/2650517/count-the-number-of-lines-in-a-file-without-reading-entire-file-into-memory — AbM, Nov 30 '17 at 14:37
https://stackoverflow.com/questions/4662438/count-the-length-number-of-lines-of-a-csv-file — seancdavis, Nov 30 '17 at 16:23

score 0 · Accepted Answer · answered Dec 14 '17 at 12:49

0

Eventually I found solution, CSV.read(file.path).length is faster than CSV.parse(f.read).length

answered Dec 14 '17 at 12:49

Premanandh Selvakumarasamy

1,332
7
27

Your question is about speed, but there can be memory problem if you have over 100k rows and use `CSV.read` so I'm still looking for better solution – kangkyu Nov 27 '18 at 01:36

Rails how to get the row count from CSV fast

1 Answers1