Here's a benchmark against a real-life log file. Of the methods used to read the file, only the one using foreach is scalable because it avoids slurping the file.
Using lazy adds overhead, resulting in slower times than map alone.
Notice that foreach is right in there as far as processing speed goes, and results in a scalable solution. Ruby won't care if the file is a zillion lines or a zillion TB, it's still only seeing a single line at a time. See "Why is "slurping" a file not a good practice?" for some related information about reading files.
People often gravitate to using something that pulls in an entire file at once, then splitting it into parts. That ignores the job Ruby then has to do to rebuild the array based on line ends using split or something similar. That adds up, and is why I think foreach pulls ahead.
Also notice that the results shift a little between the two benchmark runs. This is probably due to system tasks running on my Mac Pro as the jobs are running. The important thing is that shows the difference is a wash, confirming to me that using foreach is the right way to process big files, because it's not going to kill the machine if the input file exceeds available memory.
require 'benchmark'
REGEX = /\bfoo\z/
LOG = 'debug.log'
N = 1
# each_line: "Splits str using the supplied parameter as the record separator
# ($/ by default), passing each substring in turn to the supplied block."
#
# Because the file is read into a string, then split into lines, this isn't
# scalable. It will work if Ruby has enough memory to hold the string plus all
# other variables and its overhead.
def lazy_map(filename)
File.open("lazy_map.out", 'w') do |fo|
fo.puts File.readlines(filename).lazy.map { |li|
li.gsub(REGEX, 'bar')
}.force
end
end
# each_line: "Splits str using the supplied parameter as the record separator
# ($/ by default), passing each substring in turn to the supplied block."
#
# Because the file is read into a string, then split into lines, this isn't
# scalable. It will work if Ruby has enough memory to hold the string plus all
# other variables and its overhead.
def map(filename)
File.open("map.out", 'w') do |fo|
fo.puts File.readlines(filename).map { |li|
li.gsub(REGEX, 'bar')
}
end
end
# "Reads the entire file specified by name as individual lines, and returns
# those lines in an array."
#
# As a result of returning all the lines in an array this isn't scalable. It
# will work if Ruby has enough memory to hold the array plus all other
# variables and its overhead.
def readlines(filename)
File.open("readlines.out", 'w') do |fo|
File.readlines(filename).each do |li|
fo.puts li.gsub(REGEX, 'bar')
end
end
end
# This is completely scalable because no file slurping is involved.
# "Executes the block for every line in the named I/O port..."
#
# It's slower, but it works reliably.
def foreach(filename)
File.open("foreach.out", 'w') do |fo|
File.foreach(filename) do |li|
fo.puts li.gsub(REGEX, 'bar')
end
end
end
puts "Ruby version: #{ RUBY_VERSION }"
puts "log bytes: #{ File.size(LOG) }"
puts "log lines: #{ `wc -l #{ LOG }`.to_i }"
2.times do
Benchmark.bm(13) do |b|
b.report('lazy_map') { lazy_map(LOG) }
b.report('map') { map(LOG) }
b.report('readlines') { readlines(LOG) }
b.report('foreach') { foreach(LOG) }
end
end
%w[lazy_map map readlines foreach].each do |s|
puts `wc #{ s }.out`
end
Which results in:
Ruby version: 2.0.0
log bytes: 733978797
log lines: 5540058
user system total real
lazy_map 35.010000 4.120000 39.130000 ( 43.688429)
map 29.510000 7.440000 36.950000 ( 43.544893)
readlines 28.750000 9.860000 38.610000 ( 43.578684)
foreach 25.380000 4.120000 29.500000 ( 35.414149)
user system total real
lazy_map 32.350000 9.000000 41.350000 ( 51.567903)
map 24.740000 3.410000 28.150000 ( 32.540841)
readlines 24.490000 7.330000 31.820000 ( 37.873325)
foreach 26.460000 2.540000 29.000000 ( 33.599926)
5540058 83892946 733978797 lazy_map.out
5540058 83892946 733978797 map.out
5540058 83892946 733978797 readlines.out
5540058 83892946 733978797 foreach.out
The use of gsub is innocuous since every method uses it, but it's not needed and was added for a bit of frivolous resistive loading.
