Benchmarking methods in Ruby

Question

I am trying to benchmark a set of computations like so -

def benchmark(func, index, array)
    start = Time.now
    func(index, array)
    start - Time.now #returns time taken to perform func
end

def func1(index, array)
    #perform computations based on index and array
end 

def func2(index, array)
    #more computations....
end

benchmark(func1, index1, array1)
benchmark(func1, index2, array2)

Now I'm wondering how can I achieve this. I tried this example, but is spits out

`func1': wrong number of arguments (0 for 2) (ArgumentError)

If I try -

benchmark(func1(index1, array1), index1, array1)

It spits out...

undefined method `func' for main:Object (NoMethodError)

I saw a similar question asked about it, but it was for python. Passing functions with arguments to another function in Python? Can someone assist? Thanks.

Answer 1

In Ruby, methods can be called without including empty parentheses after the method name, like so:

def func1
  puts "Hello!"
end

func1 # Calls func1 and prints "Hello!"

Because of this, when you write benchmark(func1, index1, array1), you're actually calling func1 with no arguments and passing the result to benchmark, not passing func1 to the benchmark function as you expected. In order to pass func1 as an object, you may obtain a wrapper object for the function using the method method, like this:

def func1
  puts "Hello!"
end

m = method(:func1) # Returns a Method object for func1
m.call(param1, param2)

Most of the time though, that's not something you really want to do. Ruby supports a construct called blocks which is much better suited for this purpose. You may already be familiar with blocks from the each iterator Ruby uses for looping through arrays. Here's what it would look like to use blocks for your use case:

def benchmark
  start = Time.now
  yield
  Time.now - start # Returns time taken to perform func
end

# Or alternately:
# def benchmark(&block)
#   start = Time.now
#   block.call
#   Time.now - start # Returns time taken to perform func
# end

def func1(index, array)
    # Perform computations based on index and array
end 

def func2(index, array)
    # More computations....
end

benchmark { func1(index1, array1) }
benchmark { func1(index1, array2) }

In fact, Ruby has a standard library for benchmarking called Benchmark which uses blocks and probably already does exactly what you want.

Usage:

require 'benchmark'

n = 5000000
Benchmark.bm do |x|
  x.report { for i in 1..n; a = "1"; end }
  x.report { n.times do   ; a = "1"; end }
  x.report { 1.upto(n) do ; a = "1"; end }
end

The result:

    user     system      total        real
1.010000   0.000000   1.010000 (  1.014479)
1.000000   0.000000   1.000000 (  0.998261)
0.980000   0.000000   0.980000 (  0.981335)

Answer 2

Here's how I do it. I begin by creating a module contain all the methods to be tested:

module Methods
  def bob(array)
    ...
  end

  def gretta(array
    ...
  end

  def wilma(array)
    ...
  end
end

then I include the module and put the methods into an array:

include Methods
@methods = Methods.public_instance_methods(false)
  #=> [:bob, :gretta, :wilma]

That allows me to execute send(meth, *args) for each method meth in @methods.

Here's an example of this approach. You'll see I also have code to check that all methods return the same result and to format the output.

The main routine might look something like this:

test_sizes.each do |n|
  puts "\nn = #{n}"
  arr = test_array(n)
  Benchmark.bm(@indent) do |bm|
    @methods.each do |m|
      bm.report m.to_s do
        send(m, arr)
     end
    end
  end
end

I use a module so that methods to be benchmarked can be added, deleted or renamed without having to touch any code outside the module.

Answer 3

It seems like you're attempting to use Ruby methods as functions. It's uncommon, but totally possible.

def benchmark(func, index, array)
    start = Time.now
    func.call(index, array) # <= (C)
    start - Time.now
end

def func1(index, array)
    #perform computations based on index and array
end 

def func2(index, array)
    #more computations....
end

benchmark(method(:func1), index1, array1) # <= (A)
benchmark(method(:func1), index2, array2) # <= (B)

The changes to your code are as follows:

A, B) Create a Method object out of your previously-defined methods. A Method object is like a Proc object in that it has a call method that allows you to invoke it later. In your code, when you just use func1 rather than method(:func1), what's happening is that you're immediately calling the method and passing its result to benchmark, rather than passing the function itself into benchmark for later invocation.

C) Use the call method. Ruby doesn't allow you to arbitrarily call variables as functions using parentheses the way some other languages do, but if it is an object type with a call method, like a Method or a Proc, you can use call to invoke the function when you're ready, instead of immediately before trying to pass it to another method, as in your original code.

Answer 4

class SimpleBenchmarker
    def self.go(how_many=1, &block)
        $sorted_time = Array.new
        puts "\n--------------Benchmarking started----------------"
        start_time = Time.now
        puts "Start Time:\t#{start_time}\n\n"
        how_many.times do |a|
            print "."
            block.call
        end
        print "\n\n"
        end_time = Time.now
        puts "End Time:\t#{end_time}\n"
        puts "-------------Benchmarking finished----------------\n\n"
        result_time = end_time - start_time
        puts "Total time:\t\t#{result_time.round(3)} seconds\n\n"
        puts "The run times for the iterations from shortest to longest was: #{$sorted_time.sort.join("s, ")}s\n\n"
    end
end
    print "How many times? "
    t = gets.to_i
SimpleBenchmarker.go t do
    time = rand(0.1..1.0).round(3)
    $sorted_time.push(time)
    sleep time
end

Answer 5

Please check my new gem which can profile your ruby method (instance or class) - https://github.com/igorkasyanchuk/benchmark_methods.

No more code like this:

t = Time.now
user.calculate_report
puts Time.now - t

Now you can do:

benchmark :calculate_report # in class

And just call your method

user.calculate_report