wrk is displaying odd results on Rack vs Sinatra benchmark

Question

I'm benchmarking a "hello world" equivalent using sinatra and rack.

Command in question wrk -t12 -c400 -d30s: 12 threads, 400 open HTTP connections, 30 seconds.

Rack:

require 'rack'

app = Proc.new do |env|
    ['200', {'Content-Type' => 'text/html'}, ['A barebones rack app.']]
end

Rack::Handler::Thin.run app

# wrk $ wrk -t12 -c400 -d30s http://localhost:8080
# Running 30s test @ http://localhost:8080
#   12 threads and 400 connections
#   Thread Stats   Avg      Stdev     Max   +/- Stdev
#     Latency    11.82ms   38.97ms 488.51ms   99.32%
#     Req/Sec   705.04    568.62     2.20k    61.82%
#   16576 requests in 30.08s, 1.55MB read
#   Socket errors: connect 157, read 274, write 0, timeout 0
# Requests/sec:    551.05
# Transfer/sec:     52.74KB

Sinatra:

require 'sinatra'

get '/' do
  status 200
  headers \
    'Content-Type' => 'text/html'
  'A barebones rack app.'
end

# wrk $ wrk -t12 -c400 -d30s http://localhost:4567
# Running 30s test @ http://localhost:4567
#   12 threads and 400 connections
#   Thread Stats   Avg      Stdev     Max   +/- Stdev
#     Latency    40.12ms   90.46ms   1.39s    98.67%
#     Req/Sec   265.47    147.50     1.17k    73.15%
#   90322 requests in 30.08s, 18.78MB read
#   Socket errors: connect 157, read 333, write 0, timeout 0
# Requests/sec:   3002.52
# Transfer/sec:    639.21KB

Specs:

If both Rack and Sinatra run Thin, how come Sinatra manages 3002.52~ req/s while pure Rack manages only 551.05 req/s? What am I missing?

Answer 1

I didn't install wrk, I wasn't in the mood for an install, but I ran both your examples using time:

# run.sh
for i in {1..1000}
do 
  curl localhost:8080/ > /dev/null 2>&1
done

# sinatra, run via `bundle exec ruby sinatra.rb -p 8080`
$ time sh run.sh             
sh run.sh  3.67s user 2.79s system 66% cpu 9.768 total

# rack, run via `bin/rackup config.ru`
$ time sh run.sh
sh run.sh  3.65s user 2.87s system 74% cpu 8.799 total

# sinatra with the puma server, by adding the line `set :server, "puma"`
$ time sh run.sh      
sh run.sh  3.67s user 2.71s system 92% cpu 6.924 total

I got very similar results, so perhaps it's something to do with wrk or your system. I'm running OSX Mavericks, but I found this answer that says to benchmark with time you should really use chrt to avoid contention with other processes. I'm not sure what the OSX equivalent of chrt is though. Perhaps that's what is happening on your system, or perhaps the ports are running at different speeds, for whatever reason.

I also sandboxed my gems using bundle install --binstubs --path vendor.noindex, perhaps that would have an effect on your system too.

Otherwise, I can't see any reason for such disparity. I a little surprised by the results I got as I'd expect Sinatra to be quite a bit heavier than Rack, but perhaps things seem different with more complex or larger apps. Or perhaps not, Sinatra is such a neat little library.