benchmarking bottle and flask - what is wrong?

Question

I am attempting to benchmark flask and bottle to make some new project decision. I am seeing bottle fail miserably while flask seems to work. I cannot believe that the frameworks are so far apart (are they?!). I am guess I am doing something simple and wrong. I quite cannot figure it out why.

Client

First step I am attempting to do a benchmark using Wrk as the client to generate load. My Wrk cmdline looks like this

./wrk -c 20 -d 30 -t 10 --latency http://localhost:8888/hello/world

Bottle

The Bottle sample server looks like this

#!/usr/bin/env python

from bottle import route, run, template

@route('/hello/<name>')
def index(name):
        return template('<b>Hello {{name}}</b>!', name=name)

run(host='localhost', port=8888)

The output from the benchmark is

Running 30s test @ http://localhost:8888/hello/world
  10 threads and 20 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.07s     4.44s   13.38s    55.49%
    Req/Sec   184.27    311.26     1.33k    73.56%
  Latency Distribution
     50%    7.29s 
     75%    7.42s 
     90%   13.38s 
     99%   13.38s 
  34208 requests in 30.02s, 5.61MB read
  Socket errors: connect 0, read 0, write 0, timeout 177
Requests/sec:   1139.47
Transfer/sec:    191.40KB

I am seeing a whole bunch of errors on the server logs (related to the timeouts I guess)

Exception happened during processing of request from ('127.0.0.1', 56893)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
    self.wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

Flask

Now I repeated the same test with flask

Flask server

#!/usr/bin/env python

from flask import Flask
app = Flask(__name__)

@app.route("/hello/<name>")
def hello(name):
        return "Hello {0}!".format(name)

if __name__ == "__main__":
    app.run(port=8888)

Now the output from the run

Running 30s test @ http://localhost:8888/hello/world
  10 threads and 20 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.83ms  691.95us  22.49ms   93.54%
    Req/Sec   187.71     22.99   285.00     83.73%
  Latency Distribution
     50%   10.74ms
     75%   11.02ms
     90%   11.42ms
     99%   12.52ms
  55203 requests in 30.00s, 8.69MB read
Requests/sec:   1840.00
Transfer/sec:    296.48KB

This looks more "normal". I am doing this on my laptop (i5/8G/Mint). No special installation of either frameworks (pip install).

Scratching my head on this one!

i'm not familiar with bottle but you might try to run the server on 127.0.0.1 instead of localhost — Or Duan, Jan 03 '15 at 06:07
My hosts file has 127.0.0.1 mapped to localhost. How would using IP be any different? BTW tried it no improvement. — vivekv, Jan 03 '15 at 06:57

score 5 · Accepted Answer · answered Jan 03 '15 at 16:47

Both Flask and Bottle are WSGI application frameworks; they let you build a web application through the WSGI protocol.

To make it easier for you to develop your code, both frameworks come with a built-in, basic WSGI server, so you don't need to install something separate just to test your code. Neither server is suitable for production use however; one of the reasons being that they are not set up for heavy loads, for example.

I suspect you wanted to test how the frameworks stack up, and not their basic development servers. Use a proper WSGI deployment to test the framework against, and make sure you use the same configuration for both frameworks. Be careful not to end up testing the container (server)'s ability to handle load.

benchmarking bottle and flask - what is wrong?

1 Answers1