Python serving html async

Question

I have a server serving HTML with nginx, uwsgi and flask. It's a simple webapp. When you refresh a page the webapp makes some API calls elsewhere and then treats the results returned from the APIs and generate the HTML which is then returned.

Now, most of the time is spend waiting for the APIs to respond. Because of this, I'm using gevent to make the API calls and monkey patching etc etc to make it all concurrent. If for every page refresh you need to make 3 API calls, you don't need to wait for the first call to return to initiate the second call. You initiate all three of them, wait for their responses, and then generate the HTML to be returned.

However, a separate question is whether or not the page refreshes from beginning to end are themselves concurrent. To test this I wrote a small concurrent script to make refreshes to my server:

from gevent import monkey
monkey.patch_all()

import requests, datetime

import gevent

def call_func(n):
    before = datetime.datetime.now()
    print 'initiating call '+str(n),before
    requests.get('http://examplemyserver.com')
    after = datetime.datetime.now()
    print 'finishing call '+str(n),after, after-before

gevent_list = [ gevent.spawn(call_func, n=n) for n in range(10) ]

gevent.joinall(gevent_list)

Here are the results:

initiating call 0 2016-05-04 23:03:49.437540
initiating call 1 2016-05-04 23:03:49.446304
initiating call 2 2016-05-04 23:03:49.447911
initiating call 3 2016-05-04 23:03:49.450232
initiating call 4 2016-05-04 23:03:49.451774
initiating call 5 2016-05-04 23:03:49.453331
initiating call 6 2016-05-04 23:03:49.454759
initiating call 7 2016-05-04 23:03:49.456301
initiating call 8 2016-05-04 23:03:49.457663
initiating call 9 2016-05-04 23:03:49.458981
finishing call 0 2016-05-04 23:03:51.270078 0:00:01.832538
finishing call 6 2016-05-04 23:03:52.169842 0:00:02.715083
finishing call 3 2016-05-04 23:03:52.907892 0:00:03.457660
finishing call 1 2016-05-04 23:03:53.498008 0:00:04.051704
finishing call 8 2016-05-04 23:03:54.150188 0:00:04.692525
finishing call 4 2016-05-04 23:03:55.032089 0:00:05.580315
finishing call 7 2016-05-04 23:03:55.994570 0:00:06.538269
finishing call 2 2016-05-04 23:03:56.659146 0:00:07.211235
finishing call 9 2016-05-04 23:03:57.149156 0:00:07.690175
finishing call 5 2016-05-04 23:03:58.002210 0:00:08.548879

So you see, ten page refreshes are initiated virtually at the same time, however they are returned sequentially with a difference of about 1 second. This suggests that while internally the external API calls are done concurrently, until an HTTP response is given, no new HTTP responses are being accepted.

So, given that most of the webapp time is spent waiting for API responses, how can I make the whole process concurrent? Is it something that I need to change with nginx? With uwsgi? With flask? I have no idea about the architecture of these things.

Thanks.

EDIT 1: Following Rob's idea, I replaced the server to be called with 'http://httpbin.org/delay/1'. This is the result I obtain:

initiating call 0 2016-05-04 23:36:18.697813
initiating call 1 2016-05-04 23:36:18.706510
initiating call 2 2016-05-04 23:36:18.708645
initiating call 3 2016-05-04 23:36:18.711055
initiating call 4 2016-05-04 23:36:18.712679
initiating call 5 2016-05-04 23:36:18.714051
initiating call 6 2016-05-04 23:36:18.715471
initiating call 7 2016-05-04 23:36:18.717015
initiating call 8 2016-05-04 23:36:18.718412
initiating call 9 2016-05-04 23:36:18.720193
finishing call 0 2016-05-04 23:36:20.599483 0:00:01.901670
finishing call 2 2016-05-04 23:36:20.600829 0:00:01.892184
finishing call 8 2016-05-04 23:36:20.611292 0:00:01.892880
finishing call 5 2016-05-04 23:36:20.618842 0:00:01.904791
finishing call 7 2016-05-04 23:36:20.620065 0:00:01.903050
finishing call 6 2016-05-04 23:36:20.621344 0:00:01.905873
finishing call 1 2016-05-04 23:36:20.622407 0:00:01.915897
finishing call 4 2016-05-04 23:36:20.623392 0:00:01.910713
finishing call 9 2016-05-04 23:36:20.624236 0:00:01.904043
finishing call 3 2016-05-04 23:36:20.625075 0:00:01.914020

This is what you'd expect from a server that can take HTTP calls concurrently. So this shows that the above code that I used for the test is correct, and that the problem is indeed on the server. It's not taking HTTP calls concurrently.

EDIT2: I was playing around and checked that the simplest possible flask app has the same behavior as my server, so I'm giving that as my "minimal example"

Consider the following server:

import time

from flask import Flask, jsonify
app = Flask(__name__)

@app.route("/")
def hello():
    time.sleep(2)
    return ''

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80, debug=True)

Now connect to it by taking the previous code and pointing it to http://localhost. The result is the following:

initiating call 0 2016-05-04 23:49:12.629250
initiating call 1 2016-05-04 23:49:12.638554
initiating call 2 2016-05-04 23:49:12.643844
initiating call 3 2016-05-04 23:49:12.645630
initiating call 4 2016-05-04 23:49:12.647866
initiating call 5 2016-05-04 23:49:12.649332
initiating call 6 2016-05-04 23:49:12.650853
initiating call 7 2016-05-04 23:49:12.652448
initiating call 8 2016-05-04 23:49:12.653865
initiating call 9 2016-05-04 23:49:12.655348
finishing call 0 2016-05-04 23:49:14.671281 0:00:02.042031
finishing call 1 2016-05-04 23:49:16.673649 0:00:04.035095
finishing call 2 2016-05-04 23:49:18.676576 0:00:06.032732
finishing call 3 2016-05-04 23:49:20.679746 0:00:08.034116
finishing call 4 2016-05-04 23:49:22.681615 0:00:10.033749
finishing call 5 2016-05-04 23:49:24.683817 0:00:12.034485
finishing call 6 2016-05-04 23:49:26.686309 0:00:14.035456
finishing call 7 2016-05-04 23:49:28.688079 0:00:16.035631
finishing call 8 2016-05-04 23:49:30.691382 0:00:18.037517
finishing call 9 2016-05-04 23:49:32.696471 0:00:20.041123

So basically a simple flask webapp does not take HTTP calls concurrently. How can I change this?

It is also possible that http://examplemyserver.com is single-threaded. It "examplemyserver.com" a production server or something you whipped up for this experiment? — Robᵩ, May 04 '16 at 22:10
Possible dup, certainly related: http://stackoverflow.com/questions/9501663/how-enable-requests-async-mode — Robᵩ, May 04 '16 at 22:15
With Python2.7.6, I cannot reproduce your results. I replaced the URL with `http://httpbin.org/delay/1`, but otherwise my code is identical to yours. For me, all of the requests lines occur more-or-less simultaneously, there is a 1-sec delay, and all of the response lines occur more-or-less simultaneously. — Robᵩ, May 04 '16 at 22:23
@Robᵩ I replaced the url of my server for privacy. That's not the real url. — , May 04 '16 at 22:32
@Robᵩ Also, it's not related to the question you linked. In that question, the OP is doing something wrong in the calls. In my case each of the calls has the behavior expected, but I think there's something wrong in the server. — , May 04 '16 at 22:35
Yes, I realize that the url is made up. That's why I replaced it with a publicly-visible URL that has a built-in one-second delay. What happens when you run your sample program against `http://httpbin.org/delay/1`? — Robᵩ, May 04 '16 at 22:35
Oh, I misunderstood the role of "examplemyserver.com" I didn't realize that was running the webapp that you are asking about. Ignore pretty much everything I've said. Starting over, can you post a simplified version of the webapp and nginix configuration that demonstrates the problem? That is, can you produce a [mcve]? — Robᵩ, May 04 '16 at 22:42
Consider https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html#adding-concurrency-and-monitoring and its advice to use `--processes` and `--threads` switches to the uwsgi program. — Robᵩ, May 04 '16 at 22:50
@Robᵩ So flask can't do concurrrency? You have to have several flask processes, is that it? — , May 04 '16 at 23:15
You're comparing apples and oranges. When you run `flask` standalone, you are running the `werkzeug` web server. When you run `flask` in your preferred configuration, you're running the `nginix` web server. — Robᵩ, May 04 '16 at 23:19
@Robᵩ ok, but the question still stands: to respond to HTTP calls concurrently do I need several flask processes, or not? — , May 04 '16 at 23:28

Robᵩ · Accepted Answer · 2016-05-05T13:29:26.417

Short answer: use --processes=N and/or --threads=M when you start uwsgi.

Long answer: The stack you've described involved three components: nginx in the role of a reverse-proxy web server; uwsgi in the role of an application server; and flask as an application framework. The data flow occurs in that order, with the response flowing in the oppose order.

nginx is inherently multi-tasking and flask is inherently single-tasking. uwsgi can be configured to be either. By default, uwsgi is single-tasking.

Here is the setup I used to investigate this. My OS is Ubuntu 14.04.

/etc/nginx/sites-available/myapp:

server {
        listen 8100;
        location / {
                include uwsgi_params;
                uwsgi_pass unix:/tmp/uwsgi.sock;
        }
}

uwsgi command line:

One of:
  uwsgi -s /tmp/uwsgi.sock -w app:app
  uwsgi -s /tmp/uwsgi.sock -w app:app --processes=4
  uwsgi -s /tmp/uwsgi.sock -w app:app --threads=4

Followed by:

  chmod 777 /tmp/uwsgi.sock

app.py:

import time
from flask import Flask, render_template_string
app = Flask(__name__)

@app.route('/')
@app.route('/<name>')
def hello(name='World'):
    time.sleep(1)
    return render_template_string('''
        <html><body><p>
        Hello, <b>{{ name }}</b>!
        </p></body></html>''', name=name)

if __name__=="__main__":
    app.run(debug=True)

Client: I used your gevent/requests client from above, substituting the following url: http://localhost:8100/World.

Typical result:

initiating call 0 2016-05-04 21:08:16.473978
initiating call 1 2016-05-04 21:08:16.485589
initiating call 2 2016-05-04 21:08:16.487258
initiating call 3 2016-05-04 21:08:16.490140
initiating call 4 2016-05-04 21:08:16.493158
initiating call 5 2016-05-04 21:08:16.494417
initiating call 6 2016-05-04 21:08:16.495620
initiating call 7 2016-05-04 21:08:16.497180
initiating call 8 2016-05-04 21:08:16.498413
initiating call 9 2016-05-04 21:08:16.499528
finishing call 3 2016-05-04 21:08:17.512949 0:00:01.022809
finishing call 6 2016-05-04 21:08:17.515944 0:00:01.020324
finishing call 7 2016-05-04 21:08:17.517655 0:00:01.020475
finishing call 4 2016-05-04 21:08:17.519555 0:00:01.026397
finishing call 2 2016-05-04 21:08:18.520249 0:00:02.032991
finishing call 0 2016-05-04 21:08:18.522902 0:00:02.048924
finishing call 5 2016-05-04 21:08:18.524902 0:00:02.030485
finishing call 1 2016-05-04 21:08:18.526682 0:00:02.041093
finishing call 9 2016-05-04 21:08:19.526909 0:00:03.027381
finishing call 8 2016-05-04 21:08:19.528926 0:00:03.030513

References:

http://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html - "There is no magic rule for setting the number of processes or threads to use. It is very much application and system dependent"
https://www.digitalocean.com/community/tutorials/how-to-deploy-python-wsgi-applications-using-uwsgi-web-server-with-nginx
http://uwsgi-docs.readthedocs.io/en/latest/Configuration.html
http://flask.pocoo.org/docs/0.10/deploying/uwsgi/
http://uwsgi-docs.readthedocs.io/en/latest/Nginx.html

I think you nailed it with your distinction between the several components. Thank you very much. — , May 05 '16 at 08:29
Hi, I just tested this and it totally works. I didn't use your config though. Since I already had my own config in place, with Emperor. The only thing I needed to do was in the uwsgi .ini file add the line `processes = 4`, it's that simple. Like you said the optimal number depends on many things. In my case since 99% of the time is spent waiting for external APIs, I suspect that my optimal number of processes will be quite a big number. Thanks again. — , May 05 '16 at 10:31

Python serving html async

1 Answers1