Tornado server caused Django unable to handle concurrent requests

Question

I wrote a Django website that handles concurrent database requests and subprocess calls perfectly fine, if I just run "python manage.py runserver"

This is my model

class MyModel:
    ...
    def foo(self):
        args = [......]
        pipe = subprocess.Popen(args, stdout=subproccess.PIPE, stderr=subprocess.PIPE)

In my view:

def call_foo(request):
    my_model = MyModel()
    my_model.foo()

However, after I wrap it using Tornado server, it's no longer able to handle concurrent request. When I click my website where it sends async get request to this call_foo() function, it seems like my app is not able to handle other requests. For example, if I open the home page url, it keeps waiting and won't display until the above subprocess call in foo() has finished.

If I do not use Tornado, everything works fine.

Below is my code to start the tornado server. Is there anything that I did wrong?

MAX_WAIT_SECONDS_BEFORE_SHUTDOWN = 5

def sig_handler(sig, frame):
    logging.warning('Caught signal: %s', sig)
    tornado.ioloop.IOLoop.instance().add_callback(force_shutdown)

def force_shutdown():
    logging.info("Stopping tornado server")
    server.stop()
    logging.info('Will shutdown in %s seconds ...', MAX_WAIT_SECONDS_BEFORE_SHUTDOWN)
    io_loop = tornado.ioloop.IOLoop.instance()
    deadline = time.time() + MAX_WAIT_SECONDS_BEFORE_SHUTDOWN

    def stop_loop():
        now = time.time()
        if now < deadline and (io_loop._callbacks or io_loop._timeouts):
            io_loop.add_timeout(now + 1, stop_loop)
        else:
            io_loop.stop()
            logging.info('Force Shutdown')
    stop_loop()

def main():
    parse_command_line()
    logging.info("starting tornado web server")
    os.environ['DJANGO_SETTINGS_MODULE'] = 'mydjango.settings'
    django.setup()
    wsgi_app = tornado.wsgi.WSGIContainer(django.core.handlers.wsgi.WSGIHandler())
    tornado_app = tornado.web.Application([
        (r'/(favicon\.ico)', tornado.web.StaticFileHandler, {'path': "static"}),
        (r'/static/(.*)', tornado.web.StaticFileHandler, {'path': "static"}),
        ('.*', tornado.web.FallbackHandler, dict(fallback=wsgi_app)),
      ])
    global server
    server = tornado.httpserver.HTTPServer(tornado_app)
    server.listen(options.port)

    signal.signal(signal.SIGTERM, sig_handler)
    signal.signal(signal.SIGINT, sig_handler)

    tornado.ioloop.IOLoop.instance().start()

    logging.info("Exit...")

if __name__ == '__main__':
    main()

Answer 1

There is nothing wrong with your set-up. This is by design.

So, WSGI protocol (and so Django) uses syncronous model. It means that when your app starts processing a request it takes control and gives it back only when request is finished. That's why it can process single request at once. To allow simultaneous requests one usually launches wsgi application in multithreaded or multiprocessed mode.

The Tornado server on other side uses asynchronous model. The idea here is to have own scheduler instead of OS scheduler that works with threads and processes. So your code runs some logic, then launches some long task (DB call, URL fetch), sets up what to run when task finishes and gives control back to scheduler.

Giving controll back to scheduler is crucial part, it allows async server to work fast because it can start processing new request while previous is waiting for data.

This answer explains sync/async detailed. It focuses on client, but I think you can see the idea.

So whats wrong with your code: Popen does not give control to IOLoop. Python does nothing until your subprocess is finished, and so can not process other requests, even not Django's requests. runserver "works" here because it's multithreaded. So while locking entirely the thread, other threads can still process requests.

For this reason it's usually not recommended to run WSGI apps under async server like tornado. The doc claims it will be less scalable, but you can see the problem on your own code. So if you need both servers (e.g. Tornado for sockets and Django for main site), I'd suggest to run both behind nginx, and use uwsgi or gunicorn to run Django. Or take a look at django-channels app instead of tornado.

Besides, while it works on test environment, I guess it's not a recomended way to do what you try to achieve. It's hard to suggest the solution, as I don't know what do you call with Popen, but it seams to be something long running. Maybe you should take a look at Celery project. It's a package for running long-term background job.

However, back to running sub-processes. In Tornado you can use tornado.process.Subprocess. It's a wrapper over Popen to allow it to work with IOLoop. Unfortunately I don't know if you can use it in wsgi part under tornado. There are some projects I remember, like django futures but it seems to be abandoned.

As another quick and dirty fix - you can run Tornado with several processes. Check this example on how to fork server. But I will not recoment using this in production anyway (fork is OK, running wsgi fallback is not).

So to summarize, I would rewrite your code to do one of the following:

1. Run the Popen call in some background queue, like Celery
2. Process such views with Tornado and use tornado.processes module to run subprocess.

And overall, I'd seek for another deployment infrastructure, and would not run Durango under tornado.