184

Which of these frameworks / libraries would be the best choise for building modern multiuser web application? I would love to have an asynchronous webserver which will allow me to scale easly. What solution will give the best performance / scalability / most useful framework (in terms of easy of use and easy of developing)?

It would be great if it will provide good functionality (websockets, rpc, streaming, etc).

What are the pros and cons of each solution?

Wojciech Danilo
  • 11,573
  • 17
  • 66
  • 132
  • Does it need to be one of these frameworks? What is it your planning to do and could something like Django, Pylons, etc... work? – Joe Doherty Dec 18 '12 at 22:25
  • No, of corse not, but I would love if it will be asynchronous and would have good support for websockets - I've updated the question also. Thank you. – Wojciech Danilo Dec 18 '12 at 22:40
  • 3
    Sort of a broad question, isn't it? – Jean-Paul Calderone Dec 19 '12 at 02:25
  • Your choice depends on libraries, that you want to use. Your libraries - on task, that you want to solve. – Nikolay Fominyh Dec 19 '12 at 05:39
  • 1
    Yes it is broad, but maybes olvable. I'm wondering if these libraries are used in production and somebody, who is using some of them everyday is able to tell what they are great for, what they lack etc. What should be the baseline to choose a library - maybe Twisted should be concidered depreciated when talking about Autobahn or Cyclone? Or mmaybe Autobahn and Cyclone are not production ready and its maintance is under question? Or maybe Tornado has more modern design and it future will be probably shine and great and it should be THE choise for startup project? – Wojciech Danilo Dec 19 '12 at 10:09
  • It sounds like question about "best practices". No silver bullet. Use anything and be able to switch fast. In production used all of these technologies. – Nikolay Fominyh Dec 19 '12 at 15:30
  • there are microframeworks such as flask, cherrypy, and bottle – ealeon Oct 03 '15 at 19:39

4 Answers4

232

"Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design". If you are building something that is similar to a e-commerce site, then you should probably go with Django. It will get your work done quick. You dont have to worry about too many technology choices. It provides everything thing you need from template engine to ORM. It will be slightly opinionated about the way you structure your app, which is good If you ask me. And it has the strongest community of all the other libraries, which means easy help is available.

"Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions". Beware - "microframework" may be misleading. This does not mean that Flask is a half-baked library. This mean the core of flask is very, very simple. Unlike Django, It will not make any Technology decisions for you. You are free to choose any template engine or ORM that pleases you. Even though it comes with Jinja template engine by default, you are always free to choose our own. As far as I know Flask comes in handy for writing APIs endpoints (RESTful services).

"Twisted is an event-driven networking engine written in python". This is a high-performance engine. The main reason for its speed is something called as deferred. Twisted is built on top of deferreds. For those of you who dont know about defereds, it is the mechanism through with asynchronous architecture is achieved. Twisted is very fast. But is not suitable for writing conventional webapps. If you want to do something low-level networking stuff, twisted is your friend.

"Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user". Tornado stands some where between Django and Flask. If you want to write something with Django or Flask, but if you need a better performance, you can opt for Tornado. it can handle C10k problem very well if it is architected right.

"Cyclone is a web server framework for Python that implements the Tornado API as a Twisted protocol". Now, what if you want something that is nearly as performant as Twisted but easy to write conventional webapps? Say hello to cyclone. I would prefer Cyclone over Tornado. It has an API that is very similar to Tornado. As a matter of fact, this is a fork of Tornado. But the problem is it has relativly small community. Alexandre Fiori is the only main commiter to the repo.

"Pyramid is a general, open source, Python web application development framework. Its primary goal is to make it easier for a Python developer to create web applications." I haven't really used Pyramid, but I went through the documentation. From what I understand, Pyramid is very similar to Flask and I think you can use Pyramid wherever Flask seems appropriate and vice-versa.

EDIT: Request to review any other frameworks are welcomed!

Source: http://dhilipsiva.com/2013/05/19/python-libraries-django-twisted-tornado-flask-cyclone-and-pyramid.html

dhilipsiva
  • 3,688
  • 1
  • 24
  • 35
  • 1
    Django is my prefered framework. Its documentation is very good and the ORM is easy. South is great for database schema migrations .... But the question contained this: "I would love to have an asynchronous webserver which will allow me to scale easly.". Django was made for classic request+response sites, not for async. – guettli Nov 15 '13 at 10:34
  • Both Tornado and Cyclone will suit your need. – dhilipsiva Nov 15 '13 at 10:39
  • 1
    Can you write about Pyramid? – Fizer Khan Apr 16 '14 at 17:02
  • @FizerKhan I have never used Pyramid. Give me 3 or 4 days. I ll use and update my answer accordingly. – dhilipsiva Apr 18 '14 at 08:33
  • 5
    @FizerKhan : As per your request, I updated the answer. Sorry it took so long. Didn't have time to use Pyramid. But I went through the documentation. – dhilipsiva May 03 '14 at 04:07
  • @Sriram Never used Paste before. Not sure If I will be having enough time to go through docs. If I ever manage to go through the docs, I will definitely come back and update this answer. – dhilipsiva Feb 05 '16 at 14:03
  • Adding some sense of the community size for each package would be helpful. – Praxeolitic Feb 24 '16 at 01:30
  • @Praxeolitic By community, I mean number of Contributors / Stars / Forks / Commits on GitHub. I am aware that this does not reflect the size of community as a whole. But IMHO, I think these things roughly translates to size and activity of a community (approx.). I am not aware of any other source that can be more accurate. Please correct me if I am wrong. – dhilipsiva Feb 25 '16 at 08:39
  • Sorry, I meant it would be helpful to add something about community size besides just for Django and Cyclone -- especially for the ones that cover similar ground like Twisted and Tornado. I don't know of a better way to measure community size than GitHub if we're staying strictly objective but I think your own personal "down in the trenches" sense of popularity would be useful for those of us unfamiliar with these frameworks. (There are sites claiming to show popularity of PyPI packages but I don't know where they get their data and their rankings seem strange.) – Praxeolitic Feb 25 '16 at 10:43
  • @Praxeolitic Sorry for the late reply - I some how missed your last comment. I am just reading this. In that case, the order that I mentioned above reflects my perception (But I might be wrong). The first one has bigger community than the next. But there is one change though, Its been a long time since I wrote the answer. Seems like flask is growing at faster pace than Django in terms in community size (Just looking at the stars / forks metrics here). – dhilipsiva Mar 21 '16 at 15:46
  • @dhilipsiva You may also mention python celery that works really well when combined with Django ... – Antoine Brunel Aug 03 '16 at 08:21
  • But celery is not a Web Framework. Yes - its built around web frameworks. Its more of a task scheduler / manager. – dhilipsiva Aug 03 '16 at 11:57
  • 1
    I'd add that the single-threaded environment that Tornado uses is much more bug-prone - a mistake that makes the code for a single API endpoint slower will easily cause problems for all endpoints. – Abel Molina Sep 24 '16 at 03:13
  • 1
    Could you make a comment on cherrypy? – Stavros Avramidis Jun 04 '17 at 08:07
  • is there any real alternative to twisted, I mean for low level networking stuff? – MsA Oct 12 '18 at 13:53
  • You can always use Python's inbuilt socket library. – dhilipsiva Nov 04 '18 at 04:58
64

This is obviously a somewhat biased answer, but that is not the same thing as a wrong answer; you should always use Twisted. I've answered similar questions before, but since your question is not quite the same, here are some reasons:

"Best Performance"

Twisted continuously monitors our performance at the speed.twistedmatrix.com website. We were also one of the first projects to be monitored by PyPy's similar site, thereby assuring the good performance of Twisted on the runtime that anyone concerned with high-performance applications in Python.

"Scalability"

To my knowledge, none of the listed frameworks have any built-in support for automatic scaling; they're all communication frameworks, so you have to do the work to communicate between your scaling nodes. However, Twisted has an advantage in its built-in support for local multi-processing. In fairness, there is a third-party add-on for Tornado that allows you to do the same thing. In recent releases, Twisted has added features that increase the number of ways you can share work between cores, and work is ongoing in that area. Twisted also has a couple of well-integrated, "native" RPC protocols which offer a construction-kit for whatever scaling idiom you want to pursue.

"Most Useful"

Lots of people seem to find Twisted very useful. So much so that many of them have extended it and made their extensions available to you.

"Functionality"

Out of the box, Twisted includes:

In this last department, at least, Twisted seems a clear winner for built-in functionality. And all this, in a package just over 2 megabytes!

Community
  • 1
  • 1
Glyph
  • 31,152
  • 11
  • 87
  • 129
  • Thank you for yours response. Could you pleae answer one more question - I need to use websocket based rpc. I found out that Autobahn, a project based on twisted, has support for such thing. I don't understeand why autobahn is separate project, not an extension for twisted, because it has probably less developers and smaller community and may be not so good solution as twisted. What do you think about it? – Wojciech Danilo Dec 20 '12 at 10:49
  • 7
    why so many people say that they no longer using Twisted, but GEvent? – remdezx Dec 20 '12 at 11:02
  • 2
    There are a lot of people telling that it is hard to maintain big applications using Twisted (because of its callback architecture): http://stackoverflow.com/questions/3048012/eventlet-or-gevent-or-stackless-twisted-pylons-django-and-sql-alchemy Would it not be better to use gevent or gevent based Twisted? – Wojciech Danilo Dec 20 '12 at 11:33
  • This is a matter of taste, I think. Personally, I find it easy to maintain large Twisted applications. Maintaining a large GEvent application is effectively the same as maintaining a large multithreaded application, similarly hard to debug, although slightly more determistic and better-performing for I/O driven workloads. If you want a "gevent based Twisted", use Eventlet, which can use the Twisted event loop. – Glyph Dec 24 '12 at 22:23
  • 9
    @remdezx As to your question, there are two reasons. One is that people find Twisted hard to understand because concurrent programming is hard to understand. They then switch to GEvent because it is superficially easy to understand - as long as there is no concurrency, everything works just like you'd expect. The other reason is that it is a lot less work to port over code to GEvent that wasn't written using an event-driven API to get the performance benefits of event-driven I/O. If your code doesn't share too much state, such a port probably works fine. – Glyph Dec 24 '12 at 22:37
  • 1
    @Glyph, you're talking about concurrency, but ... neither Gevent nor Twisted support concurrency (of course you can use multiprocessing with gevent and use a pool of real threads and pool of greenlets on each thread - which works fine for me and you can use Twisted plugin to do exactly the same - run multiple twisted intances next to each other). But does Twisted gices you something more than gevent? I think even with multiple instances of gevent/ twisted, gevent is easier to understeand without its explicite io callbacks. Am I missing something? – Wojciech Danilo Jan 02 '13 at 16:24
  • 2
    @danilo2 Yes, you're missing at least one thing :). Specifically you're misunderstanding the word "concurrency" to mean "concurrent parallel execution on multiple CPUs". Twisted can do concurrent I/O scheduling via asynchronous (callback-based) I/O. GEvent can do concurrent I/O scheduling via a micro-thread scheduler. In Twisted, using `spawnProcess`, this I/O scheduling can be translated into CPU scheduling as well. – Glyph Jan 03 '13 at 18:27
  • 1
    In fact Twisted support *multi-threading* from its API: http://twistedmatrix.com/documents/current/core/howto/threading.html – Robert Zaremba Jan 29 '13 at 20:41
  • To clarify what concurrency is and what parallelism is: http://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurrency/ – Erik Kaplun Mar 04 '13 at 23:32
  • @Glyph: I do not know how following applies to websockets, but there were benchmarks showing a performance superiority of Tornado over Twisted (on simple http requests). It would be great to find websocket (server) benchmarks/comparisons somewhere. http://programmingzen.com/2009/09/13/benchmarking-tornado-vs-twisted-web-vs-tornado-on-twisted/ – Levite Jun 16 '14 at 15:23
  • @Levit - Microbenchmarks like that are *extremely* misleading. If you want to see something demonstrating that Twisted is basically exactly equivalent to Tornado, please see http://floss.zoomquiet.io/data/20120414163248/index.html – Glyph Jun 17 '14 at 20:22
  • @Glyph: thx for the additional bench! But isn't that also showing a superiority of Tornado over Twisted? With only cyclone (which `is a web server framework for Python that implements the Tornado API as a Twisted protocol`) being slightly ahead of Tornado, and twisted (as well as gevent) being about 17% behind them both? – Levite Jun 18 '14 at 09:21
  • "Twisted" in that example refers to `twisted.web` which is the built-in web server in Twisted. There are probably some things that could be optimized there, but performance on a "hello world" benchmark is not the only measurement of a web server ;-). – Glyph Jun 19 '14 at 02:55
50

I like @Glyph response. Twisted is very comprehensive, rich python framework. Twisted and Tornado have a very similar design. And I like this design very much:

  • it's fast
  • easy to understand
  • easy to extend
  • doesn't require c-extensions
  • works on PyPy.

But I want to highlight Tornado, which I prefer and recently gain popularity. Tornado, like Twisted, uses callback style programming, but it can be inlined using tornado.gen.engine (twisted.internet.inlineCallbacks in Twisted).

Codebase

The best comment is from http://cyclone.io site. cyclone tries to mix Twisted and Tornado because:

Twisted is one of the most mature libraries for non-blocking I/O available to the public. Tornado is the open source version of FriendFeed’s web server, one of the most popular and fast web servers for Python, with a very decent API for building web applications.

The idea is to bridge Tornado's elegant and straightforward API to Twisted's Event-Loop, enabling a vast number of supported protocols.

But in 2011 tornado.platform.twisted was out which brings similar functionality.

Performance

Tornado has much better performance. It also works seamlessly with PyPy, and get huge gain.

Scalability

The same like Twisted. Tornado has tornado.process and a lot of rpc services implemented on top of it.

Functionality

There are 71 Tornado based package, compared to 148 Twisted's and 48 Gevent's. But if you look carefully and compute median of packages upload time, you will see that Twisted ones are the oldest, then Gevent and Tornado the freshest. Furthermore there is tornado.platform.twisted module which allows you to run code written for Twisted on Tornado.

Summary

With Tornado you can use a code from Twisted. There is no need to use cyclone which only twists your code (your code becomes more messy).

As for 2014, Tornado is considered as widely accepted and default async framework which works both on python2 and python3. Also the latest version 4.x brings a lot of functionality from https://docs.python.org/dev/library/asyncio.html.

I wrote an article, explaining why I consider that Tornado - the best Python web framework where I wrote much more about Tornado functionality.

Robert Zaremba
  • 8,081
  • 7
  • 47
  • 78
16

(UPDATE: I'm sadly surprised about how few answers here recommend or even mention Gevent—I don't think it's in proportion to the popularity, performance and ease of use of this excellent library!)

Gevent and Twisted are not mutually exclusive, even though the contrary might seem obvious at first. There is a project called geventreactor which allows one to relatively smoothly leverage the best of both worlds, namely:

  • The efficient and cheap (cooperative green) thread model of Gevent, which is much easier to program in when it comes to concurrency—frankly, Twisted's inlineCallbacks is simply not up to the job in terms of performance when it comes to many coroutines, and neither in terms of ease/transparency of use: yield and Deferreds everywhere; often hard to build some abstractions; horrifyingly useless stack traces with both bare Deferreds as well as, and even more so with @inlineCallbacks.
  • All the built-in functionality of Twisted you can ever dream of, including but not limited to IReactorProcess.spawnProcess.

I'm personally currently using Gevent 1.0rc2 with Twisted 12.3 bridged by geventreactor. I have implemented my own as-of-yet unpublished additions and enhancements to geventreactor which I will publish soon, hopefully as part of geventreactor's original GitHub repository: https://github.com/jyio/geventreactor.

My current layout allows me to program in the nice programming model of Gevent, and leverage things such as a non-blocking socket, urllib2 and other modules. I can use regular Python code for doing regular things, as opposed to the learning curve and inconvenience of doing even simple, basic things the Twisted way. I can also easily use most 3rd party libraries that are normally either out of question with Twisted, or require the use of threads.

I can also completely avoid the awkward and often overly complex callback based programming by using greenlets (instead of Deferreds and callbacks, and/or @inlineCallbacks).

(This answer was written based on my personal experiences having used both Twisted and Gevent in real life projects, with significantly more experience using Twisted (but I don't claim to be a Twisted expert). The software I've had to write hasn't had to use too many of Twisted's features, so depending on the set of features you require of Twisted, the (relatively painless) extra complexity of mixing Gevent and Twisted might not be worth the trouble.)

Erik Kaplun
  • 37,128
  • 15
  • 99
  • 111