0

I am trying to design a web application that processes large quantities of large mixed-media files coming from asynchronous processes. Each process can take several minutes.

The files are either uploaded as a POST body or pulled by the web server according to a source URL provided. The files can be processed by a variety of external tools in a synchronous or asynchronous way.

I need to be able to load balance this application so I can process multiple large files simultaneously for as much as I can afford to scale.

I think Python is my best choice for this project, but beside this, I am open to any solution. The app can either deliver the file back or rely on a messaging channel to notify the clients about the process completion.

Some approaches I thought I might use:

1) Use a non-blocking web server such as Tornado that keeps the connection open until the file processing is done. The external processing command is launched and the web server waits until the file is ready and pipes the resulting IO stream directly back to the web app that returns it. Since the processes sending requests are asynchronous, they might afford to wait (unless memory or some other issues come up).

2) Use a regular web server like Cherrypy (which I am more confident with) and have the webapp use a messaging channel to report the processing progress. The web server returns a HTTP response as soon as it receives the file, validates it and sends it to a background process. At the same time it sends a message notifying the process start. The background process then takes care of delivering the file to an available location and sending another message to the channel notifying the location of the new file. This solution looks more flexible than 1), but requires writing a separate script to handle the messages outside the web application, as well as a separate storage space for the temp files that have to be cleaned up at a certain point.

3) Use some internal messaging capability of any of the webserves mentioned above, which I am not familiar with...

Edit: something like CherryPy's pub-sub engine (http://cherrypy.readthedocs.org/en/latest/extend.html?highlight=messaging#publish-subscribe-pattern) could be a good solution.

Any suggestions?

Thank you,

gm

gattu marrudu
  • 247
  • 4
  • 16
  • Regarding 2) there should actually be a way to handle the whole subprocess lifecycle and callback within the same application, as suggested in [this discussion](http://stackoverflow.com/a/2581943/1069841). – gattu marrudu Jul 31 '14 at 15:51

1 Answers1

1

I had a similar situation come up with a really large scale data processing engine that my team implemented. We wanted to build our api calls in Flask, some of which can take many hours to complete, but have a way to notify the user in real time what is going on.

Basically what I came up with is was what you described as option 2. On the same machine that I am serving the flask app through apache, I created a tornado app that serves up a websocket that reports progress to the end user. Once my main page is served, it establishes the websocket connection to the tornado server, and the flask app periodically sends updates to the tornado app, and down to the end user. Even if the browser is closed during the long running application, apache keeps the request alive and processing, and upon logging back in, I can still see the current progress.

I wrote about this solution in some more detail here:

http://jonfeatherstone.com/2013/08/01/mongo-and-websockets-for-application-logging/

Good luck!

feathj
  • 3,019
  • 2
  • 22
  • 22
  • Thanks a lot for the tips @feathj. It looks like a similar scenario, except that I don't have a browser at the end watching for updates, but a messaging broker. – gattu marrudu Jul 31 '14 at 15:07