0

Our server has a lot if CPUs, and some web requests could be faster if request handlers would do some parallel processing.

Example: Some work needs to be done on N (about 1-20) pictures, to severe one web request.

Caching or doing the stuff before the request comes in is not possible.

What can be done to use several CPUs of the hardware:

  • threads: I don't like them
  • multiprocessing: Every request needs to start N processes. Many CPU cycles will be lost for starting a new process and importing libraries.
  • special (hand made) service, which has N processes ready for processing
  • cellery (rabbitMQ): I don't know how big the communication overhead is...
  • Other solution?

Platform: Django (Python)

guettli
  • 25,042
  • 81
  • 346
  • 663
  • 2
    Well, you already excluded the most common ways - threads ( because you don't like them? seriously? ) and multiprocessing, so good luck with that. :) – freakish Jan 17 '13 at 09:52
  • If you want a hammer, try Twisted. – phant0m Jan 17 '13 at 09:55
  • Why I don't like threads is explained in "cons of threads": http://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python#answer-3046201 – guettli Jan 17 '13 at 10:03
  • 1
    I'd put the data in a queue (i.e. RabbitMQ) and have a number of worker processes/threads. See http://www.slideshare.net/tarequeh/life-in-a-queue-using-message-queue-with-django – Alex L Jan 17 '13 at 10:11
  • Some more info: https://devcenter.heroku.com/articles/background-jobs-queueing and celery/django docs: http://docs.celeryproject.org/en/latest/django/index.html – Alex L Jan 17 '13 at 10:15

2 Answers2

3

Regarding your second and third alternatives: you do not need to start a new process for every request. This is what process pools are for. New processes are created when your app starts up. When you submit a request to the pool, it is automatically queued until a worker is available. The disadvantage is that requests are blocking- if no worker is available at the moment, your user will sit and wait.

mbatchkarov
  • 15,487
  • 9
  • 60
  • 79
1

You could use the standard library module asyncore.

This module provides the basic infrastructure for writing asynchronous socket service clients and servers.

There is an example for how to create a basic HTML client.

Then there's Twisted, it can do lots and lots of things, which is why it's somewhat daunting. Here is an example using its HTTP client.

Twisted "speaks HTTP", asyncore does not, you'll have to.

Other libraries:

phant0m
  • 16,595
  • 5
  • 50
  • 82