0

I have a django site that servers matplotlib images generated from numbers stored in a mysql database.

One of the views will display up to 5 images. The src value for the images do not link to static files but instead to django views that will generate png images. The images can take up to 5 seconds to load. The same django view function is not being called repeatedly. The five images correspond to different types of plotting methods, each has its own function in django view.

When multiple requests for the images are sent, the server freezes up. I am developing the site with xampp. I cannot load the site in another tab, or any of the php sites I have running on the same server.

If I try to load all the images with the initial page load the server freezes. If I load the page then request the images separately using jquery/ajax (and wait patiently) then there is no problem. If I quickly make two ajax requests then the server freezes.

Can anyone explain what might be the source of the problem?

Can I force the serialized execution of the requests on the server?

Can I use javascript to delay each request until the prior image is returned (and not just when the link is update as is currently the case)?

Or can I put a time limit on the requests to kill these hang ups on the server?

Thanks. It would be a big help if someone could help me resolve this.

sequoia
  • 3,025
  • 8
  • 33
  • 41

2 Answers2

0

You could use a two stage process to generate these pages:

  • On the initial request, push the image generation tasks into a queuing system such as celery. Use a separate process to execute these tasks, and tune it to run as many as your system will cope with in parallel.
  • On the client side, use ajax to poll the server to determine when the image generation tasks are complete, and finalise the display.
GregJ
  • 16
  • 1
  • thanks. those both sound like helpful suggestions. I will read the documentation on the celery site to see how to implement that. Can you elaborate on the ajax suggestion or point me to an example? Also do you think this problem will occur when multiple users are accessing the site simultaneously? Can I moderate the requests by multiple users in the same way? – sequoia Jul 23 '11 at 05:40
  • Requests from different users will be processed in the order they arrive in the queue. Your ajax code should sit in a wait-loop, polling the server until the task completes. There is sample code for the django view to support this in the task_status view at: https://github.com/ask/django-celery/blob/master/djcelery/views.py. – GregJ Jul 23 '11 at 07:23
  • This looks good, but definitely going to push the boundaries of my past experience, which is a good thing too. I am going try to implement this over the weekend. Do you have examples of the wait-loop javascript people use with django-celery as well? I have been looking around this site, but nothing jumps out as the solution I am looking for. – sequoia Jul 23 '11 at 17:49
  • The technique is "HTTP Push" or [Comet](http://en.wikipedia.org/wiki/Comet_%28programming%29) programming. There is a good discussion [here](http://stackoverflow.com/questions/4310706/django-comet-push-least-of-all-evils) and [here](http://stackoverflow.com/questions/621802/choosing-and-deploying-a-comet-server) around integrating it into a django environment. A good [Comet tutorial](http://www.screenr.com/SNH) is available - PHP on the backend unfortunately but it describes the client-side code well. – GregJ Jul 25 '11 at 08:13
  • Thanks Greg. I am working my way through your suggestions. I am trying to get celery up and running on the windows machine I use for development. I installed django-celery and RabbitMQ. I can start the rabbitmq-service. however when I try to run the celery worker server as specified [here](http://packages.python.org/django-celery/getting-started/first-steps-with-django.html) then I get the message that the host machine actively refused connection. I am trying to adjust the windows firewall, but to no avial. Suggestions? I tried to associate 5672, 5673 with Rabbitmq, but what protocol? – sequoia Jul 26 '11 at 02:40
  • @ Greg -two questions, see above as well. The djcelery views link you sent me, would I add those views to my existing module or would I create a new module at myproject.tasks? – sequoia Jul 26 '11 at 02:57
  • I don't develop under windows so I can't really comment on the Windows firewall interactions. I'd be surprised if it blocks local connections by default. Regarding the location of the views - if you consider that you may want to reuse them in another project then it would be best to create a new django application to hold them. Otherwise just use your existing application. – GregJ Jul 27 '11 at 01:47
  • In the end it was not a firewall issue. I had failed to install rabbitmq properly initially then I opted for the windows binaries which placed the server in a different location. I had conflicts with the windows service and registry values. I have resolved that, but now I am try to figure out how to get django celery up and running. – sequoia Jul 28 '11 at 03:59
0

Are you serving your images with apache? if that so you could try a lighter server for your media like nginx, lighttpd, TUX, Cherokee or even a A stripped-down version of Apache.

to be able to do this you could save your image in a tmp directory and serve it there if you do not wish to save the image.

You can see some extended documentation on https://docs.djangoproject.com/en/dev/howto/static-files/

The problem i think its happening is that apache stays serving media (heavy media) and you get out of open connections. There is when a lighter server can help since its able to manage a lot more connections than apache could.

Hassek
  • 8,715
  • 6
  • 47
  • 59
  • I have apache and mysql running the site. There are about 40 tables in the database, 30 that serve as logs and static value lists that will eventually get new values and then 10 data tables, which are huge. Do you recommend that I leave the log tables to be queried in the mysql database on apache, then move the large data tables over to another database on a separate server? The images are served as pngs, but the are generated in this manner [link](http://www.scipy.org/Cookbook/Matplotlib/Django). Images are generated each time because the combo of data selected could vary every time. – sequoia Jul 23 '11 at 17:16
  • you should check if your logs are giving you troubles with a profiler, here is a nice [link](http://serverfault.com/questions/3120/how-do-i-profile-mysql) that talks about it. If that so, having two different databases could help, you should have the databases in a separate machine. I recommend you to check this [link](http://blip.tv/pycon-us-videos-2009-2010-2011/django-deployment-workshop-3651591). Its a deployment workshop on Django at pycon. It has helped me a lot in different occasions – Hassek Jul 23 '11 at 23:11
  • Thanks for the video link. It looks very useful. I'll probably set aside a chunk of time to watch it this weekend. – sequoia Jul 28 '11 at 04:00