2

According to the following Stack Overflow question: Does Django scale? Django is great for a web apps that anticipates thousands of users per day/month, however to achieve this, you have to have the right design.

Now my scenario is as follows:

I've just joined a startup as a tech lead / lead developer, and while I have 3 years experience with Django, I have never used it for as a large scale app. The latest large scale app I was involved with used Zope, and relied on SQL Server to handle the caching, etc (horrible practice in my opinion). Given that this startup now has only about 20,000 users only after 6 months of existence, and given that they've also hired a marketing specialist at the same time they hired me, I can anticipate (or at least hope to anticipate) a growth opportunity. Much of the existing Django code needs refactoring; i.e. using static html form rather than ModelForm, etc.

My question is: as I refactor this App, what are the design patterns I should follow to get it done right, such that we can scale to say 100,000 to 500,000 users per month? What are some gottchas that I should lookout for? In terms of Model design, what is considered scalable model design? In terms of DB support, what kind of DB setup can help me achieve this scalability? In terms of load balancing, what architecture is appropriate for load balancing?

I assume using Django out of the box, will not achieve this, and I don't want to find myself in a situation, where I need to ask to hire a scalability expert...I also would like to know what are some load testing techniques/tools that I can use to measure current performance/scalability? I've only been with them a few days, so I'm not sure of the prod environment just yet, so if I were to feel that I need to change the prod env, what are some good options (cloud solutions), that are appropriate yet affordable?

Edit: They also seem to be using Ajax for everything and that has me worried. Supposing I can't convince them to change that, what are some good Ajax techniques libraries (I know of django-dajax/dajaxice), that can be used and are still scalable.

thanks,

Sam

Community
  • 1
  • 1
Sam Hammamy
  • 10,819
  • 10
  • 56
  • 94
  • I found this set of slides http://www.slideshare.net/mmalone/scaling-django-1393282 which are helpful. Can someone share their experience with this using a scenario. Also if anyone can fid the video of the slides, I'd be greatful... – Sam Hammamy Apr 15 '12 at 18:57
  • 3
    Just my 2 cents, I heard this from a django talk recently... "don't go fancy, find the bottleneck, and find the cheapest and the easiest solution because your specifications will change over time, and in the meantime you are suffering from your scaling problem. You can't stop the whole site in order to make all the changes. You have sort out priority. Find your bottleneck. The other thing is avoid using 3rd party apps as much as possible unless the apps are really really reputable and you are willing to invest developer hours to contribute as well because the project can go dead. No guarantee. – CppLearner Apr 15 '12 at 23:00

1 Answers1

6

Unfortunately there aren't a lot of specifics in your question. Google searches will yield a lot of information on scaling django.

http://www.morethanseven.net/2011/06/30/Django-performance-1-measuring-performance.html

This link provides about 5 different ways to measure different performance metrics. Since I can't tell you specifics on how to scale your app, here are some general good principals for any site.

  1. Make sure static media is served by your web server and not by django
  2. Make sure you profile everything EXPLAIN is crucial.
  3. Cache everything you can. Memory is lightning fast and django makes it easy to implement.
  4. Be careful of django ORM gotchas in your templates. Use django-debug toolbar to make sure you are not inadvertantly making multiple queries when iterating over result sets.
  5. For databases make sure your data is normalized.
  6. Even better use a CDN and make sure that your web servers and DB servers are on seperate machines.

Dont optimize prematurely!! Django can handle A LOT out of the box. I wouldn't worry about ajax or model forms or anything unless you find out that they are specific bottlenecks to you growing your site. I believe just as important as performance is making sure your site is modular, under version control and easily movable. This will facilitate in development and when you grow large enough will let you bring up new machines instantaneously.

dm03514
  • 54,664
  • 18
  • 108
  • 145
  • With respect to metrics and testing, after I posted this question, I ran some google searches and found httperf and autobench; a perl wrapper for httperf, which generates a CSV that can be graphed. I ran it across the home page at in increment of 5 requests until 50 simultaneous requests, and a total of 500, and found that the site didn't respond at 25 and at 35. I also want to look at testing things that are behind login, such as retrieving user profiles, and searches and the like – Sam Hammamy Apr 16 '12 at 00:27
  • 1
    I also found this http://highscalability.com/scaling-django-web-apps-mike-malone which helped me quite a bit in understanding the tasks I need to take. – Sam Hammamy Apr 16 '12 at 00:28
  • Any observations about the tests I ran, and/or suggestions for other tests? – Sam Hammamy Apr 16 '12 at 00:28