0

I am now working on a big backend system for a real-time and history tracking web service.
I am highly experienced in Python and intend to use it with sqlalchemy (MySQL) to develop the backend.
I don't have any major experience developing robust and sustainable backend systems and I was wondering if you guys could point me out to some documentation / books about backend design patterns? I basically need to feed data to a database by querying different services (over HTML / SOAP / JSON) at realtime, and to keep history of that data.

Thanks!

Raiders
  • 181
  • 2
  • 7

2 Answers2

1

Can you define "backend" more precisely? Normally, in web dev, I follow a MVC'ish structure where my "front-end", html/css/js and code dealing with displaying either, is loosly coupled with my "backend" model (business objects and data persistence; i.e. database).

I like Django's Model/View/Template approach:

http://docs.djangoproject.com/en/dev/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names

But, you haven't really defined what you mean by "backend" so its hard to give advice on design patterns. You said you are experienced in Python, have you ever developed a database driven web application before?

update

Based on your comment, I won't be able to help much as I don't have much experience doing "backends" like that. However, seeing as how you are pulling in resources from the web, your latency/throughput is going to be pretty high. So, in order to increase overall effectiveness, you are going to want to have something that can run multiple threads or processes with pretty high concurrency. I suggest you check out the answers on this thread (and search for similar ones):

Concurrent downloads - Python

Specifically, I found the example for the recursive web server and the example following it to probably be a very good start on your solution:

http://eventlet.net/doc/examples.html#recursive-web-crawler

As for taking that idea and then turning it into a robust/continuous process, that's going to depend a lot on your platform and how well you do error handling. Basically:

  • run it in a loop and make sure you handle any error that can possibly be thrown
  • have some kind process monitoring your worker process to kill/restart it if it hangs or dies
  • make sure you have a monitoring solution to notify you if it stops working (nagios, etc.)

One of the best ways to keep things "robust" is to make them as simple (not simplistic) as possible. If all you are doing is pulling in info from the web, parsing it in some way, and then storing that info in a DB, then try to keep the process that simple. Don't add unnecessarily complexity in an effort to make it more robust. If you end up with a 200 line script that does what you want, great!

Community
  • 1
  • 1
Randy Syring
  • 1,971
  • 1
  • 15
  • 19
  • thanks for the response! my system comprises of 3: front-end, db, and back-end. the data is fetched constantly by the back-end and inserted into the db, and the data is being displayed at the same time by the front-end. so the back-end is an infinite script that pulls data off the internet and writes it to the db. I developed Python scripts that write to databases, but nothing that runs infinitely and nothing that has to be extremely well designed and robust. Thanks! – Raiders Apr 14 '11 at 23:13
  • 1
    @Raiders: Do not post facts like this in a comment. Please **update** the question with relevant information. – S.Lott Apr 15 '11 at 03:07
0

Use Apache, Django and Piston.

Use REST as the protocol.

Write as little code as possible.

Django models, forms, and admin interface.

Piston wrapppers for your resources.

S.Lott
  • 384,516
  • 81
  • 508
  • 779
  • thank you so much! could you clarify how django would come into play here? I used it in the past to write actual websites, but it has its own data model and I need to use MySQL. Is this possible? about Piston, that will be a separate part of my system that runs an API right? – Raiders Apr 15 '11 at 12:47
  • @Raiders: "it has its own data model and I need to use MySQL"? I don't understand the question. Django uses MySQL as a backend. Piston is part of Django -- it enables RESTful access to the Django model objects. – S.Lott Apr 15 '11 at 13:08
  • reading about it more, I now understand. However I don't really see how to use django as a constant running backend framework - I need my system to run 24/7 at realtime. reading about django, I don't see where the forms come into play in my system, nor do I understand how to write a non-request serving django system.. Thanks again! – Raiders Apr 15 '11 at 13:11
  • @Raiders: Apache + mod_wsgi are what you use to run 24/7. "realtime" doesn't mean anything. Django forms are used for input validation. Please **read** the Django documentation. – S.Lott Apr 15 '11 at 13:14
  • thanks for the response. I spend a lot of time reading about these frameworks, and I think I have a better idea of how to approach matters. i will use django and Apache with mod_wsgi, but I need something to run tasks continuously. Is Celery the right choice here? Furthermore, I looked into Piston and its just a way to provide an API to my website. That great because I need that, but I don't see how to wrap my 'data fetchers' - aka the scripts that fetch data from the web. Where will that part be? – Raiders Apr 15 '11 at 19:20
  • @Raiders: "run tasks continuously"? Is that in the original question? If not please **update** the question to clarify this requirement. "Piston and its just a way to provide an API to my website". That's false. "how to wrap my 'data fetchers' - aka the scripts that fetch data from the web". That's what you implement in the Piston handlers. Please **read** the Piston documentation. You write code to subclass Piston classes. – S.Lott Apr 15 '11 at 19:58