1

I have a massive data set of customer information (100s of millions of records, 50+ tables).

I am writing a python (twisted) app that I would like to interact with the dataset, performing table manipulation. What I really need is an abstraction of 'table', so I can add/remove/alter columns/tables without having to resort to only creating SQL.

Is there an ORM that will not add significant overhead to my application, considering the size of the dataset?

mikewaters
  • 3,668
  • 3
  • 28
  • 22
  • Since ORM implies SQL, the question is a little unclear. Can you clarify what "without having to resort to only creating SQL" means. – S.Lott Jan 21 '11 at 22:34

4 Answers4

1

There's twistar, a native ORM for twisted. It works on top of twisted.enterprise.adbapi.

http://findingscience.com/twistar/

fiorix
  • 3,619
  • 3
  • 19
  • 12
1

See

They seem to be in favor of SQLAlchemy

Community
  • 1
  • 1
Navi
  • 8,580
  • 4
  • 34
  • 32
0

I thought that ORM solutions had to do with DQL (Data Query Language), not DDL (Data Definition Language). You don't use ORM to add, alter, or remove columns at runtime. You'd have to be able to add, alter, or remove object attributes and their types at the same time.

ORM is about dynamically generating SQL and developer's lift, not what you're alluding to.

duffymo
  • 305,152
  • 44
  • 369
  • 561
  • I'd left out the fact that I require massive amounts of dql, but your answer was the most appropriate. Thank you. – mikewaters Jan 21 '11 at 23:39
0

If I'd have to pick the Python ORM, I would pick SQLAlchemy.

But with Twisted you have to be cautious since Twisted is asynchronous.

See: Twisted + SQLAlchemy and the best way to do it

Community
  • 1
  • 1
esamatti
  • 18,293
  • 11
  • 75
  • 82