Searchable database-like structure for python objects?

Question

I am looking for a database-like structure which can hold python objects, each of which has various fields which can be searched. Some searching yields ZODB but I'm not sure that's what i want.

To explain, I have objects which can be written/read from disk in a given file format. I would like a way to organize and search many sets of those objects. Currently, I am storing them in nested dictionaries, which are populated based on the structure of the file-system and file names.

I would like a more data-base like approach, but i don't think i need a database. i would like to be able to save the structure to disk, but don't want to interface a server or anything like that. perhaps i just want to use numpy's structured arrays? http://docs.scipy.org/doc/numpy/user/basics.rec.html

Please extend your question or comment the answers, if you're not happy with the answers yet. Otherwise, please check the answered button. Thanks! — Michael, Oct 02 '12 at 08:43

score 3 · Accepted Answer · answered Sep 07 '12 at 17:11

3

Python comes with a database... http://docs.python.org/library/sqlite3.html

answered Sep 07 '12 at 17:11

thebjorn

26,297
11
96
138

1

Note that sqlite3 is an embedded SQL database, no server required. Ideal for the use-case of the OP. – Martijn Pieters Sep 07 '12 at 17:17
it doesnt appear that you can store a python object in a sqlite database. http://stackoverflow.com/questions/2047814/is-it-possible-to-store-python-class-objects-in-sqlite – alex Sep 07 '12 at 17:29
@alex: You can use pickle/json for the python object to store in a database – GodMan Sep 07 '12 at 17:31
pickling my object fails. but this solution is probably most practical in general – alex Oct 09 '12 at 16:32

score 1 · Answer 2 · answered Sep 07 '12 at 17:18

1

I think what you are looking for is python shelve if you are just looking to save the data to disk. It is also almost 100% compliant with the dict interface so you will have to change very little code.

answered Sep 07 '12 at 17:18

anijhaw

8,954
7
35
36

score 1 · Answer 3 · answered Sep 26 '12 at 22:31

ZODB will do what you are looking for. You can have your data structured there exactly as you have it already. Instead of using builtin dicts, you'll have to switch to persistent mappings, and if they tend to be large, you maybe should start with btrees right from the beginning. Here is some good starting point for further reading with simple examples and tutorials. As you'll see, ZODB doesn't provide some SQL-like access language. But as it looks like you already have structured your data to fit into some dictionaries, you probably won't miss that anyways. ZODB compared to plain shelve also offers to zip your data on the fly before writing it to disk. In general, it provides a lot more options than shelve, and shows now significant downside compared to that, so I can just warmly recommend it. I switched to it a year ago, because typical relational DBs together with ORM performed so horribly slow especially for insert-or-update operations. With ZODB I'm about an order of magnitude faster here, as there is no communication overhead between the application and the database, as the database is part of your application already.

Searchable database-like structure for python objects?

3 Answers3