Efficient method to store Python dictionary on disk?

Question

What is the most efficient method to store a Python dictionary on the disk? The only methods I know of right now are plain-text and the pickle module.

Edit: Sorry for not being very clear. By efficient I meant fastest execution speed. The dictionary will contain mutable objects that will hold information to be parsed and modified.

This depends a lot on exactly how you want to use the dictionary. With your question as stated, there is no one good answer. — Omnifarious, Jan 14 '10 at 22:28
Fastest execution speed compared to what? Are you reimplementing a database in a Python dictionary? How big is the dictionary? SQLite or BDB might be faster if the number of entries is high enough. — jmucchiello, Jan 14 '10 at 22:59
The number of keys will go in the thousands. Would a dictionary suffer noticeably in performance compared to a database with this number of values? — linkmaster03, Jan 15 '10 at 01:23
Link to related question, [database - Python Disk-Based Dictionary - Stack Overflow](https://stackoverflow.com/questions/226693/python-disk-based-dictionary) — user202729, Apr 18 '23 at 03:13

jspcal · Accepted Answer · 2010-01-14T23:02:54.533

9

shelve is pretty nice as well

or this persistent dictionary recipe

for a convenient method that keeps your objects synchronized with storage, there's the ORM SQLAlchemy for python

if you just need a way to store string values by some key, theres the dbm.ndbm and dbm.gnu modules.

if you need a hyper efficient, distributed key value cache, something like memcached for python...

edited Jan 14 '10 at 23:02

answered Jan 14 '10 at 21:36

jspcal

50,847
7
72
76

`shelve` is usually all you'll need, unless you need super-fast performance, in which case you should be using an actual database anyway. – Sasha Chedygov Jan 14 '10 at 21:59
Does shelve use cPickle when possible? – linkmaster03 Jan 14 '10 at 22:17
it indeed uses cPickle if available, you could override it by setting shelve.Pickler as well – jspcal Jan 14 '10 at 22:57
Thanks. I implemented my dictionary using shelve and it works great. – linkmaster03 Jan 18 '10 at 01:15

S.Lott · Answer 2 · 2010-01-14T22:49:06.067

2

JSON and YAML work well, also.

Depends on what you mean by "efficient"? Size of file? Time required? Amount of code you need to write?

You have the timeit module available to determine what meets your specific criteria for "efficient".

edited Jan 14 '10 at 22:49

answered Jan 14 '10 at 22:23

S.Lott

384,516
81
508
779

@linkmaster03: Please update your question with additional, useful facts. Hiding important information on comments on an answer doesn't help anyone understand your question. – S.Lott Jan 14 '10 at 22:47

Efficient method to store Python dictionary on disk?

2 Answers2