What offers better performance for large datasets? Nested dictionaries or a dictionary of objects?

Question

I find myself repeating this pattern when I am fetching from multiple database tables:

records = {'p_key': { "record": r, "A": list(), "B": list(), "C" : list() } for r in db_records}

I often have to group data this way because I cannot do joins across databases or there might be a situation where multiple queries is faster than multiple joins.

But performance-wise I am not sure if there is a lot of overhead to nesting dictionaries like this, and if I would be better served by creating an object with these attributes that becomes the value in the records dictionary. By performance I mean the overall cost in space and time when using a large set of nested dictionaries vs a dictionary of objects.

If you're doing joins you probably should consider using pandas. — Barmar, Nov 04 '19 at 19:15
I can't see how objects would play faster than dicts since (1) everything is basically an object, (2) almost every object has a `__dict__` — geckos, Nov 04 '19 at 19:22
*every* dict is a dict of objects. A dict is an object. Everything is an object in Python. Can you be more specific? — juanpa.arrivillaga, Nov 04 '19 at 19:25

score 2 · Accepted Answer · answered Nov 04 '19 at 19:46

There's basically no difference in performance between dictionaries and regular class objects because internally objects are using dictionaries to handle their attributes.

However, you should consider using classes with __slots__. Here is detailed explanation about what it is and its performance.

Another option is using pandas library to work with big dataset.

What offers better performance for large datasets? Nested dictionaries or a dictionary of objects?

1 Answers1