Dealing with lots of objects in a Python Program

Question

I am building a python program which will be working with thousands of python objects. I am curious about what is the most elegant way to organize them, and I am also not clear on when things get copied in python.

For example if I have a manager object which has a list of thousands of objects, but it also has a dictionary with keys to each of those thousands of objects, do the objects exist twice in memory? If I were to change the properties of an object in the list, will it also change the properties of the corresponding object in the dict?

I can't find much good reading on this, so if someone could point me in the right direction, that would be helpful.

Here is what I am thinking:

class Manager():
    def __init__(self,list_of_objects):
        self.myobjects = []
        self.myobjectsdict= {}

        self.myobjects = list_of_objects

        for object in self.myobjects:
            self.myobjectsdict[object.name] = object

Then, if I were to create a second manager object and give it some of the first manager's objects, would they be copies? Or would changes made by the second manager apply to the objects owed by the first? For instance, what happens here:

manager2 = Manager(manager1.myobjects[5000:10000])
manager2.myobjects[71].name = "newname"

Does the name change for that object in the first manager? Will the dict key still be the old name?

Finally, is a list or dict of thousands of objects really the best way to organize lots of python objects, or is there a better way I haven't come across? Thanks so much for any help.

Sounds like you're getting close to needing a database, ORM, and classes. — David Ehrmann, Jul 04 '16 at 04:09

score 0 · Accepted Answer · edited May 23 '17 at 12:32

This post talks about passing a variable by reference (Python passes by assignment).

[If] I have a manager object which has a list of thousands of objects, but it also has a dictionary with keys to each of those thousands of objects, do the objects exist twice in memory? If I were to change the properties of an object in the list, will it also change the properties of the corresponding object in the dict?

When you normally use an object name for a key or value in a dictionary, you're passing a reference to that object. If you were to update the some value in the object, the value in the dictionary (or list) would also be updated.

>>> my_object = MyClass() # Myclass is instantiated with x = 10
>>> my_dict = {"my_obj":my_object}
>>> my_dict["my_obj"] is my_object
True
>>> my_dict["my_obj"].x
10

If you were to place this dict inside of an object it would behave the same way.

Then, if I were to create a second manager object and give it some of the first manager's objects, would they be copies?

No, they would also be references. If you change the name from either of the managers, then the other manager is also affected.

Finally, is a list or dict of thousands of objects really the best way to organize lots of python objects, or is there a better way I haven't come across?

What are you using the Python objects for? Do you need to traverse through them? Are you trying to filter objects based on some criteria? Do you need to check quickly for the existence of some value? Is it necessary to keep thousands of objects in memory?

Without more information this part of the question can't be easily answered.

Thanks, that post you linked is exactly what I was looking for. — csc, Jul 04 '16 at 15:46

Dealing with lots of objects in a Python Program

1 Answers1