What is the purpose of deepcopy's second parameter, memo?

Question

from copy import* 
a=[1,2,3,4]
c={'a':'aaa'}
print c
#{'a': 'aaa'}
b=deepcopy(a,c)
print b

print c
# print {'a': 'aaa', 10310992: 3, 10310980: 4, 10311016: 1, 11588784: [1, 2, 3, 4, [1, 2, 3, 4]], 11566456: [1, 2, 3, 4], 10311004: 2}

why c print that

Please try to use the code, rather than text, because my English is not very good, thank you

in django.utils.tree.py

def __deepcopy__(self, memodict):
        """
        Utility method used by copy.deepcopy().
        """
        obj = Node(connector=self.connector, negated=self.negated)
        obj.__class__ = self.__class__
        obj.children = deepcopy(self.children, memodict)
        obj.subtree_parents = deepcopy(self.subtree_parents, memodict)
        return obj



import copy
memo = {}
x1 = range(5)
x2=range(6,9)
x3=[2,3,4,11]
y1 = copy.deepcopy(x1, memo)
y2=copy.deepcopy(x2, memo)
y3=copy.deepcopy(x3,memo)
print memo
print id(y1),id(y2),id(y3)
y1[0]='www'
print y1,y2,y3
print memo

print :

{10310992: 3, 10310980: 4, 10311016: 1, 11588784: [0, 1, 2, 3, 4, [0, 1, 2, 3, 4]], 10311028: 0, 11566456: [0, 1, 2, 3, 4], 10311004: 2}
{11572448: [6, 7, 8], 10310992: 3, 10310980: 4, 10311016: 1, 11572368: [2, 3, 4, 11], 10310956: 6, 10310896: 11, 10310944: 7, 11588784: [0, 1, 2, 3, 4, [0, 1, 2, 3, 4], 6, 7, 8, [6, 7, 8], 11, [2, 3, 4, 11]], 10311028: 0, 11566456: [0, 1, 2, 3, 4], 10310932: 8, 10311004: 2}
11572408 11581280 11580960
['www', 1, 2, 3, 4] [6, 7, 8] [2, 3, 4, 11]
{11572448: [6, 7, 8], 10310992: 3, 10310980: 4, 10311016: 1, 11572368: [2, 3, 4, 11], 10310956: 6, 10310896: 11, 10310944: 7, 11588784: [0, 1, 2, 3, 4, [0, 1, 2, 3, 4], 6, 7, 8, [6, 7, 8], 11, [2, 3, 4, 11]], 10311028: 0, 11566456: ['www', 1, 2, 3, 4], 10310932: 8, 10311004: 2}

Answer 1

No one above gave a good example of how to use it.

Here's what I do:

def __deepcopy__(self, memo):
    copy = type(self)()
    memo[id(self)] = copy
    copy._member1 = self._member1
    copy._member2 = deepcopy(self._member2, memo)
    return copy

Where member1 is an object not requiring deepcopy (like a string or integer), and member2 is one that does, like another custom type or a list or dict.

I've used the above code on highly tangled object graphs and it works very well.

If you also want to make your classes pickleable (for file save / load), there is not analogous memo param for getstate / setstate, in other words the pickle system somehow keeps track of already referenced objects, so you don't need to worry.

The above works on PyQt5 classes that you inherit from (as well as pickling - for instance I can deepcopy or pickle a custom QMainWindow, QWidget, QGraphicsItem, etc.)

If there is some initialization code in your constructor that creates new objects, for instance a CustomWidget(QWidget) that creates a new CustomScene(QGraphicsScene), but you'd like to pickle or copy the scene from one CustomWidget to a new one, then one way is to make a new=True parameter in your __init__ and say:

def __init__(..., new=True):
    ....
    if new:
       self._scene = CustomScene()

def __deepcopy__(self, memo):
    copy = type(self)(..., new=False)
    ....
    copy._scene = deepcopy(self._scene, memo)
    ....

That ensures you don't create a CustomScene (or some big class that does a lot of initializing) twice! You also should use the same setting (new=False) in your __setstate__ method, eg.:

 def __setstate__(self, data):
     self.__init__(...., new=False)
     self._member1 = data['member 1']
     .....

There are other ways to get around the above, but this is the one I converged to and use frequently.

Why did I talk about pickling as well? Because you will want both in any application typically, and you maintain them at the same time. If you add a member to your class, you add it to setstate, getstate, and deepcopy code. I would make it a rule that for any new class you make, you create the above three methods if you plan on doing copy / paste an file save / load in your app. Alternative is JSON and save / loading yourself, but then there's a lot more work for you to do including memoization.

So to support all the above, you need __deepcopy__, __setstate__, and __getstate__ methods and to import deepcopy:

from copy import deepcopy

, and when you write your pickle loader / saver functions (where you call pickle.load()/ pickle.dump() to load / save your object hierarchy / graph) do import _pickle as pickle for the best speeds (_pickle is some faster C impl which is usually compatible with your app requirements).

Answer 2

It's the memo dict, where id-to-object correspondence is kept to reconstruct complex object graphs perfectly. Hard to "use the code", but, let's try:

>>> import copy
>>> memo = {}
>>> x = range(5)
>>> y = copy.deepcopy(x, memo)
>>> memo
{399680: [0, 1, 2, 3, 4], 16790896: 3, 16790884: 4, 16790920: 1,
 438608: [0, 1, 2, 3, 4, [0, 1, 2, 3, 4]], 16790932: 0, 16790908: 2}
>>> 

and

>>> id(x)
399680
>>> for j in x: print j, id(j)
... 
0 16790932
1 16790920
2 16790908
3 16790896
4 16790884

so as you see the IDs are exactly right. Also:

>>> for k, v in memo.items(): print k, id(v)
... 
399680 435264
16790896 16790896
16790884 16790884
16790920 16790920
438608 435464
16790932 16790932
16790908 16790908

you see the identity for the (immutable) integers.

So here's a graph:

>>> z = [x, x]
>>> t = copy.deepcopy(z, memo)
>>> print id(t[0]), id(t[1]), id(y)
435264 435264 435264

so you see all the subcopies are the same objects as y (since we reused the memo).

Answer 3

You can read more by checking the Python online documentation:

http://docs.python.org/library/copy.html

The deepcopy() function is recursive, and it will work its way down through a deeply nested object. It uses a dictionary to detect objects it has seen before, to detect an infinite loop. You should just ignore this dictionary.

class A(object):
    def __init__(self, *args):
        self.lst = args

class B(object):
    def __init__(self):
        self.x = self

def my_deepcopy(arg):
    try:
        obj = type(arg)()  # get new, empty instance of type arg
        for key in arg.__dict__:
            obj.__dict__[key] = my_deepcopy(arg.__dict__[key])
        return obj
    except AttributeError:
        return type(arg)(arg)  # return new instance of a simple type such as str

a = A(1, 2, 3)
b = B()
b.x is b  # evaluates to True
c = my_deepcopy(a)  # works fine
c = my_deepcopy(b)  # stack overflow, recurses forever

from copy import deepcopy
c = deepcopy(b)  # this works because of the second, hidden, dict argument

Just ignore the second, hidden, dict argument. Do not try to use it.

Answer 4

Here's a quick illustration I used for explaining this to myself:

a = [1,2,3]
memo = {}
b = copy.deepcopy(a,memo)
# now memo = {139907464678864: [1, 2, 3], 9357408: 1, 9357440: 2, 9357472: 3, 28258000: [1, 2, 3, [1, 2, 3]]}

key = 139907464678864
print(id(a) == key)               #True
print(id(b) == key)               #False
print(id(a) == id(memo[key]))     #False
print(id(b) == id(memo[key]))     #True

in other words:

memo[id_of_initial_object] = copy_of_initial_object