Recursion error with Python 3 copy.deepcopy

Question

I am currently using the BioPython Bio.PDB module to parse and read protein structure files (PDB files). I have run into recursion issues when trying to create a deep copy of the structure object returned by the PDBParser().get_structure(...) method.

I was under the impression that the copy.deepcopy function was designed to deal with recursion? Strangely, this is only an issue using Python 3. Running equivalent code in Python 2.7 works fine (uses urllib2 instead of urllib).

Is this a bug in copy.deepcopy? Or an issue with Biopython?

A minimal example (Python 3.6, Biopython 1.68) is:

import urllib.request
from io import StringIO
from copy import deepcopy
import Bio.PDB

pdb_name = '1zro'
#Download file
pdb_url = "http://www.rcsb.org/pdb/files/" + pdb_name + ".pdb"
pdb_file = urllib.request.urlopen(pdb_url)
#Create string stream for Bio.PDB to read:
pdb_input_stream = StringIO(pdb_file.read().decode('utf-8'))

#Make Bio.PDB structure object
parser = Bio.PDB.PDBParser()
structure = parser.get_structure(pdb_name, pdb_input_stream)

#Attempt to perform a deepcopy on Bio.PDB Structure:
structure_copy = deepcopy(structure)

Full traceback:

  File "<stdin>", line 2, in <module>
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/copy.py", line 281, in _reconstruct
    if hasattr(y, '__setstate__'):
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/site-packages/Bio/PDB/Entity.py", line 207, in __getattr__
    if not hasattr(self, 'selected_child'):
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/site-packages/Bio/PDB/Entity.py", line 207, in __getattr__
    if not hasattr(self, 'selected_child'):
  File "/home/andrew/Programs/anaconda3/envs/py36/lib/python3.6/site-packages/Bio/PDB/Entity.py", line 207, in __getattr__
    if not hasattr(self, 'selected_child'):
  [Previous line repeated 318 more times]
RecursionError: maximum recursion depth exceeded

You might be interesting about this: http://stackoverflow.com/questions/3323001/what-is-the-maximum-recursion-depth-in-python-and-how-to-increase-it — sangheestyle, Mar 03 '17 at 04:26
Thanks @sangheestyle, I have already tried increased the recursion limit to stupid levels, and have achieved nothing more than a segfault. But that was my first thought as well. — Andrew Guy, Mar 03 '17 at 04:27
Just googled _"python3.6 deepcopy issues"_ and looking at some of the posts, namely [this](https://github.com/cloudtools/troposphere/issues/648) one. It could quite possibly be an issue with deepcopy for 3.6, see if python3.5 gives same issue since I've seen a couple problems arise due to 3.6 — Steven Summers, Mar 03 '17 at 04:39
Have a look at [this recipe](https://code.activestate.com/recipes/302535/). I haven't tested it, but it might help. And, according the [documentation](https://docs.python.org/3/library/copy.html), this is always something to watch out for when using deepcopy: "Recursive objects (compound objects that, directly or indirectly, contain a reference to themselves) may cause a recursive loop." — Dan, Mar 03 '17 at 04:42
@DanWelty, yep, that's one option for an immediate fix. More interested in solving the issue for deepcopy though, as that seems to be the go-to tool when performance isn't an issue. — Andrew Guy, Mar 03 '17 at 04:48
@AndrewGuy Do the test cases in the post mentioned by DanWelty also have the same result for Python 2.7? — Steven Summers, Mar 03 '17 at 04:55
@StevenSummers The test cases in DanWelty's post have the same result in both 2.7 and 3.6, unlike my scenario. — Andrew Guy, Mar 03 '17 at 04:59
@AndrewGuy Maybe submit an issue on the github page to check. But I can still only think that deepcopy for python 3 is the issue. Maybe check the max recursion depth of the structure to see how deep it goes before exiting on both versions. Also try `>>> deepcopy(nested(496))` should work, but any higher and it hits the limit in 3.5 on my device (Windows 10). — Steven Summers, Mar 03 '17 at 05:14
@StevenSummers Thanks for the link to that troposphere issue. The fix suggested there solves the issue for BioPython as well - cheers for the pointer in the right direction! — Andrew Guy, Mar 07 '17 at 01:49

score 1 · Answer 1 · answered Mar 06 '17 at 16:41

1

Andrew kindly filed this as Biopython issue https://github.com/biopython/biopython/issues/787 - although we're still not sure what is going wrong under Python 3 here.

answered Mar 06 '17 at 16:41

Peter Cock

1,585
1
9
14

Thanks for flagging that Peter. I'm more and more convinced that it's a `deepcopy` issue it its root, but haven't yet figured out what exactly is going on. Going to try and put together a minimal example that replicates the issues when I have time. – Andrew Guy Mar 06 '17 at 23:10

Recursion error with Python 3 copy.deepcopy

1 Answers1