1

I can't show much code, but see this sample:

class Product(metaclass=abc.ABCMeta):
    __attributes__ = []
    __special_attributes__ = []
    __json_additional_attributes__ = []

    def __init__(self, original_object):
        for attribute in [c_attr for c_attr in self.__attributes__
                            if c_attr not in self.__special_attributes__]:
            try:
                if hasattr(original_object, attribute):
                    # ...
                elif original_object.IsAttributeUsed(attribute):
                    # RAISES HERE - this is a clr binding
                    # ...
            except Exception as err:
                print('cant retrieve attr ' + attribute)

    # ...

I've removed lines for concision. Here, original_object is an object obtained from a clr binding. Sometimes original_object.IsAttributeUsed(attribute) raises an exception on the DLL side. The exception is caught as expected in the except block.

The problem is: for some reason, it leaks memory because the traceback object from this exception never gets collected. The traceback holds reference to the whole frames in the stack which in turn reference all locals. This is a batch, so this stack holds up 8Gb of memory, so I run pretty quickly out of memory.

I've got the graph of the references thanks to the objgraph package, and the references path from leaked objects go up to that traceback object, then nothing. If no exception is thrown, then the memory is freed.

Here is the graph:

objgraph

Sorry for big image, I've cropped. Stacks on the left are me playing with the debugger, don't mind them. I've highlighted the leak.

martineau
  • 119,623
  • 25
  • 170
  • 301
BiAiB
  • 12,932
  • 10
  • 43
  • 63
  • See [**_Why is there a need to explicitly delete the sys.exc_info() traceback?_**](https://stackoverflow.com/questions/1658293/why-is-there-a-need-to-explicitly-delete-the-sys-exc-info-traceback) – martineau Jun 21 '17 at 17:55
  • If you do `import gc` and then `gc.collect()` do you still see the leak? – MSeifert Jun 21 '17 at 17:58
  • @MSeifret yes, and even when calling clr gc – BiAiB Jun 21 '17 at 23:17
  • @martineau I'm not referering to the traceback anywhere , why would I need to delete it ? – BiAiB Jun 22 '17 at 08:07
  • In Python 3, a traceback is attached to the exception when it's created (see [this answer](https://stackoverflow.com/a/14564261/355230)), so you don't need to explicitly call `sys.exc_info()` for a reference to be created. Your code has one stored in the local variable `err`. According to [this answer](https://stackoverflow.com/a/1658335/355230): a "traceback contains references to all the active frames, which in turn contain references to all the local variables in those various frames". You might be able to avoid the problem by using just `except Exception:`. – martineau Jun 22 '17 at 11:40
  • @martineau thanks for your commen t, I will try that ! I have read this answer in saw in comments: `Python 3 resolves the issue by deleting the exception target on exiting the except block, per PEP 3110`. Should that be solving this problem ? – BiAiB Jun 23 '17 at 15:10
  • BiAiB: Yes, that sounds like it _should_ — perhaps there's a bug somewhere...you mentioned something about the exception being raised on the DLL side, so the bug, if there is one, may have something to do with that. – martineau Jun 23 '17 at 18:18
  • BiAiB: Perhaps you can explicitly delete it using a `finally:` clause as shown in [PEP 3110](https://www.python.org/dev/peps/pep-3110/#semantic-changes) (even though technically, according to the PEP, this shouldn't be necessary). – martineau Jun 23 '17 at 18:30

0 Answers0