3

In order to be able to pickle a nested function (for multiprocessing), I decorated the nested function with the decorator:

def globalize_one(func):
    def wrapper_one(*args,**kwargs):
        return func(*args,**kwargs)
    setattr(modules['__main__'],'sdfsdf',wrapper_one)
    return wrapper_one

However, this decorator does not work, when I ran this

def test_one():
    @globalize_one
    def inside_one():
        return 1

    try:
        pickle.dumps(inside_one)
    except Exception as e:
        print(e)

test_one()

I received the exception Can't pickle local object 'globalize_one.<locals>.wrapper_one'

To make the decorator work, all I need is to change the __qualname__ of wrapper_one to sdfsdf in globalize_one right before the line setattr(modules['__main__'],'sdfsdf',wrapper_one).

def globalize_two(func):
    def wrapper_two(*args,**kwargs):
        return func(*args,**kwargs)

    # the single extra line as compared to globalize_one
    wrapper_two.__qualname__ = 'sdfsdf'

    setattr(modules['__main__'],'sdfsdf',wrapper_two)
    return wrapper_two

def test_two():
    @globalize_two
    def inside_two():
        return 1

    try:
        pickle.dumps(inside_two)
    except Exception as e:
        print(e)

As you can see by running the code, the nested function inside_two can be pickled now.

My confusion is, why by changing the __qualname__, the decorator will work properly? I thought changing the name of a function have no real effect.

meTchaikovsky
  • 7,478
  • 2
  • 15
  • 34
  • I'm curious what you read that made you think of changing the `__qualname__`? – roganjosh Sep 27 '20 at 08:58
  • why do you need to pickle a function? I always thought of pickle as a way to store instance data on disc so you can retrieve this instance from disc again - but you would need the classes definition in the code that pickles and unpickles the file? (just curious, followed and uv to see how this Q develops) – Patrick Artner Sep 27 '20 at 09:03
  • 1
    @PatrickArtner because that's how multiprocessing works. The function is pickled before being passed to the separate processes. It's not just a file format but a way to serialize objects and pass them between processes – roganjosh Sep 27 '20 at 09:10
  • 1
    @PatrickArtner maybe the most relevant part of the docs is [here](https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues) but I'm not aware of a decent treatise on the overall use of pickle in the process – roganjosh Sep 27 '20 at 09:16
  • Not sure if you saw that alreadybut [here](https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function) they use a fork of multiprocess that can pickle "better" using _dill_ (which almost makes sense as dill is a herb used to pickle cucumbers to make them more tasty:D ) - thread is 8y old so not sure its still valid and applicable to your situation – Patrick Artner Sep 27 '20 at 09:49
  • @PatrickArtner to my knowledge, `dill` is still the go-to for this. I don't think we're any closer to answering this question, though. It seems very strange that `__qualname__` does anything with pickle to fix this issue. It's going on my watch list :) – roganjosh Sep 27 '20 at 10:01
  • @roganjosh Actually, I didn't think of changing the `__qualname__` and only have `setattr(modules['__main__'],'sdfsdf',wrapper)` in the first place. Then, I realized by only doing this, `python` will never know when to call `sdfsdf`, so I changed the `__qualname__` of `wrapper` to 'sdfsdf'. By running more tests, I think the solution lies within the way how `pickle` pickles a function, I have asked another [question](https://stackoverflow.com/q/64095346/8366805) about this, hope you can check it out :) – meTchaikovsky Sep 28 '20 at 05:54

0 Answers0