0

Unpickling objects in a pytest test causes an exception where the pickle module can't find the pickled class:

$ pytest
======================================================================== test session starts ========================================================================
platform win32 -- Python 3.7.9, pytest-7.1.3, pluggy-1.0.0
rootdir: C:\Users\me\enum_test
collected 1 item

test_pickling.py F                                                                                                                                             [100%] 

============================================================================= FAILURES ============================================================================== 
__________________________________________________________________________ test_read_enums __________________________________________________________________________ 

    def test_read_enums():
        with open("pickles.json", 'rt') as f_in:
            d = json.load(f_in)
        for key, val in d.items():
>           loaded_container = pickle.loads(b64.decodebytes(val.encode()))
E           AttributeError: Can't get attribute 'ContainerClass' on <module '__main__' from 'C:\\Users\\me\\enum_test\\venv\\Scripts\\pytest.exe\\__main__.py'>

test_pickling.py:31: AttributeError
====================================================================== short test summary info ====================================================================== 
FAILED test_pickling.py::test_read_enums - AttributeError: Can't get attribute 'ContainerClass' on <module '__main__' from 'C:\\Users\\me\\enum_test\\...
========

This is produced from the following example:

import pickle
import json
import base64 as b64


class ContainerClass:
    def __init__(self, value):
        self.value = value

    def __repr__(self):
        return str(self.value)


def construct_pickles():
    d = {}
    for value in range(10):
        obj = ContainerClass(value)
        d[value] = b64.encodebytes(pickle.dumps(obj)).decode()
    with open("pickles.json", 'wt') as f_out:
        json.dump(d, f_out)


def test_pickles():
    with open("pickles.json", 'rt') as f_in:
        d = json.load(f_in)
    for key, val in d.items():
        loaded_container = pickle.loads(b64.decodebytes(val.encode()))
        print(loaded_container.value)
        assert loaded_container.value == int(key)
        print("It worked.")


if __name__ == '__main__':
    construct_pickles()
    test_pickles()

When run as python example.py the test_pickles function runs fine. With the file constructed and present, running pytest in the same directory causes the exception above.

I opened a bug on pytest believing it was a bug, but was told this was user error. Specifically, that I should deploy a console_script entrypoint. But as far as I can tell this only relates to setup.py, and since I'm not packaging this test suite, I have no use for setup.py.

Is the only way to unpickle in a pytest environment to use a custom unpickler as in this answer?

mas
  • 1,155
  • 1
  • 11
  • 28
  • When you pickle ContainerClass, it's part of the ``__main__`` module. When you run pytest, your ``__main__`` module doesn't contain it. You could try creating a package or module that you import into python_example.py. See if that helps. – saquintes Sep 21 '22 at 13:35
  • I did try moving ContainerClass to another file and importing it, if that's what you mean by module, and the problem persisted. I haven't tried a package and I don't like how heavy that solution is. – mas Sep 21 '22 at 14:18
  • You could also try using pytest fixtures which will create the pickle files in the same "context" as what's unpickling them. – saquintes Sep 21 '22 at 15:20
  • The point of the tests though are to ensure backward compatibility with older pickle files, so the files need to be precreated and revisioned and pinned. – mas Sep 21 '22 at 17:25

0 Answers0