importlib_resources backport not compatible with pyfakefs?

Question

I'm refactoring a library to use importlib.resources for python 3.7+. I'm using the importlib_resources backport for python 3.6 compatibility. The code works for pythons 3.6-3.8. However, the pytest tests, using pyfakefs, fail for 3.6. Under testing conditions, the path returned from use of importlib_resources is mangled (but under "real-world" conditions they return correctly).

A minimal example: I have the following library structure:

mypackage
├── mypackage
│   ├── __init__.py
│   ├── bin
│   │   └── __init__.py
│   └── find_bin.py
└── tests
    └── test_find_bin.py

In the actual library, the bin folder holds binaries (plus an empty __init__). Code elsewhere in the library needs a path to it. find_bin.py demonstrates a function that will return the path:

import sys
if sys.version_info >= (3, 7):
    from importlib import resources
else:
    import importlib_resources as resources

import mypackage.bin


def find_bin():
    init_path_context = resources.path(mypackage.bin, '__init__.py')
    with init_path_context as p:
        init_path = p
    bin_path = init_path.parent
    return bin_path

The pytest test test_find_bin.py:

import pathlib

from mypackage.find_bin import find_bin


def test_findbin(fs):
    test_bin = (pathlib.Path(__file__)
                .resolve()
                .parent.parent
                .joinpath('mypackage', 'bin'))
    print('test bin is ', test_bin)
    expected_bin = find_bin()
    print('expected bin is ', expected_bin)
    assert not expected_bin.exists()
    print('test creating fakefs ', test_bin)
    fs.create_dir(test_bin)
    assert expected_bin.exists()

Python 3.7+ work as expected with the tests. In python 3.6, the expected_bin path is mangled:

test bin is  /Users/geoffreysametz/Documents/mypackage/mypackage/bin  # correct
expected bin is  /var/folders/bv/m5cg5cp15t38sh8rxx244hp00000gn/T  # ?!?!?!

I tried to trace through the execution of the find_bin function, and it's long and convoluted. However, I saw that importlib_resources makes use of python's FakeFilesystem class. My hypothesis is that the problem arises from both importlib_resources and pytest using fake file systems at the same time.

Is my hypothesis correct? Is there a workaround to getting pytest to test code that uses importlib_resources?

Can you add an issue in [pyfakefs](https://github.com/jmcgeheeiv/pyfakefs) please? I will have a look later. — MrBean Bremen, Feb 05 '20 at 08:35

score 1 · Accepted Answer · answered Feb 05 '20 at 19:28

The fake filesystem in pyfakefs is empty at test start (except for the temp file path needed by the tempfile module), so if you want to access any files in the real filesystem in your test, you have to map them into the fake filesystem:

def test_findbin(fs):
    test_bin = (pathlib.Path(__file__)
                .resolve()
                .parent.parent
                .joinpath('mypackage', 'bin'))
    fs.add_real_directory(test_bin)
...

In your real test, you don't know the path in advance, so you have to add some parent path that you know to contain your resources. The subdirectories and files will be read and copied to the fake filesystem on access.

As to the behavior of your test in Python 3.6: the code in the contextlib_package first checks if the (correct) path exists, and if it doesn't (as it checks in the fake filesystem, it won't exist), it creates a temporary file where it tries to copy some data - thus the temp path in your output.

In Python 3.8, it seems not to check for the existence of the path, it just creates a Path object from the correct path. This would fail anyway the moment you try to access some resources in that path, as they don't exist in the fake filesystem, so you have to map the real file system in this case, too.

(copied and slightly adapted from the answer in the pyfakefs issue)

importlib_resources backport not compatible with pyfakefs?

1 Answers1