4

I clone this toy repository that demonstrates how namespace packages work:

C:\workspace>git clone https://github.com/pypa/sample-namespace-packages.git

Specifically, I'll use its pkg_resources directory, which has the structure

pkg_a/
  setup.py
  example_pkg/
    __init__.py
    a/
      __init__.py
pkg_b/
  setup.py
  example_pkg/
    __init__.py
    b/
      __init__.py

The example_pkg package is a pkg_resources-style namespace package (explained here).

I set up my python environment:

C:\workspace>\Python35\python.exe -m venv localpython

C:\workspace>localpython\Scripts\activate.bat
(localpython) C:\workspace>python -m pip install --upgrade pip setuptools

I install pkg_a from the toy repository:

(localpython) C:\workspace>python -m pip install c:\workspace\sample-namespace-packages\pkg_resources\pkg_a

I put pkg_b from the toy repository on my PYTHONPATH:

(localpython) C:\workspace>set PYTHONPATH=c:\workspace\sample-namespace-packages\pkg_resources\pkg_b

I write a test suite for pkg_b, consisting of one line:

(localpython) C:\workspace>echo import example_pkg.b > test_b.py

Now, if I run that test suite in pytest 4.5 or less, it succeeds:

(localpython) C:\workspace>python -m pip install pytest==4.5.0
Collecting pytest==4.5.0
...

(localpython) C:\workspace>pytest test_b.py
================================================= test session starts =================================================
platform win32 -- Python 3.5.2, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
rootdir: C:\workspace
collected 0 items

============================================ no tests ran in 0.02 seconds =============================================

But if I run it in pytest 4.6 or greater, it errors:

(localpython) C:\workspace>python -m pip install pytest==4.6.0
Collecting pytest==4.6.0
...

(localpython) C:\workspace>pytest test_b.py
================================================= test session starts =================================================
platform win32 -- Python 3.5.2, pytest-4.6.0, py-1.8.0, pluggy-0.12.0
rootdir: C:\workspace
collected 0 items / 1 errors

======================================================= ERRORS ========================================================
_____________________________________________ ERROR collecting test_b.py ______________________________________________
ImportError while importing test module 'C:\workspace\test_b.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test_b.py:1: in <module>
    import example_pkg.b
E   ImportError: No module named 'example_pkg.b'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================== 1 error in 0.08 seconds ===============================================

The import error is understandable: It's presumably looking in the example_pkg in the site-packages and therefore not finding the b package, which is under the example_pkg on the PYTHONPATH.

Yet pytest 4.5 or earlier manages to find example_pkg.b.

Edit: According to a comment, the relevant difference is that pytest 4.5 and earlier imported pkg_resources. Indeed, if I add the line

import pkg_resources

at the top of my test file, test_b.py, then the test succeeds even in pytest 4.6 or greater. Furthermore, if I have multiple test files that try to import example_pkg.b, then it's necessary and sufficient to import pkg_resources in whichever of them pytest happens to run first.

That means that some side effect of importing pkg_resources makes example_pkg.b importable. What exactly does pkg_resources do that achieves that effect, and can I do it directly instead of getting it as a side effect of a seemingly unused import? And does my needing that side effect mean the setup is invalid?

shuttle87
  • 15,466
  • 11
  • 77
  • 106
ben-g-
  • 131
  • 9
  • I think you'd be better off raising this as [an issue against pytest itself](https://github.com/pytest-dev/pytest/issues), you're likely to get more specialised / knowledgeable answers there – Tom Dalton Jul 26 '19 at 15:41
  • @TomDalton I might do that. But it's sort of a question about pytest and sort of a question about my setup. That is, I think the question 'Should the test pass?' is equivalent to the question 'Is my setup valid (for testing)?'. I can see why someone might say the setup is not valid, but I think there should be a way to test a project that depends on a package with which it shares a namespace package. Or maybe someone will answer that a package should never share a namespace package with a package on which it depends. So the answer might not be pytest-related. – ben-g- Jul 26 '19 at 18:08
  • 2
    This has not much to do with `pytest` itself, but rather with the early import of `pkg_resources` which was removed in [PR #5063](https://github.com/pytest-dev/pytest/pull/5063). To bring back the old behaviour, add `import pkg_resources` in `test_b.py` before `import example_pkg.b`. – hoefling Jul 28 '19 at 14:41
  • @hoefling You're right, importing `pkg_resources` makes the test succeed. But it's not normal to have to import a name that's not referenced in the code, so it's not an entirely satisfying answer. I've updated the question accordingly. Thanks for the information! – ben-g- Jul 29 '19 at 10:56
  • 1
    First of all, don't forget that you're testing with old-style namespace packages that explicitly require `pkg_resources` to be importable. Second, I guess (didn't test though) that it's the installation of `example_pkg_a` that brings the difference here - once `pkg_resources` is imported, it loads dists into working set (if you're interested in the exact code spot, it's [here](https://github.com/pypa/setuptools/blob/5b128e8c82f4886b4ff110219de80a61bf8aa3b0/pkg_resources/__init__.py#L3276)), including the `example_pkg` import. – hoefling Jul 29 '19 at 11:22
  • Overall, the old-style namespace packages are IMO unusable and need to be dropped; had some rough time with them myself until I gave them up and never looked back, see for example [this question of mine](https://stackoverflow.com/q/39787557/2650249). – hoefling Jul 29 '19 at 11:25
  • @hoefling Okay, I guess importing `pkg_resources` is the answer. If you want to add that as an answer (rather than a comment), I'll mark it as accepted. – ben-g- Jul 29 '19 at 14:00

1 Answers1

0

Import your packages through sys as written in this project (using unittest not pytest)

import sys
sys.path.insert(0,"pkg_a")
sys.path.insert(0,"pkg_b")

#Test goes here

My project load test/testSpecie.py which tests files in src/...

IQbrod
  • 2,060
  • 1
  • 6
  • 28