0

Issue: pytest stopped generating coverage

Description: I upgraded python version from 3.6 to 3.11 (Ubuntu 18.04). I followed this https://www.debugpoint.com/install-python-3-11-ubuntu/ in order to do that. Had issues with pip, followed this advice: https://stackoverflow.com/a/72685573/15923186

I have a project with following structure:

test.sh
|-sorting
  |- env (the virtual environment for sorting)
  |- requirements.txt
  |- pyproject.toml
  |- __init__.py
  |- merge_sort.py
  |- tests
     |-__init__.py
     |-test_merge_sort.py
  
|-something_else (etc.)

requirements.txt are following:

attrs==22.1.0
black==22.8.0
click==8.0.4
coverage==6.2
flake8==5.0.4
importlib-metadata==4.2.0
iniconfig==1.1.1
mccabe==0.7.0
mypy-extensions==0.4.3
packaging==21.3
pathspec==0.9.0
platformdirs==2.4.0
pluggy==1.0.0
py==1.11.0
pycodestyle==2.9.1
pyflakes==2.5.0
pyparsing==3.0.9
pytest==7.0.1
pytest-cov==4.0.0
pytest-cover==3.0.0
tomli==1.2.3
typing-extensions==4.1.1
zipp==3.6.0

Pyproject is following:

[tool.pytest.ini_options]
pythonpath = [
  "."
]

The test script is following:

#!/bin/bash

source "$1"/env/bin/activate

cd "$1" || { echo "Project not found!"; exit 1;}

pytest -vv --cov="$1" --cov-report xml --cov-report term

And is invoked like this: ./test.sh sorting

I'm not "reusing" old virtual environments, I double checked. Before running tests I prepared a new one like this (the same as previously when using python 3.6):

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

Python 3.11 is the default one for python3 command (output below):

> Python 3.11.0 (main, Oct 24 2022, 19:56:01) [GCC 7.5.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> 

Result: Nothing displayed, no coverage report generated

Expected result: Coverage report generated and displayed in the terminal

I also tried to run with:

python3 -m pytest -vv --cov="$1" --cov-report xml --cov-report term

No difference.

If I remove

 --cov="$1" --cov-report xml --cov-report term

from the test.sh script I get following output (which is good and expected)

test session starts ================================================================
platform linux -- Python 3.11.0, pytest-7.0.1, pluggy-1.0.0 -- /path/to/the/env/which/is/irrelevant/sorting/env/bin/python3
cachedir: .pytest_cache
rootdir: /home/mysusrname/folder_name/subfolder_name/sorting, configfile: pyproject.toml
plugins: cov-4.0.0
collected 3 items                                                                                                                                  

tests/test_merge_sort.py::test_merge_sort[numbers0] PASSED                                                                                   [ 33%]
tests/test_merge_sort.py::test_merge_sort[numbers1] PASSED                                                                                   [ 66%]
tests/test_merge_sort.py::test_merge_sort[numbers2] PASSED                                                                                   [100%]

3 passed in 0.01s =================================================================
Gameplay
  • 1,142
  • 1
  • 4
  • 16
  • Did you reinstall `pytest` after upgrading to 3.11? Typically, commands like `pytest`, `pip3`, etc. are kind of dangerous, because if you upgrade the underlying Python, the version of Python invoked with `python3` will have upgraded, but the commands built on it will not have upgraded. – ShadowRanger Dec 05 '22 at 15:53
  • There is a known issue with `pytest-cov` on Python 3.11, see: https://github.com/pytest-dev/pytest-cov/issues/564. – Alex Rothberg Feb 03 '23 at 21:46

2 Answers2

1

According to your requirement.txt, you are using pytest 7.0.1 which came out before python 3.11 (https://docs.pytest.org/en/stable/changelog.html#pytest-7-0-1-2022-02-11), which makes it unlikely to support it as it didn't exist at that time. (N-B: sometimes it works, but it is not officially supported by the developers of pytest)

For your case, as you need to upgrade several Python version at the time, I would use the following process to upgrade Python version and dependencies.

  • In Python 3.6, upgrade all dependencies to the latest version that supports Python 3.6
  • Test
  • Upgrade to Python 3.7
  • Test
  • In Python 3.7, upgrade all dependencies to the latest version that supports Python 3.7
  • Test
  • Keep going

It is of course possible to skip some python version to speed up the process, but you might run into the same issues as you encountered.

Usually, there are between 3 and 4 supported version of Python out at the same time (https://endoflife.date/python), which means that most library will support those versions (not always the case, for example numpy support less versions: https://numpy.org/neps/nep-0029-deprecation_policy.html), so it is usually possible to skip 1 version or 2. 3 is likely to not work.

Coding thermodynamist
  • 1,340
  • 1
  • 10
  • 18
  • Sounds reasonable, but shouldn't it be fixed by removing version from pytest in requirements.txt and recreating the venv? I should end up with latest version supported by python 3.11. But either that's not the case or I'm missing something (it's note working). – Gameplay Dec 05 '22 at 18:31
  • Also sounds like a bit messy solution, cause I'll end up with multiple 3.x versions (or uninstall each after step 3.7, 3.8,3.9,3.10) and I'm going towards having a single version. – Gameplay Dec 05 '22 at 18:35
  • Having several version of Python is not messy as long as you now each time which one your are using (I have personally 6 different version of Python installed and each project use a virtualenv and activate a specific one). You could also try your suggestion with removing the version of pytest from `requirements.txt`: it might work but you might encounter the same issue with other package and it is harder to detect breaking changes in your dependencies. The solution I proposed is the complicated but guaranteed to work solution, and it has the benefit to be doable incrementally. – Coding thermodynamist Dec 06 '22 at 09:04
0

So I couldn't agree with @Nathan and kept digging and actually my first comment is right and seems to be faster and imho cleaner solution.

After upgrading python like I described and having a requirements.txt with fixed versions:

  • delete the old virtual environment

  • remove versions from requirements.txt (example below):

      attrs
      black
      click
      coverage
      flake8
      importlib-metadata
      iniconfig
      mccabe
      mypy-extensions
      packaging
      pathspec
      platformdirs
      pluggy
      py
      pycodestyle
      pyflakes
      pyparsing
      pytest
      pytest-cov
      pytest-cover
      tomli
      typing_extensions
      zipp
    
  • after creating a "fresh" venv and sourcing it run pip install --upgrade pip to make sure the pip is up to date

  • install the packages (utilizing the requirements.txt with NO versions specified in the venv mentioned above) pip install -r requirements.txt

  • the newest versions supported for your python distro will be used in such case

  • run pip freeze > requirements.txt to overwrite the requirements file and again have stable version of it with versions of libraries fixed (example corresponding "clean" file below)

      attrs==22.1.0
      black==22.12.0
      click==8.1.3
      coverage==6.5.0
      flake8==6.0.0
      importlib-metadata==5.1.0
      iniconfig==1.1.1
      mccabe==0.7.0
      mypy-extensions==0.4.3
      packaging==22.0
      pathspec==0.10.3
      platformdirs==2.6.0
      pluggy==1.0.0
      py==1.11.0
      pycodestyle==2.10.0
      pyflakes==3.0.1
      pyparsing==3.0.9
      pytest==7.2.0
      pytest-cov==4.0.0
      pytest-cover==3.0.0
      tomli==2.0.1
      typing_extensions==4.4.0
      zipp==3.11.0
    

Side notes: I don't think having multiple versions of python is convenient to use, nor it is required in most cases actually. Probably one version as a "default"/to go and maybe a new release / release candidate as a second one is enough. Cannot imagine a situation where I need 5 or 6 versions...The venvs manage versions of python libraries and e.g. docker containers (or other virtualization method) can abstract anything else (not necessarily python related). Not to be misunderstood: Is @Nathan 's solution bad? Nope, it's good, but there are some points I disagree with, that's all. Also his comments helped me out to come up with a solution of my own, so upvote from me :)

Gameplay
  • 1,142
  • 1
  • 4
  • 16