1

I am trying to send a Pull Request to an opensource library.

But I keep getting ModuleNotFoundError: No module named 'coverage'

This is the log in Github CI

Run tox -e lint,py310-dj40
lint: freeze> python -m pip freeze --all
lint: pip==23.0.1,setuptools==67.4.0,wheel==0.38.4
lint: commands[0]> poetry install --no-root -v
Using virtualenv: /home/runner/work/django-pattern-library/django-pattern-library/.tox/lint
Updating dependencies
Resolving dependencies...

Writing lock file

Finding the necessary packages for the current system

Package operations: 39 installs, 0 updates, 0 removals

  • Installing six (1.16.0)
  • Installing markupsafe (2.1.2)
  • Installing python-dateutil (2.8.2)
  • Installing pyyaml (6.0)
  • Installing smmap (5.0.0)
  • Installing click (8.1.3)
  • Installing ghp-import (2.1.0)
  • Installing jinja2 (3.1.2)
  • Installing gitdb (4.0.10)
  • Installing markdown (3.3.7)
  • Installing mergedeep (1.3.4)
  • Installing packaging (23.0)
  • Installing pyyaml-env-tag (0.1)
  • Installing watchdog (3.0.0)
  • Installing asgiref (3.6.0)
  • Installing gitpython (3.1.31)
  • Installing mccabe (0.6.1)
  • Installing mypy-extensions (1.0.0)
  • Installing mkdocs (1.4.2)
  • Installing mkdocs-material-extensions (1.1.1)
  • Installing pathspec (0.11.1)
  • Installing platformdirs (3.2.0)
  • Installing pycodestyle (2.7.0)
  • Installing pyflakes (2.3.1)
  • Installing pygments (2.14.0)
  • Installing pymdown-extensions (8.2)
  • Installing pytz (2023.3)
  • Installing soupsieve (2.4)
  • Installing sqlparse (0.4.3)
  • Installing tomli (2.0.1)
  • Installing beautifulsoup4 (4.12.0)
  • Installing black (22.12.0)
  • Installing coverage (6.5.0)
  • Installing django (3.2.18)
  • Installing flake8 (3.9.2)
  • Installing isort (5.11.5)
  • Installing mkdocs-git-revision-date-plugin (0.3.2)
  • Installing mkdocs-material (5.5.14)
  • Installing mkdocs-redirects (1.2.0)
lint: commands[1]> poetry run flake8
lint: commands[2]> poetry run isort --check --diff .
Skipped 2 files
lint: commands[3]> poetry run black --check --diff .
All done! ✨  ✨
34 files would be left unchanged.
lint: OK ✔ in 1 minute 2.2 seconds
py310-dj40: install_deps> python -I -m pip install 'Django<4.1,>=4.0'
py310-dj40: freeze> python -m pip freeze --all
py310-dj40: asgiref==3.6.0,Django==4.0.10,pip==23.0.1,setuptools==67.4.0,sqlparse==0.4.3,wheel==0.38.4
py310-dj40: commands[0]> poetry run python -X dev -W error runtests.py
Traceback (most recent call last):
  File "/home/runner/work/django-pattern-library/django-pattern-library/runtests.py", line 10, in <module>
    import coverage
ModuleNotFoundError: No module named 'coverage'
py310-dj40: exit 1 (1.10 seconds) /home/runner/work/django-pattern-library/django-pattern-library> poetry run python -X dev -W error runtests.py pid=2239
  lint: OK (62.20=setup[1.42]+cmd[57.42,1.26,0.86,1.25] seconds)
  py310-dj40: FAIL code 1 (5.90=setup[4.80]+cmd[1.10] seconds)
  evaluation failed :( (68.80 seconds)

I repeated the log here as it will be removed over time.

What I have tried:

  1. add coverage in envlist see https://github.com/torchbox/django-pattern-library/pull/213/commits/1d91a71eb13c7e3463524ed512352640f7008cba
  2. switch to poetry install --no-root -v see https://github.com/torchbox/django-pattern-library/pull/213/commits/2ee1b0c31a1e7f1154e127eeacf741e73a80f12a
  3. install coverage as part of tool.poetry.dependencies in pyproject.toml see https://github.com/torchbox/django-pattern-library/pull/213/commits/eb0c83eaafc43d7aaed9d6b9bcd3cfd6acc56e91
  4. switch to isolated_build=True in tox.ini https://github.com/torchbox/django-pattern-library/pull/213/commits/d3fc32ea5398219c05278fe46dacdbc26b1b905f
  5. add coverage to deps in tox.ini and this error is overcome see https://github.com/torchbox/django-pattern-library/pull/213/commits/332524c63c6ee94d7a009d5a90aed957c7d264b1 => which begs the question. Given that in pyproject.toml I already stated coverage and other test related dependencies, why wasn't this installed?
  6. i added poetry install -q in the testenv's commands see https://github.com/torchbox/django-pattern-library/pull/213/commits/18608a1138b1005e0450e25f61b5532f826e08ec this solution appears the same as ensuring tool.poetry.dev-dependencies are all installed, but now i get a failed test at py311-dj41 which I have traced to a python deprecation warning about 3.13. And it appears that Django 4.1 wasn't actuallt installed in the env py311-dj41 see https://github.com/torchbox/django-pattern-library/actions/runs/4587113230/jobs/8100417144?pr=213
 File "/home/runner/work/django-pattern-library/django-pattern-library/.tox/py311-dj41/lib/python3.11/site-packages/django/http/__init__.py", line 2, in <module>
    from django.http.request import (
  File "/home/runner/work/django-pattern-library/django-pattern-library/.tox/py311-dj41/lib/python3.11/site-packages/django/http/request.py", line 1, in <module>
    import cgi
  File "/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/cgi.py", line 57, in <module>
    warnings._deprecated(__name__, remove=(3,13))
  File "/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/warnings.py", line 514, in _deprecated
    warn(msg, DeprecationWarning, stacklevel=3)
DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
py311-dj41: exit 1 (0.81 seconds) /home/runner/work/django-pattern-library/django-pa

None of them work. What should i do?

sinoroc
  • 18,409
  • 2
  • 39
  • 70
Kim Stacks
  • 10,202
  • 35
  • 151
  • 282
  • 2
    Might sound overly simple but have you tried to add `coverage` in the `deps` of tox? – Philippe Remy Apr 02 '23 at 04:26
  • @PhilippeRemy doesn't that defeat the purpose of having tool.poetry.dev-dependencies in pyproject.toml? and also the log clearly shows installing coverage. so why is it still missing? – Kim Stacks Apr 02 '23 at 05:04
  • @PhilippeRemy you're right and it works after i add to deps in tox.ini. i cannot possibly repeat what i already have in pyproject.toml under [tool.poetry.dev-dependencies] what do you recommend? – Kim Stacks Apr 02 '23 at 05:09
  • https://python-poetry.org/docs/faq/#is-tox-supported -- https://stackoverflow.com/a/59522588 -- https://github.com/python-poetry/poetry/issues/1941 -- https://github.com/python-poetry/poetry/issues/1745 – sinoroc Apr 02 '23 at 09:01
  • @sinoroc I read the 4 links you provided. From what I understand is you recommend take out poetry from tox.ini. But the original open source maintainers wrote a bunch of command in tox.ini that I am not familiar on how to improve. can advise? – Kim Stacks Apr 02 '23 at 16:38
  • specifically these commands `poetry run python -X dev -W error runtests.py` and `poetry run django-admin render_patterns --settings=tests.settings.dev --pythonpath=. --dry-run deps =` – Kim Stacks Apr 02 '23 at 16:41
  • 1
    I looked at the current situation of [*torchbox/django-pattern-library*](https://github.com/torchbox/django-pattern-library/) and I must say I do not understand the approach taken in this project. The `tox_install.sh` confuses me. Basically there are 2 approaches I am comfortable with. Either *tox* is configured to let *Poetry* handle all the installations and do the `poetry run` things. Or *Poetry* is completely left out of the `tox.ini`. Personally I prefer the latter. This project seems to have a mix of both and I do not understand why. Maybe this solution is more clever than I am. – sinoroc Apr 02 '23 at 17:09
  • In the end I adopted your idea at this comment. https://github.com/orgs/python-poetry/discussions/4307#discussioncomment-1041658 NOt sure whether the maintainer will accept my PR at https://github.com/torchbox/django-pattern-library/pull/219 – Kim Stacks Apr 02 '23 at 17:56
  • 1
    @KimStacks I'm not knowledgeable enough to answer you on that. I'm just an "average" tox user :) – Philippe Remy Apr 03 '23 at 03:50

1 Answers1

1

The interaction between tox and poetry is highly confusing.

While I do not fully understand what @sinoroc solution is, I do get the idea that

Either tox is configured to let Poetry handle all the installations and do the poetry run things. Or Poetry is completely left out of the tox.ini.

I adopted their comment at https://github.com/orgs/python-poetry/discussions/4307#discussioncomment-1041658

and did the following

In tox.ini:

  • Replace whitelist_externals with allowlist_externals because whitelist_externals is deprecated
  • Added isolated_build = True
  • Remove install_command where it calls tox_install.sh
  • Drop tool.poetry.dev-dependencies and switch to using tool.poetry.extras in order for tox to detect them

In ci.yml

  • Because now use tool.poetry.extras so when call poetry install need to incldue —extras ‘test’ as argument

The last 2 points about dropping dev-dependencies and using extras are specifically from github.com/orgs/python-poetry/discussions/4307#discussioncomment-1041658

The new PR is at https://github.com/torchbox/django-pattern-library/pull/219

The specific changes are

tox.ini

from

[tox]
envlist = py{37,38,39,310,311}-dj{32,40,41,main}, lint
skipsdist = true

[testenv]
whitelist_externals =
    poetry
install_command =
    ./tox_install.sh {packages}

to

[tox]
envlist =
    py{37,38,39,310}-dj32
    py{38,39,310,311}-dj40
    py{38,39,310,311}-dj41
    py{38,39,310,311}-dj42
    py{310,311}-djmain
    lint
isolated_build = True

[testenv]
allowlist_externals =
    poetry
extras =
    test

pyproject.toml

from

[tool.poetry.dev-dependencies]
beautifulsoup4 = "^4.8"
coverage = "^6.2"
flake8 = "^3.7"
isort = "^5.10.1"
mkdocs = "^1.1.2"
mkdocs-material = "^5.5.14"
pymdown-extensions = "^8.0"
mkdocs-git-revision-date-plugin = "^0.3.1"
mkdocs-redirects = "^1.0.3"
black = "^22.3.0"

to

# A list of all of the dev dependencies, some of which are included in the
# below `extras`. They can be opted into by apps.
beautifulsoup4 = { version = "^4.8", optional = true }
coverage = { version = "^6.2", optional = true }
flake8 = { version = "^3.7", optional = true }
isort = { version = "^5.10.1", optional = true}
mkdocs = { version = "^1.1.2", optional = true}
mkdocs-material = { version = "^5.5.14", optional = true}
pymdown-extensions = { version = "^8.0", optional = true}
mkdocs-git-revision-date-plugin = { version = "^0.3.1", optional = true}
mkdocs-redirects = { version = "^1.0.3", optional = true}
black = { version = "^22.3.0", optional = true}

[tool.poetry.extras]
test = [
    "beautifulsoup4", "coverage", "flake8",
    "isort", "mkdocs", "mkdocs-material",
    "pymdown-extensions", "mkdocs-git-revision-date-plugin",
    "mkdocs-redirects", "black"
]

ci.yml

from: - run: poetry install to: - run: poetry install --extras 'test'

Kim Stacks
  • 10,202
  • 35
  • 151
  • 282