Python - Using pytest to skip test unless specified

Question

Background

I have am using pytest to test a web scraper that pushes the data to a database. The class only pulls the html and pushes the html to a database to be parsed later. Most of my tests use dummy data to represent the html.

Question

I want to do a test where a webpage from the website is scraped but I want the test to be automatically turned off unless specified. A similar scenario could be if you have an expensive or time consuming test that you do not want to always run.

Expected Solution

I am expecting some kind of marker that suppresses a test unless I give pytest to run all suppressed tests, but I do not see that in the documentation.

What I have done

• I am currently using the skip marker and comment it out.

• Tried to use the skipif marker and and give arguments to python script using this command from command prompt pytest test_file.py 1 and the following code below in the test file. The problem is that when I try to provide an argument to my test_file, pytest is expecting that to be another file name so I get an error "no tests run in 0.00 seconds, ERROR: file not found: 1"

  if len(sys.argv) == 1:
    RUN_ALL_TESTS = False
  else:
    RUN_ALL_TESTS = True
  ...
  # other tests
  ...
  
  @pytest.mark.skipif(RUN_ALL_TESTS)
  def test_scrape_website():
    ...
  

• I might be able to treat the test as a fixture and use @pytest.fixture(autouse=False), not sure how to override the autouse variable though

• A similar solution was stated in How to skip a pytest using an external fixture? but this solutions seems more complicated than what I need.

Answer 1

The docs describe exactly your problem: https://docs.pytest.org/en/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option. Copying from there:

Here is a conftest.py file adding a --runslow command line option to control skipping of pytest.mark.slow marked tests:

# content of conftest.py

import pytest


def pytest_addoption(parser):
    parser.addoption(
        "--runslow", action="store_true", default=False, help="run slow tests"
    )


def pytest_collection_modifyitems(config, items):
    if config.getoption("--runslow"):
        # --runslow given in cli: do not skip slow tests
        return
    skip_slow = pytest.mark.skip(reason="need --runslow option to run")
    for item in items:
        if "slow" in item.keywords:
            item.add_marker(skip_slow)

We can now write a test module like this:

# content of test_module.py
import pytest


def test_func_fast():
    pass


@pytest.mark.slow
def test_func_slow():
    pass

Answer 2

There's a couple ways to handle this, but I'll go over two common approaches I've seen in Python baselines.

1) Separate your tests by putting the "optional" tests in another directory.

Not sure what your project layout looks like, but you can do something like this (only the test directory is important, the rest is just a toy example layout):

README.md
setup.py
requirements.txt
test/
    unit/
        test_something.py
        test_something_else.py
    integration/
        test_optional.py
application/
    __init__.py
    some_module.py

Then, when you invoke pytest, you invoke it by doing pytest test/unit if you want to run just the unit tests (i.e. only test_something*.py files), or pytest test/integration if you want to run just the integration tests (i.e. only test_optional.py), or pytest test if you want to run all the tests. So, by default, you can just run pytest test/unit.

I recommend wrapping these calls in some sort of script. I prefer make since it is powerful for this type of wrapping. Then you can say make test and it just runs your default (fast) test suite, or make test_all, and it'll run all the tests (which may or may not be slow).

Example Makefile you could wrap with:

.PHONY: all clean install test test_int test_all uninstall

all: install

clean:
    rm -rf build
    rm -rf dist
    rm -rf *.egg-info

install:
    python setup.py install

test: install
    pytest -v -s test/unit

test_int: install
    pytest -v -s test/integration

test_all: install
    pytest -v -s test

uninstall:
    pip uninstall app_name

2) Mark your tests judiciously with the @pytest.mark.skipif decorator, but use an environment variable as the trigger

I don't like this solution as much, it feels a bit haphazard to me (it's hard to tell which set of tests are being run on any give pytest run). However, what you can do is define an environment variable and then rope that environment variable into the module to detect if you want to run all your tests. Environment variables are shell dependent, but I'll pretend you have a bash environment since that's a popular shell.

You could do export TEST_LEVEL="unit" for just fast unit tests (so this would be your default), or export TEST_LEVEL="all" for all your tests. Then in your test files, you can do what you were originally trying to do like this:

import os

...

@pytest.mark.skipif(os.environ["TEST_LEVEL"] == "unit")
def test_scrape_website():
  ...

Note: Naming the test levels "unit" and "integration" is irrelevant. You can name them whatever you want. You can also have many many levels (like maybe nightly tests or performance tests).

Also, I think option 1 is the best way to go, since it not only clearly allows separation of testing, but it can also add semantics and clarity to what the tests mean and represent. But there is no "one size fits all" in software, you'll have to decide what approach you like based on your particular circumstances.

HTH!

Answer 3

A very simply solution is to use the -k argument. You can use the -k parameter to deselect certain tests. -k tries to match its argument to any part of the tests name or markers You can invert the match by using not (you can also use the boolean operators and and or). Thus -k 'not slow' skips tests which have "slow" in the name, has a marker with "slow" in the name, or whose class/module name contains "slow".

For example, given this file:

import pytest

def test_true():
    assert True

@pytest.mark.slow
def test_long():
    assert False

def test_slow():
    assert False

When you run:

pytest -k 'not slow'

It outputs something like: (note that both failing tests were skipped as they matched the filter)

============================= test session starts =============================
platform win32 -- Python 3.5.1, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: c:\Users\User\Documents\python, inifile:
collected 3 items

test_thing.py .                                                          [100%]

============================= 2 tests deselected ==============================
=================== 1 passed, 2 deselected in 0.02 seconds ====================

Because of the eager matching you might want to do something like putting all your unittests in a directory called unittest and then marking the slow ones as slow_unittest (so as to to accidentally match a test that just so happens to have slow in the name). You could then use -k 'unittest and not slow_unittest' to match all your quick unit tests.

More pytest example marker usage

Answer 4

Form a little class for reuse of @xverges code on multiple marks/cli options;

@dataclass
class TestsWithMarkSkipper:
    ''' Util to skip tests with mark, unless cli option provided. '''

    test_mark: str
    cli_option_name: str
    cli_option_help: str

    def pytest_addoption_hook(self, parser):
        parser.addoption(
            self.cli_option_name,
            action="store_true",
            default=False,
            help=self.cli_option_help,
        )

    def pytest_collection_modifyitems_hook(self, config, items):
        if not config.getoption(self.cli_option_name):
            self._skip_items_with_mark(items)

    def _skip_items_with_mark(self, items):
        reason = "need {} option to run".format(self.cli_option_name)
        skip_marker = pytest.mark.skip(reason=reason)
        for item in items:
            if self.test_mark in item.keywords:
                item.add_marker(skip_marker)

Usage example (must be put in conftest.py):

slow_skipper = TestsWithMarkSkipper(
    test_mark='slow',
    cli_option_name="--runslow",
    cli_option_help="run slow tests",
)
pytest_addoption = slow_skipper.pytest_addoption_hook
pytest_collection_modifyitems = slow_skipper.pytest_collection_modifyitems_hook